• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>Information Processing & Management >3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?
【24h】

3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?

机译：3.5K跑步次数，5K主题，3M评估和70M度量：Ad hoc式CLEF十年的趋势是什么？

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

Multilingual information access and retrieval is a key concern in today global society and, despite the considerable achievements over the past years, it still presents many challenges. In this context, experimental evaluation represents a key driver of innovation and multilinguality is tackled in several evaluation initiatives worldwide, such as CLEF in Europe, NTCIR in Japan and Asia, and FIRE in India. All these activities have run several evaluation cycles and there is a general consensus about their strong and positive impact on the development of multilingual information access systems. However, a systematic and quantitative assessment of the impact of evaluation initiatives on multilingual information access and retrieval over the long period is still missing. Therefore, in this paper we conduct the first systematic and large-scale longitudinal study on several CLEF Adhoc-ish tasks - namely the Adhoc, Robust, TEL., and GeoCLEF labs - in order to gain insights on the performance trends of monolingual, bilingual and multilingual information access systems, spanning several European and non-European languages, over a range of 10 years. We learned that monolingual retrieval exhibits a stable positive trend for many of the languages analyzed, even though the performance increase is not always steady from year to year due to the varying interests of the participants, who may not always be focused on just increasing performances. Bilingual retrieval demonstrates higher improvements in recent years - probably due to the better language resources now available - and it also outperforms monolingual retrieval in several cases. Multilingual retrieval shows improvements over the years and performances are comparable to those of bilingual and monolingual retrieval, and sometimes even better. Moreover, we have found evidence that the rule-of-thumb of a 3-year duration for an evaluation task is typically enough since top performances are usually reached by the third year and sometimes even by the second year, which then leaves room for research groups to investigate relevant research issues other than top performances. Overall, this study provides quantitative evidence that CLEF has achieved the objective which led to its establishment, i.e. making multilingual information access a reality for European languages. However, the outcomes of this paper not only indicate that CLEF has steered the community in the right direction, but they also highlight the many open challenges for multilinguality. For instance, multilingual technologies greatly depend on language resources and targeted evaluation cycles help not only in developing and improving them, but also in devising methodologies which are more and more language-independent.
机译：多语言信息的访问和检索是当今全球社会关注的主要问题，尽管过去几年取得了巨大成就，但它仍然提出了许多挑战。在这种情况下，实验评估代表了创新的关键驱动力，并且全球范围内的多种评估计划都致力于解决多种语言问题，例如欧洲的CLEF，日本和亚洲的NTCIR以及印度的FIRE。所有这些活动已经运行了多个评估周期，并且对这些活动对多语言信息访问系统的发展产生了积极而积极的影响已达成了普遍共识。但是，长期缺少评估活动对多语种信息访问和检索的影响的系统和定量评估。因此，在本文中，我们对几个CLEF Adhoc任务（即Adhoc，Robust，TEL。和GeoCLEF实验室）进行了首次系统的大规模纵向研究，以了解单语言，双语的性能趋势。以及跨多种欧洲和非欧洲语言的多语言信息访问系统，历时10年。我们了解到，尽管由于参与者的兴趣变化（而不是总是专注于提高性能），参与者之间的性能增长并不总是稳定的，但单语言检索对于许多分析的语言都呈现出稳定的积极趋势。近年来，双语检索显示出更高的改进-可能是由于现在可获得更好的语言资源-在某些情况下，它也优于单语检索。多语言检索显示了多年来的改进，其性能可与双语和单语检索相媲美，有时甚至更好。此外，我们发现有证据表明，评估任务为期3年的经验法则通常就足够了，因为通常在第三年甚至第二年就可以达到最高绩效，这为研究留下了空间小组调查除最佳表现以外的相关研究问题。总体而言，这项研究提供了定量证据，表明CLEF已实现了导致其成立的目标，即使多语言信息访问成为欧洲语言的现实。但是，本文的结果不仅表明CLEF已引导社区朝着正确的方向发展，而且还突显了多种语言面临的许多开放挑战。例如，多语言技术极大地依赖于语言资源，针对性的评估周期不仅有助于开发和改进它们，而且还有助于设计越来越与语言无关的方法。

著录项

相似文献

  • 外文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号