CSpace  > 中国科学院计算技术研究所期刊论文  > 英文
Are Neural Ranking Models Robust?
Wu, Chen1,2; Zhang, Ruqing1,2; Guo, Jiafeng1,2; Fan, Yixing1,2; Cheng, Xueqi1,2
2023-04-01
发表期刊ACM TRANSACTIONS ON INFORMATION SYSTEMS
ISSN1046-8188
卷号41期号:2页码:36
摘要Recently, we have witnessed the bloom of neural ranking models in the information retrieval (IR) field. So far, much effort has been devoted to developing effective neural ranking models that can generalize well on new data. There has been less attention paid to the robustness perspective. Unlike the effectiveness, which is about the average performance of a system under normal purpose, robustness cares more about the system performance in the worst case or under malicious operations instead. When a new technique enters into the real-world application, it is critical to know not only how it works in average, but also how would it behave in abnormal situations. So, we raise the question in this work: Are neural ranking models robust? To answer this question, first, we need to clarify what we refer to when we talk about the robustness of ranking models in IR. We show that robustness is actually a multi-dimensional concept and there are three ways to define it in IR: (1) the performance variance under the independent and identically distributed (I.I.D.) setting; (2) the out-of-distribution (OOD) generalizability; and (3) the defensive ability against adversarial operations. The latter two definitions can be further specified into two different perspectives, respectively, leading to five robustness tasks in total. Based on this taxonomy, we build corresponding benchmark datasets, design empirical experiments, and systematically analyze the robustness of several representative neural ranking models against traditional probabilistic ranking models and learning-to-rank (LTR) models. The empirical results show that there is no simple answer to our question. While neural ranking models are less robust against other IR models in most cases, some of them can still win two out of five tasks. This is the first comprehensive study on the robustness of neural ranking models. We believe thewaywe study the robustness as well as our findings would be beneficial to the IR community. We will also release all the data and codes to facilitate the future research in this direction.
关键词Robustness Ranking Models Systematic Analysis
DOI10.1145/3534928
收录类别SCI
语种英语
资助项目National Natural Science Foundation of China (NSFC)[62006218] ; National Natural Science Foundation of China (NSFC)[61902381] ; National Natural Science Foundation of China (NSFC)[61872338] ; Youth Innovation Promotion Association CAS[20144310] ; Youth Innovation Promotion Association CAS[2021100] ; Foundation and Frontier Research Key Program of Chongqing Science and Technology Commission[cstc2017jcyjBX0059] ; Lenovo-CAS Joint Lab Youth Scientist Project
WOS研究方向Computer Science
WOS类目Computer Science, Information Systems
WOS记录号WOS:000971779000004
出版者ASSOC COMPUTING MACHINERY
引用统计
被引频次:4[WOS]   [WOS记录]     [WOS相关记录]
文献类型期刊论文
条目标识符http://119.78.100.204/handle/2XEOYT63/21412
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Guo, Jiafeng; Cheng, Xueqi
作者单位1.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing, Peoples R China
2.Univ Chinese Acad Sci, 6 Kexueyuan South Rd, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714
Wu, Chen,Zhang, Ruqing,Guo, Jiafeng,et al. Are Neural Ranking Models Robust?[J]. ACM TRANSACTIONS ON INFORMATION SYSTEMS,2023,41(2):36.
APA Wu, Chen,Zhang, Ruqing,Guo, Jiafeng,Fan, Yixing,&Cheng, Xueqi.(2023).Are Neural Ranking Models Robust?.ACM TRANSACTIONS ON INFORMATION SYSTEMS,41(2),36.
MLA Wu, Chen,et al."Are Neural Ranking Models Robust?".ACM TRANSACTIONS ON INFORMATION SYSTEMS 41.2(2023):36.
条目包含的文件
条目无相关文件。
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Wu, Chen]的文章
[Zhang, Ruqing]的文章
[Guo, Jiafeng]的文章
百度学术
百度学术中相似的文章
[Wu, Chen]的文章
[Zhang, Ruqing]的文章
[Guo, Jiafeng]的文章
必应学术
必应学术中相似的文章
[Wu, Chen]的文章
[Zhang, Ruqing]的文章
[Guo, Jiafeng]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。