Sycophancy in vision-language models: A systematic analysis and an inference-time mitigation framework

doi:10.1016/j.neucom.2025.131217

	Sycophancy in vision-language models: A systematic analysis and an inference-time mitigation framework
	Zhao, Yunpu 1; Zhang, Rui 2; Xiao, Junbin 3; Ke, Changxin 2; Hou, Ruibo 4; Hao, Yifan 2; Li, Ling 5
	2026
发表期刊	NEUROCOMPUTING
ISSN	0925-2312
卷号	659 页码:14
摘要	Large Vision-Language Models (LVLMs) have shown significant capability in vision-language understanding. However, one critical issue that persists in these models is sycophancy, where models are unduly influenced by leading or deceptive prompts, resulting in biased outputs and hallucinations. Despite the rapid development of LVLMs, evaluating and mitigating sycophancy remains largely under-explored. In this work, we fill this gap by systematically analyzing sycophancy across multiple vision-language benchmarks and propose an inference-time mitigation framework. We curate leading queries and quantify the susceptibility of state-of-the-art LVLMs to prompt-induced bias, revealing consistent performance degradation and instability across models and tasks. Our analysis further uncovers model-specific behavioral traits, such as sentiment sensitivity and prediction polarity shifts under sycophancy. To mitigate these issues, we propose a training-free, model-agnostic framework that operates entirely at inference time. Our approach first employs a query neutralizer, leveraging a language model to suppress implicit sycophantic bias in user queries. We then introduce a sycophancy-aware contrastive decoding mechanism that dynamically recalibrates token-level output distributions by contrasting responses to neutralized and leading queries. Finally, an adaptive logits refinement module further modifies the contrasted logits by integrating both an adaptive plausibility filter and query sentiment scaler, ensuring coherent and robust generation. Extensive experiments demonstrate that this framework effectively mitigates sycophancy across all evaluated models, while maintaining performance on neutral prompts. Our results suggest that sycophancy in LVLMs is a general and urgent challenge, and that inference-time strategies offer a promising path toward trustworthy multimodal reasoning.
关键词	Vision-language models Contrastive decoding Model hallucinations
DOI	10.1016/j.neucom.2025.131217
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2023YFB4502200] ; NSF of China[62302478] ; NSF of China[U22A2028] ; NSF of China[62341411] ; Strategic Priority Research Program of the Chinese Academy of Sciences[XDB0660200] ; Strategic Priority Research Program of the Chinese Academy of Sciences[XDB0660201] ; Strategic Priority Research Program of the Chinese Academy of Sciences[XDB0660202] ; CAS Project for Young Scientists in Basic Research[YSBR-029] ; Youth Innovation Promotion Association CAS
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001601182700001
出版者	ELSEVIER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41593
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Rui
作者单位	1.Univ Sci & Technol China, Sch Comp Sci & Technol, Hefei 230026, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, State Key Lab Processors, Beijing 100190, Peoples R China 3.Natl Univ Singapore, Acad Beijing, Dept Comp Sci, Key Comp Technol, Singapore 119077, Singapore 4.Univ Illinois, Chicago, IL 61820 USA 5.Chinese Acad Sci, Intelligent Software Res Ctr, Inst Software, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhao, Yunpu,Zhang, Rui,Xiao, Junbin,et al. Sycophancy in vision-language models: A systematic analysis and an inference-time mitigation framework[J]. NEUROCOMPUTING,2026,659:14.
APA	Zhao, Yunpu.,Zhang, Rui.,Xiao, Junbin.,Ke, Changxin.,Hou, Ruibo.,...&Li, Ling.(2026).Sycophancy in vision-language models: A systematic analysis and an inference-time mitigation framework.NEUROCOMPUTING,659,14.
MLA	Zhao, Yunpu,et al."Sycophancy in vision-language models: A systematic analysis and an inference-time mitigation framework".NEUROCOMPUTING 659(2026):14.