CharXiv

About CharXiv

Chart understanding is crucial for applying Multimodal Large Language Models (MLLMs) to tasks like analyzing scientific papers and financial reports. However, current datasets often use simplified charts with template-based questions, leading to overly optimistic progress assessments. We introduce CharXiv, an evaluation suite with 2,323 diverse and challenging charts from scientific papers. CharXiv includes two question types: (1) descriptive questions on basic chart elements and (2) reasoning questions requiring synthesis of complex visual information. Human experts curated and verified all charts and questions. Our findings show a significant gap in reasoning skills, with the strongest proprietary model (GPT-4o) achieving 47.1% accuracy and the best open-source model (InternVL Chat V1.5)

Key Topics

Details