We introduce QRData, the first benchmark for advanced quantitative reasoning with data, to assess models' abilities of data-based statistical and causal reasoning. To ensure the quality of our benchmark, we first gather quantitative reasoning teaching and research resources, and then annotate questions based on the materials. We collect multiple-choice questions and numerical questions from these resources, and ensure that the gold answer is unique. We also collect data descriptions like the purpose and format of the data sheets from the resources, and provide them along with questions to models.

To separate the challenge of quantitative reasoning from data analysis and analyze whether models master the quantitative reasoning skills, we create an auxiliary benchmark for comparison called Quantitative Reasoning with Text (QRText). Questions in QRText can be answered without data. QRText contains 290 questions, 100 for statistical reasoning and 190 for causal reasoning. For ease of model comparison, the ratio of statistical/causal questions of QRText is similar to QRData.

We develop several zero-shot reasoning methods as our baselines:

• Table Question Answering (Table QA): The task of table QA asks models to answer a question based on tables. To evaluate if our benchmark can be solved by existing table QA methods, we select a competitive model TableLlama.
• Chain-of-Thought (CoT): CoT prompting enables models to conduct complex natural language reasoning through intermediate reasoning steps.
• Program-of-Thoughts (PoT): PoT prompting asks models to generate a Python code program, and uses an external computer to execute the code. The output of the code is regarded as the answer.
• ReAct-style Prompting: ReAct is a prevalent prompting paradigm for agent reasoning. It combines reasoning and acting in language models for task solving. To address our task, we restrict the action space to Python code execution, and ask models to generate thoughts and actions.
• Code Interpreter Assistants: GPT models can be used as agents (called assistants), and a code interpreter tool is built-in, which could execute Python code in a sandbox to interpret the data.

We experiment with the CoT, PoT, and ReAct-style prompting methods on the following models:

• General-Purpose LLMs: Llama-2-chat, Gemini-Pro and GPT-4
• Code LLMs: CodeLlama-instruct and Deepseek-coder-instruct
• LLM for Mathematical Reasoning: WizardMath
• LLM for Agent Reasoning: AgentLM

GPT-4 with the code interpreter assistant achieves the best performance, and Deepseek-coder-instruct with PoT prompting is the best among open-source models. The best model is 18% worse than human, showing that QRData is challenging for LLMs.

We go deeper into the primary difficulties models face in addressing the task of quantitative reasoning with data. This may provide insights into how to design methods to better solve our task.

We evaluate models on our auxiliary benchmark QRText, and compare the performance on QRText and QRData to quantify the difficulty of data analysis.

All models perform better on QRText than QRData from 1.8% to 11.5%, and the gap is larger for smaller models. If we control the knowledge and skills required by restricting questions to the same source (the book OpenIntro statistics), models perform 6% better on average. These provide evidence that most models have difficulty in data analysis.

We observe the performance gap between statistical reasoning and causal reasoning on QRData, and the gap remains on QRText. GPT-4 achieves 89% accuracy in statistical questions on QRText, but only about half the accuracy in causal questions. This exhibits the unique difficulty of causal reasoning regardless of data analysis.

As shown in the failure case of GPT-4, when asked to predict the causal relation between two variables L L1 radiculopathy and R L5 radiculopathy, GPT-4 proposes a wrong plan of calculating the conditional probabilities in Step 2. As correlation does not imply causation, although GPT-4 successfully executes the plan, it makes a wrong prediction.