On This Page· 5
A 2025 study reveals that current large language models failed on nearly 70% of financial data extraction tasks. Despite their ability to generate complex text, these AI tools struggle with AI table parsing in high-stakes environments. This gap between general language skills and precise numeric retrieval raises a critical question for business leaders: can we trust these systems with our financial data readability?
The 180-Test Benchmark: How LLMs Parse Financial Data
A 2025 study evaluated three large language models on their ability to handle financial data tasks, revealing a stark gap between potential and current performance. The benchmark tested ChatGPT 4, a Custom GPT configured with financial sources, and ChatPDF. These models were challenged with 180 tests derived from annual reports across three distinct industries: pharma (Roche Group), defense (Ruag), and telecom (Swisscom). The goal was to measure how well these tools could extract specific financial figures from complex document structures.
The methodology focused on six specific financial questions, ranging from simple data retrieval like revenue and intangible assets to complex calculations such as net income and return on equity. Each question was tested in two conditions: against the full annual report, which includes narrative text, and against table-only extractions, where the context was removed. This dual approach allowed researchers to isolate the impact of surrounding text on the accuracy of LLM table interpretation. The study aimed to determine whether providing structured data to AI in its raw tabular form actually improves the reliability of the extracted numbers.
The headline statistic is difficult to ignore: an overall accuracy rate of just 31.11%. Out of 180 tests, only 56 were classified as correct. Another 122 tests (67.78%) were wrong, and merely two (1.11%) were semi-correct, requiring manual assistance to fix. This result demonstrates that financial data readability remains a significant hurdle for current LLMs. For high-stakes applications where quantitative precision is mandatory, relying on AI to read tables is currently too risky. The data suggests that while these models can process text well, their ability to accurately navigate the two-dimensional logic of financial tables is far from reliable.
Context Beats Isolation: Why Text-Rich Reports Win
The study’s most counter-intuitive finding is that feeding an AI the full annual report yielded better accuracy than providing only the financial tables. This challenges the common assumption that removing narrative text simplifies financial data readability by reducing noise. In reality, the surrounding prose acts as a crucial anchor for LLM table interpretation.
The Linear Text Constraint
Large Language Models are trained primarily on linear text, not two-dimensional matrices. A PDF table is a relational structure where the value in one cell depends on the headers in the row and column above it. Without the explanatory text that defines these line items or clarifies accounting adjustments, the model struggles to map the grid logic. The narrative text provides the semantic context needed to disambiguate similar terms, such as distinguishing between gross and net figures. When that context is stripped away, the model is left to guess the structural relationships, leading to a higher rate of error.
Format Matters: Swisscom vs. Ruag
Performance also varied significantly based on the report’s layout. The Swisscom report, which featured a standard, well-structured format with 29 images, achieved the highest response rate. In contrast, the Ruag International report, which used an unconventional layout with images on 17 of its 57 pages, resulted in lower accuracy. This contrast highlights that while images do not inherently hinder performance, a non-standard structure can confuse models during AI table parsing. The data suggests that conventional formatting helps models align with their training data, making structured documents significantly more reliable for extraction than irregular ones.
| Report | Layout Characteristics | Performance Outcome |
|---|---|---|
| Swisscom | Standard structure, 29 images | Highest response rate |
| Ruag International | Unconventional layout, images on 17/57 pages | Lower accuracy |
Query Specificity and the Failure of Complex Ratios
The difficulty of the query significantly impacts performance. While models could retrieve simple data points like revenue or intangible assets with reasonable consistency, they failed completely on complex calculations. For the Return on Equity (ROE) question, which requires identifying multiple components and performing a calculation, the rate of incorrect answers was 100% in multiple instances. This demonstrates a clear boundary: reading a number is fundamentally different from performing multi-step financial logic.
The models often failed to locate specific components like “average shareholder equity,” leading to incorrect or missing calculations. However, adding specific context to the prompt made a difference. When instructions included the correct fiscal year and the company name, accuracy improved. This suggests that while the models lack inherent financial logic, they can be guided if the query is precise enough to limit their search space.
FAQ: Can We Trust AI for Financial Table Parsing?
Is ChatGPT 4 better than ChatPDF for financial reports? No. ChatPDF achieved the highest correct rate at 34.48%, but performance across all models remained clustered near the 30% mark. In LLM table interpretation, model choice matters less than the quality of the source data.
Do LLMs hallucinate financial figures? Yes. Models frequently confuse similar line items or pull data from the wrong fiscal year. Because AI table parsing often misreads contextual cues, manual verification is mandatory for any quantitative decision.
How should we prepare a PDF for AI analysis? Keep the original narrative context intact. Stripping out text to isolate tables actually reduces accuracy. Reports with full text provide the anchors the AI needs to resolve ambiguity, making financial data readability significantly higher when context is preserved.
Implications for Business Decision-Makers
Current AI table parsing excels at narrative summarization but remains unsafe for quantitative financial audits. Teams should maintain human-in-the-loop workflows for any task requiring high reliability in financial data extraction.
While today’s models fail on complex metrics, research trajectories point toward future reliability in LLM table interpretation. For now, the safest approach is to treat AI as a reader, not a calculator. Use it to analyze the narrative sections of annual reports, where it excels at synthesizing complex text. However, keep all quantitative table extraction manual. Until the technology evolves to handle two-dimensional structure reliably, human verification is the only safeguard against silent financial errors.