Synthetic Data Generation with LLMs

Retrieval-Augmented Generation (RAG): Revolutionizing Financial Data Analysis

This article explores the rising popularity of Retrieval-Augmented Generation (RAG) in financial firms, focusing on how it streamlines knowledge access and addresses key challenges in LLM-driven solutions. RAG combines a retriever (locating relevant documents) with a Large Language Model (LLM) (synthesizing responses), proving invaluable for tasks like customer support, research, and internal knowledge management.

Effective LLM evaluation is crucial. Inspired by Test-Driven Development (TDD), an evaluation-driven approach uses measurable benchmarks to validate and refine AI workflows. For RAG, this involves creating representative input-output pairs (e.g., Q&A pairs for chatbots, or source documents and expected summaries). Traditionally, this dataset creation relied heavily on subject matter experts (SMEs), leading to time-consuming, inconsistent, and costly processes. Furthermore, LLMs' limitations in handling visual elements within documents (tables, diagrams) hampered accuracy, with standard OCR tools often falling short.

Overcoming Challenges with Multimodal Capabilities

The emergence of multimodal foundation models offers a solution. These models process both text and visual content, eliminating the need for separate text extraction. They can ingest entire pages, recognizing layout structures, charts, and tables, thereby improving accuracy, scalability, and reducing manual effort.

Case Study: Wealth Management Research Report Analysis

This study uses the 2023 Cerulli report (a typical wealth management document combining text and complex visuals) to demonstrate automated Q&A pair generation. The goal was to generate questions incorporating visual elements and produce reliable answers. The process employed Anthropic's Claude Sonnet 3.5, which handles PDF-to-image conversion internally, simplifying the workflow and reducing code complexity.

The prompt instructed the model to analyze specific pages, identify page titles, create questions referencing visual or textual content, and generate two distinct answers for each question. A comparative learning approach was implemented, presenting two answers for evaluation and selecting the superior response. This mirrors human decision-making, where comparing alternatives simplifies the process. This aligns with best practices highlighted in “What We Learned from a Year of Building with LLMs,” emphasizing the stability of pairwise comparisons for LLM evaluation.

Claude Opus, with its advanced reasoning capabilities, acted as the "judge," selecting the better answer based on criteria like clarity and directness. This significantly reduces manual SME review, improving scalability and efficiency. While initial SME spot-checking is essential, this dependency diminishes over time as system confidence grows.

Optimizing the Workflow: Caching, Batching, and Page Selection

Several optimizations were implemented:

• Caching: Caching significantly reduced costs. Processing the report without caching cost $9; with caching, it cost $3 (a 3x savings). The cost savings are even more dramatic at scale.
• Batch Processing: Using Anthropic's Batches API halved output costs, proving far more cost-effective than individual processing.
• Page Selection: Processing the document in 10-page batches yielded the best balance between precision and efficiency. Using clear page titles as anchors proved more reliable than relying solely on page numbers for linking Q&A pairs to their source.

Example Output and Benefits

An example shows how the LLM accurately synthesized information from tables within the report to answer a question about AUM distribution. The overall benefits include:

• Significant cost reduction through caching and batch processing.
• Reduced time and effort for SMEs, allowing them to focus on higher-value tasks.

This approach demonstrates a scalable and cost-effective solution for creating evaluation datasets for RAG systems, leveraging the power of multimodal LLMs to improve accuracy and efficiency in financial data analysis. The images from the original text are included below:

The above is the detailed content of Synthetic Data Generation with LLMs. For more information, please follow other related articles on the PHP Chinese website!