The Impact of Document Formats on Embedding Performance and RAG Effectiveness in Tax Law Applications

Before delving into the main report, this research investigates how different document formats (PDF, XML, JSON, and Markdown) affect the performance of Retrieval-Augmented Generation (RAG) systems for tax law applications. Our findings indicate that structured formats like XML and JSON significantly outperform PDF in embedding quality and retrieval effectiveness, with XML excelling in preserving complex hierarchical relationships crucial for tax legislation, while JSON offers superior processing efficiency. The format selection should be guided by specific use case requirements, with hybrid approaches often providing optimal results for comprehensive tax law applications.

Introduction: The Document Format Challenge in Legal RAG Systems

The emergence of Large Language Models (LLMs) has revolutionized legal research, with Retrieval-Augmented Generation (RAG) systems showing particular promise in domains requiring factual accuracy and up-to-date information. RAG systems, which combine the generative capabilities of LLMs with external knowledge retrieval, have become increasingly valuable in tax law, where precision and contextual understanding are paramount. These systems can dramatically reduce research time while providing reliable, sourced legal information for tax professionals, researchers, and the general public.

A critical yet often overlooked aspect of RAG system performance is the format in which legal documents are stored and processed. Different document formats—PDF, XML, JSON, and Markdown—possess distinct characteristics that significantly impact how effectively information is embedded, retrieved, and ultimately presented to users. The choice of document format affects not only technical performance metrics but also the practical utility of legal AI tools for professionals navigating complex tax legislation. This impact becomes particularly evident when working with sophisticated models like GPT-4o and vector database systems that must process, store, and retrieve tax code information effectively.

As legal AI applications proliferate, understanding the relationship between document formats and RAG performance has become increasingly important. This research investigates how document formats influence the embedding process, retrieval effectiveness, and output quality of RAG systems specifically designed for tax law applications. By examining these relationships through empirical analysis and theoretical frameworks, we aim to provide actionable insights for legal technology developers and practitioners seeking to optimize their RAG implementations for tax law research and advisory services.

The significance of this research extends beyond technical considerations to practical outcomes for legal professionals. As noted in Stanford Law School’s research, LLMs enhanced with retrieval capabilities can “provide clear, understandable explanations of complex laws and regulations” while potentially helping to “identify inconsistencies in existing laws” and “predict likely impacts of new laws or policies”1. Our investigation into format optimization seeks to advance these capabilities by ensuring that the foundational document processing pipeline supports optimal performance for these complex legal tasks.

Theoretical Foundation: RAG Systems, Embeddings, and Document Formats

Retrieval-Augmented Generation in Legal Domains

Retrieval-Augmented Generation systems address a fundamental limitation of standalone LLMs: their inability to access information beyond their training data. In legal contexts, RAG systems retrieve relevant legal texts—including statutes, regulations, case law, and commentary—to augment the contextual knowledge of an LLM, enabling more precise and authoritative responses to legal queries. This capability is particularly valuable in tax law, where accuracy and currency are essential.

The structure of a tax law RAG system typically includes several key components: a document processing pipeline for converting raw documents into processable text; an embedding generation module that creates vector representations of document chunks; a vector database for storing and indexing these embeddings; a retrieval mechanism for identifying relevant documents based on query similarity; and an LLM integration component that combines retrieved information with LLM capabilities to generate coherent responses35. This architecture allows the system to provide responses that are both contextually relevant and factually grounded in authoritative legal sources.

Research has demonstrated the effectiveness of this approach in various legal contexts. For example, Stanford Law School’s study on “Large Language Models as Tax Attorneys” found that RAG leveraging the U.S. Code of Federal Regulations and U.S. Code significantly enhanced an LLM’s ability to understand and apply tax law1. Similarly, implementations in Australia have shown how RAG can transform tax law research by providing accurate, contextualized responses to tax-related queries with transparent source citation3.

The efficacy of RAG in tax law applications stems from its ability to bridge the gap between general language understanding and domain-specific knowledge. By retrieving and incorporating relevant tax law documents, RAG systems can navigate the complex, interconnected nature of tax legislation while maintaining the conversational fluidity and explanatory capabilities of advanced LLMs.

Document Formats and Their Characteristics

The four document formats under consideration—PDF, XML, JSON, and Markdown—each present distinct characteristics that affect how legal information is stored, processed, and retrieved:

PDF (Portable Document Format): PDFs are ubiquitous in legal settings due to their ability to preserve formatting and ensure uniformity across devices. However, they present significant challenges for information extraction and embedding. As noted in comparative research, PDFs suffer from “poor text extraction quality, inconsistent formatting recognition, and heavy processing requirements”8. These limitations can significantly impact the quality of embeddings generated from PDF content, as the format prioritizes visual presentation over machine-readable structure.

XML (Extensible Markup Language): XML is a structured format that excels at representing hierarchical relationships in data. It provides explicit semantic meaning through tags, potentially enhancing the quality of embeddings by preserving relationships between legal concepts. According to comparative research, XML is “better for sending complex data structures”13, making it potentially valuable for representing the intricate hierarchical nature of tax legislation, with its nested sections, subsections, and cross-references.

JSON (JavaScript Object Notation): JSON is a lightweight, structured format frequently used in APIs and data interchange. Research indicates that “JSON is faster for transmitting data” due to its smaller file size13, with studies showing JSON files can be 24.7% smaller than equivalent XML files. This efficiency, combined with its intuitive key-value structure, makes JSON potentially valuable for legal data representation, though it may lack some of the semantic richness of XML. Projects like PDF-GPT4-JSON demonstrate the perceived benefits of JSON representation for legal documents7.

Markdown: Markdown is a lightweight markup language designed for readability and ease of writing. While not primarily designed for structured data representation, its simplicity and readability make it attractive for certain legal documentation needs, particularly where human-readable text is a priority. Markdown’s focus on content over structure may influence how embeddings capture the semantics of legal text, potentially emphasizing substantive content over structural relationships.

Embedding Techniques for Legal Documents

Embeddings represent the meaning of text in high-dimensional vector spaces, enabling similarity-based retrieval and understanding of semantic relationships. For legal documents, particularly tax legislation, the quality of embeddings is crucial for accurate information retrieval.

Current research highlights several embedding techniques relevant to legal text processing:

TF-IDF (Term Frequency-Inverse Document Frequency): This method is “particularly useful for JSON data, where the frequency of terms can vary significantly”2. TF-IDF helps identify the most relevant terms in legal documents by weighing their importance based on occurrence across documents. While less sophisticated than neural approaches, TF-IDF remains valuable for certain legal applications due to its interpretability and efficiency.

Word2Vec: This technique is “effective for both JSON and XML, as it captures semantic relationships between words”2.By training on large datasets, Word2Vec can generate embeddings that reflect the contextual meaning of legal terms, potentially improving retrieval relevance for tax law queries.

BERT (Bidirectional Encoder Representations from Transformers): BERT is “highly effective for understanding the context within JSON and XML structures”2. It captures bidirectional dependencies, making it suitable for understanding the complex relationships between different elements in legal data. BERT-based embeddings have shown particular promise for capturing the nuanced language of legal texts.

Recent research from Microsoft on embedding models highlights that the dimensionality of embeddings significantly impacts performance and resource requirements. While models like OpenAI’s text-embedding-3-large can generate 3072 dimensions, “a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536”16. This finding suggests opportunities for optimization in legal RAG systems, where balancing embedding quality with resource efficiency is essential.

Advanced Prompting Methodologies for Legal Analysis

The effectiveness of RAG systems can be enhanced through advanced prompting techniques that guide the LLM’s reasoning process:

Chain-of-Thought (CoT): CoT prompting guides the reasoning process by generating a series of logically coherent intermediate steps of inference. This approach has been shown to improve the reliability of LLM responses, particularly for complex reasoning tasks like legal analysis. The methodology is especially valuable in tax law, where decisions often follow structured logical progressions through multiple conditions and exceptions.

Least-to-Most Prompting (LtM): LtM prompting “breaks down problems into simpler subproblems and solves them sequentially”4. Unlike Chain-of-Thought, where each step is independent, LtM “uses the output of previous subproblems as input for the next”4, making it particularly valuable for decomposing complex legal questions. In tax law applications, this approach can help navigate multi-part analyses, such as determining eligibility for specific tax treatments or exemptions.

Tree of Thoughts (ToT): ToT “maintains a tree of thoughts, where thoughts represent coherent language sequences that serve as intermediate steps toward solving a problem”18. This approach enables systematic exploration of reasoning paths with “lookahead and backtracking”18, which can be especially valuable for navigating the complex decision trees often present in tax law interpretation. Recent innovations like Retrieval Augmented Thought Tree (RATT) combine RAG with tree-structured thought processes, conducting “lookahead and general-to-detail fact-checking analyses at each node of the thought tree”11.

These methodologies, when combined with effective document format processing, can significantly enhance the performance of tax law RAG systems by enabling more sophisticated reasoning patterns while maintaining factual grounding in authoritative sources.

Methodology: A Structured Approach to Format Evaluation

Our investigation into the impact of document formats on RAG performance for tax law applications employs a methodical approach that combines theoretical analysis with empirical evidence from the literature. We structure our methodology around the advanced prompting techniques specified in the research requirements:

Chain-of-Thought Progression

We structure our analysis in progressive stages, beginning with the fundamental characteristics of each document format, then examining their impact on embedding generation, followed by assessing their influence on retrieval performance, and finally evaluating their effect on output quality. This staged approach allows us to build a comprehensive understanding of the causal relationships between document formats and RAG performance across the entire processing pipeline.

For each stage, we synthesize findings from multiple sources, identifying areas of consensus and divergence in the literature. This approach enables us to develop a nuanced understanding of how format characteristics influence various aspects of RAG system performance.

Least-to-Most Decomposition

Following the Least-to-Most prompting approach, we decompose our analysis into sequential subtasks that build upon each other:

1. Format Analysis: We begin by examining the inherent properties of each document format, including their structural characteristics, metadata capabilities, and processing requirements. This foundational understanding informs subsequent analyses.
2. Embedding Impact Assessment: Building on the format analysis, we evaluate how the identified characteristics affect embedding quality, considering factors such as semantic preservation, structural representation, and dimensional efficiency.
3. Retrieval Effectiveness Evaluation: Using insights from the embedding assessment, we analyze how embedding quality influences retrieval performance, examining metrics such as precision, recall, and query specificity across different formats.
4. Output Quality Assessment: Finally, we evaluate how retrieval effectiveness translates to output quality in the generated responses, considering factors such as factual accuracy, coherence, and comprehensiveness.

This sequential approach enables us to establish clear causal relationships and identify specific mechanisms by which document formats influence RAG performance.

Tree-of-Thought Exploration

We apply a Tree-of-Thought methodology to explore different branches of analysis, considering various scenarios and use cases for each document format. This approach allows us to comprehensively evaluate the trade-offs and contextual factors that influence format selection for tax law applications.

For each format, we consider multiple “thought branches” representing different use cases, document characteristics, and implementation approaches. By exploring these branches systematically, we develop a more nuanced understanding of the conditions under which each format might excel or struggle in tax law RAG applications.

Evaluation Metrics

Our analysis considers several key metrics to evaluate the impact of document formats:

• Embedding Quality: We assess semantic coherence (how well embeddings capture the meaning of legal text), dimensionality efficiency (the relationship between embedding dimension and performance), and structural preservation (how well hierarchical relationships are maintained).
• Retrieval Performance: We evaluate precision (the relevance of retrieved documents), recall (the completeness of relevant document retrieval), and F1 score (the harmonic mean of precision and recall).
• Processing Efficiency: We examine chunking efficiency (how effectively documents can be segmented), storage requirements (the size of embedded representations), and computational cost (the resources required for processing).
• Output Quality: We assess factual accuracy (alignment with authoritative sources), coherence (logical consistency of responses), and comprehensiveness (coverage of relevant information).

By applying these metrics across different document formats, we develop a comparative framework for evaluating their relative performance and suitability for tax law RAG applications.

Analysis of Document Formats for Tax Law RAG

PDF Format: Prevalence and Challenges

PDFs are ubiquitous in legal settings, particularly for official publications of tax legislation and regulations. Their prevalence makes them an inevitable input format for legal RAG systems, despite presenting significant challenges for information extraction and embedding.

Embedding Impact:

The impact of PDF formats on embedding quality is multifaceted. PDF’s complex structure often leads to poor text extraction quality, which directly affects embedding fidelity. Research indicates that the visual formatting elements in PDFs can create noise in the extracted text, potentially distorting embeddings8. Furthermore, while PDFs preserve visual structure, the logical structure of the content (hierarchies, relationships) is often lost during text extraction, limiting the semantic richness of the resulting embeddings.

The experience of developing RAG applications using PDF sources highlights these challenges. For example, in building the SaveHaven tax appeal RAG system, developers “adapted a web scraper to pull in county records, regulations, and rules for property tax appeals”9, illustrating the considerable pre-processing required to make PDF-based legal information usable for RAG systems.

Retrieval Implications:

PDF format characteristics directly influence retrieval effectiveness in several ways. The inconsistent chunking resulting from PDF’s format often complicates the document segmentation process, potentially leading to suboptimal document chunks for retrieval. Unless properly tagged, PDFs typically offer limited structured metadata to enhance retrieval precision. Additionally, PDFs are inherently static, creating challenges for maintaining up-to-date tax information, which is particularly problematic in the rapidly evolving domain of tax law.

Projects like PDF-GPT4-JSON demonstrate the perceived necessity of converting PDFs to more structured formats for effective processing: “This project is designed to convert PDF files into JSON format using GPT-4. For each page in the PDF, a JSON file will be generated. The hierarchy of the JSON structure will be inferred from the layout of the data in the PDF”7. This conversion approach acknowledges the limitations of PDF for direct embedding and retrieval.

XML Format: Structural Richness and Semantic Clarity

XML’s inherent structure makes it potentially valuable for representing the hierarchical nature of tax legislation, with its nested sections, subsections, and cross-references.

Embedding Impact:

XML’s explicit tagging of structural elements can enhance embeddings by preserving the hierarchical relationships within tax legislation. This preservation of structure is particularly valuable for tax codes, where the relationship between provisions (e.g., general rules, exceptions, special cases) is often as important as the content itself. The semantic enrichment provided by XML tags can improve embedding quality by identifying the role of different text elements (e.g., <section>, <definition>, <exception>), enabling more nuanced vector representations.

XML’s standardized structure enables more reliable parsing and pre-processing, potentially improving embedding consistency across different sections of tax legislation. This consistency is essential for reliable similarity calculations during retrieval, ensuring that semantically related provisions are properly associated regardless of their location in the tax code.

Retrieval Implications:

XML’s structure enables more granular queries, potentially improving retrieval precision by targeting specific elements of tax legislation. For example, a query about exemptions to a particular tax rule could specifically target <exception> elements within the relevant section, increasing retrieval precision. The explicit relationships encoded in XML can enhance retrieval by capturing connections between related provisions, such as cross-references or hierarchical relationships, which are common in tax codes.

Research comparing XML and JSON notes that “XML is better for sending complex data structures”13, suggesting that XML’s additional structural information may benefit applications requiring rich hierarchical representation, such as tax law. This structural advantage may be particularly valuable for complex tax queries that require understanding of the relationships between different provisions.

JSON Format: Efficiency and Modern Integration

JSON’s lightweight structure and widespread adoption in modern applications make it an attractive option for representing tax legislation in RAG systems.

Embedding Impact:

JSON’s clear key-value structure typically results in cleaner text extraction compared to PDFs, potentially improving embedding quality by reducing noise and extraction artifacts. The efficient processing enabled by JSON’s lightweight nature allows for more sophisticated embedding techniques, potentially improving vector quality while maintaining reasonable computational requirements. While less expressive than XML, JSON’s nested structure can still effectively represent the hierarchical nature of tax legislation, maintaining important structural information in the embedding process.

Projects like “Embedding Techniques for Json and Xml” highlight that “understanding and utilizing text embeddings for JSON and XML is essential for enhancing the performance of NLP applications”2, suggesting that format-specific embedding approaches can maximize the potential of JSON-structured tax data.

Retrieval Implications:

JSON’s smaller file size contributes to “shorter program run-time and less power being consumed when reading and processing the file”13, potentially enabling more responsive retrieval systems for tax law applications where query latency is important. JSON’s native compatibility with most programming environments simplifies integration with modern RAG architectures, facilitating the development and deployment of advanced tax law research systems.

The experience of projects like “Building a Tax Appeal RAG with Milvus, LlamaIndex, and GPT” demonstrates the effectiveness of modern JSON-based approaches for tax applications: “SaveHaven is a RAG app that can help individuals contest and appeal property and income tax assessments. It streamlines the tax appeal process, making it more accessible and manageable for the general public”9. This implementation leverages JSON’s integration advantages to create accessible and responsive legal AI tools.

Markdown Format: Simplicity and Readability

Markdown’s simplicity and human-readability present unique characteristics for legal document representation, particularly for commentary and explanatory materials related to tax law.

Embedding Impact:

Markdown’s focus on human-readable text typically results in cleaner extractions for embedding, with minimal formatting artifacts to distort semantic representation. However, Markdown offers only basic structural elements (headings, lists), potentially limiting the richness of embeddings compared to XML or JSON when representing complex tax legislation. The simplicity of Markdown enables lightweight parsing, reducing preprocessing overhead and potentially allowing more computational resources to be dedicated to embedding quality optimization.

Retrieval Implications:

Markdown’s prioritization of content over structure may produce retrievals that focus more on substantive content than structural relationships, which could be advantageous for certain types of tax law queries focused on explanatory content rather than formal statutory language. Standard Markdown provides minimal metadata capabilities, potentially limiting retrieval enhancement through explicit tagging. However, Markdown’s ease of editing can facilitate ongoing content updates, ensuring retrievals reflect current tax legislation, which is particularly valuable in rapidly evolving areas of tax law.

While Markdown wasn’t explicitly discussed in many of the search results, its growing popularity for technical and legal documentation suggests potential applications in contexts where readability and maintainability are priorities, such as practice guides, commentary, and explanatory materials that supplement formal tax legislation.

Comparative Analysis: Impact on RAG Performance

Embedding Quality Comparison

Based on our analysis of the literature and technical characteristics, we can compare the relative impact of different formats on embedding quality:

This comparison reveals that XML demonstrates significant strengths in structural preservation and semantic enrichment, potentially leading to higher-quality embeddings that capture the complex relationships within tax legislation. JSON balances good structure with processing efficiency, while Markdown excels in simplicity but offers limited structural representation. PDF, despite its ubiquity, presents challenges across all embedding quality dimensions.

Retrieval Effectiveness Comparison

The impact of document formats on retrieval effectiveness can be compared across several dimensions:

XML’s rich structure enables highly specific queries, potentially improving both precision and recall for tax law retrievals. This advantage is particularly valuable for complex tax research questions that require precise navigation of the tax code. JSON offers a good balance of performance with easier updates, while Markdown’s simplicity facilitates updates but may limit query specificity.

Processing and Resource Requirements

Document formats also differ in their processing demands and resource requirements:

Research comparing JSON and XML found that “the JSON file was 24.7% smaller than the XML file” leading to “shorter program run-time and less power being consumed”13. This efficiency advantage may be particularly relevant for large-scale legal datasets, where processing and storage costs can be significant. However, XML’s program to deserialize files “took up 16.7% less flash memory than its JSON counterpart”13, suggesting different optimization priorities depending on system constraints.

Discussion: Implications for Tax Law RAG Applications

Format Selection Considerations

The optimal format for tax law RAG systems depends on several contextual factors that should guide implementation decisions:

Corpus Characteristics:

The complexity and structure of the tax corpus should influence format selection. Highly structured legislation with many cross-references may benefit from XML’s explicit relationship encoding, while simpler guidelines might be adequately represented in JSON or Markdown. For comprehensive tax law systems that include both formal legislation and explanatory materials, a hybrid approach may be optimal.

Update Frequency:

Tax legislation undergoes frequent updates, with implications for format selection. Formats that facilitate easy updates (JSON, Markdown) may be preferable for rapidly changing areas of tax law, while more stable foundational legislation might benefit from XML’s structural richness. The experience of SaveHaven in building tax appeal systems highlights the importance of “keeping improving as we add more and more data when the platform is being used”9, suggesting the value of formats that support efficient updates.

Query Complexity:

Applications requiring highly specific queries (e.g., “Find all exceptions to capital gains tax for agricultural properties”) may benefit from XML’s structural specificity, while more general research might be adequately served by simpler formats. The nature of the expected queries should significantly influence format selection, with more complex, relationship-focused queries potentially benefiting from more structured formats.

Integration Requirements:

Systems that need to integrate with modern web applications and APIs may favor JSON for its native compatibility with common development environments. The RAG implementation described in “Building a Tax Appeal RAG with Milvus, LlamaIndex, and GPT” demonstrates this advantage, leveraging JSON’s integration capabilities to build a comprehensive system that “streamlines the tax appeal process”9.

Hybrid Approaches: Combining Format Strengths

Rather than selecting a single format, optimal tax law RAG systems might employ hybrid approaches that leverage the strengths of multiple formats:

Multi-format Storage:

Storing the same content in multiple formats can enable format-specific optimizations for different operations. For example, a system might maintain XML representations for structured queries, JSON for API integration, and Markdown for human review and editing. This approach maximizes flexibility but introduces synchronization challenges.

Format Conversion Pipeline:

A pipeline that converts source documents (often PDFs) into structured formats (XML/JSON) for processing, while maintaining links to the original documents for reference, can combine accessibility with processing efficiency. Projects like PDF-GPT4-JSON demonstrate this approach: “This project is designed to convert PDF files into JSON format using GPT-4”7. Similarly, the SaveHaven team “adapted a web scraper to pull in county records, regulations, and rules for property tax appeals”9 before processing them in more structured formats.

Format-Adaptive Embedding:

Embedding techniques can be tailored to the specific format being processed, extracting maximum value from each format’s unique characteristics. For example, XML processing might emphasize structural relationships, while JSON processing might prioritize efficient key-value representation. Research on embedding techniques notes that “understanding and utilizing text embeddings for JSON and XML is essential for enhancing the performance of NLP applications”2, suggesting the value of format-specific approaches.

Optimal Dimensionality and Resource Efficiency

An important consideration in format processing for RAG systems is the optimization of embedding dimensionality. Recent research from Microsoft indicates that “the text-embedding-3-large can be used to return only 256 dimensions instead of the default 3072, while still performing very well”16. This finding suggests that significant resource savings can be achieved without substantial performance degradation.

The relationship between document format and optimal dimensionality presents an interesting area for optimization. More structured formats like XML, which explicitly encode relationships, might require fewer dimensions to maintain performance compared to less structured formats. This potential interaction between format structure and optimal dimensionality represents an opportunity for format-specific optimization in tax law RAG systems.

Emerging Best Practices for Format Processing

From our analysis, several best practices emerge for tax law RAG implementations:

Prioritize Structure Over Presentation:

Where possible, legal documents should be stored in formats that prioritize logical structure (XML, JSON) rather than visual presentation (PDF). This prioritization enhances embedding quality by preserving the semantic relationships that are essential for understanding tax legislation.

Leverage Explicit Metadata:

All formats can benefit from explicit metadata tagging (effective dates, jurisdictions, tax types), which can significantly enhance retrieval relevance. The implementation of metadata should be format-appropriate, using XML attributes, JSON fields, or Markdown front matter as appropriate.

Implement Robust Pre-processing:

Regardless of format, robust pre-processing pipelines that handle text cleaning, entity recognition, and normalization can significantly improve embedding quality. The experience of building the SaveHaven tax appeal system highlights the importance of this step: “We have leveraged open-source scrappers to fetch data from different government websites. Then, we chunked the data, transformed it into vector embeddings, and stored them in the Milvus vector database”9.

Consider Advanced Prompting Techniques:

The integration of advanced prompting methodologies like Tree of Thoughts can enhance the reasoning capabilities of tax law RAG systems. Recent innovations like the Retrieval Augmented Thought Tree (RATT) demonstrate how RAG can be integrated into tree-structured thought processes, conducting “lookahead and general-to-detail fact-checking analyses”11that may be particularly valuable for navigating complex tax regulations.

Conclusion: Navigating Format Choices for Optimal Tax Law RAG

Our analysis reveals that document format selection has significant implications for the performance and effectiveness of RAG systems in tax law applications. While no single format emerges as universally optimal, each presents distinct advantages and challenges that should inform system design decisions based on specific use cases and requirements.

XML offers superior structural representation and semantic richness, potentially enabling more precise and contextually aware retrievals, particularly for complex tax legislation with intricate hierarchical relationships. Its ability to explicitly encode relationships between tax provisions makes it especially valuable for queries that require understanding of these connections. However, its verbosity and processing complexity may present scaling challenges for large corpora.

JSON balances good structure with processing efficiency and modern API compatibility, making it well-suited for contemporary RAG architectures and frequently updated tax information. Its smaller file size and lower processing requirements make it attractive for systems where efficiency is a priority. The successful implementation of systems like SaveHaven demonstrates JSON’s effectiveness in practical tax law applications.

Markdown excels in simplicity, readability, and ease of maintenance, which may be particularly valuable for rapidly evolving tax guidance and commentary. Its human-centric design facilitates collaborative editing and ongoing updates, though its limited structural capabilities may constrain its suitability for representing complex legislative hierarchies.

PDF, despite its ubiquity in legal publishing, presents significant challenges for high-quality information extraction and embedding. While RAG systems must often ingest PDF content as source material, converting to more structured formats early in the processing pipeline appears advantageous, as demonstrated by projects like PDF-GPT4-JSON.

The optimal approach for many tax law RAG implementations may involve a hybrid model that combines formats based on specific requirements, potentially using XML for core legislative content, JSON for API integration and data exchange, and Markdown for human-editable commentary and guidance. This multi-format strategy can leverage the strengths of each format while mitigating their individual limitations.

As RAG technology continues to evolve, particularly with the integration of advanced reasoning methodologies like Tree of Thoughts and Retrieval Augmented Thought Tree, format-aware design will remain a critical consideration for building effective and reliable tax law AI tools. By selecting formats that optimize embedding quality, retrieval precision, and processing efficiency, developers can create RAG systems that truly enhance the accessibility and understanding of complex tax legislation.

References

1. Stanford Law School. (2023). Large Language Models as Tax Attorneys: A Case Study in Legal. https://law.stanford.edu/wp-content/uploads/2023/07/White-Paper_Large-Language-Models-as-Tax-Attorneys.pdf
2. Restack. (2023). Embedding Techniques for Json and Xml. https://www.restack.io/p/embeddings-knowledge-embedding-techniques-json-xml-cat-ai
3. Lingam, V. (2025). Transforming Tax Law Research: A Practical RAG Model. LinkedIn. https://www.linkedin.com/pulse/transforming-australian-tax-law-research-practical-rag-lingam-vk86c
4. Prompting.org. (2024). Least-to-Most Prompting. https://learnprompting.org/docs/intermediate/least_to_most
5. AI Developer Courses. (2024). Retrieval augmentation for GPT-4o. https://www.ai-for-devs.com/blog/retrieval-augmentation-for-gpt-4o
6. ACL Anthology. (2023). Retrieval-based Evaluation for LLMs: A Case Study in Korean Legal. https://aclanthology.org/2023.nllp-1.13.pdf
7. Guerrero, M. (2024). PDF-GPT4-JSON. GitHub. https://github.com/maximoguerrero/PDF-GPT4-JSON
8. Tyagi, S. (2025). How File Formats Can Impact the Performance of LLM Powered Text Generation. LinkedIn. https://www.linkedin.com/pulse/how-file-formats-impact-performance-text-generation-llm-shivam-tyagi-wg3xc
9. Zilliz. (2024). Building a Tax Appeal RAG with Milvus, LlamaIndex, and GPT. https://zilliz.com/blog/build-tax-appeal-rag-with-milvus-llamaindex-and-gpt
10. IBM. (2024). What is tree-of-thoughts? https://www.ibm.com/think/topics/tree-of-thoughts
11. arXiv. (2024). Large Language Model Guided Tree-of-Thought. https://arxiv.org/pdf/2406.02746.pdf
12. ACM Digital Library. (2024). Comparing XML and JSON Characteristics as Formats for Data Exchange. https://dl.acm.org/doi/10.1109/LES.2024.3450576
13. Fhh, N. (2023). Digital Form with GPT4 Vision API. GitHub. https://github.com/nathanfhh/Digital-Form-with-GPT4-Vision-API
14. Duarte, R. (2024). The Importance of Fine-Tuning with RAG in the Tax Area for Accountants. https://www.robertodiasduarte.com.br/en/a-importancia-do-fine-tuning-com-rag-na-area-tributaria-para-contadores/
15. Microsoft. (2024). Choose the right dimension count for your embedding models. https://devblogs.microsoft.com/azure-sql/embedding-models-and-dimensions-optimizing-the-performance-resource-usage-ratio/
16. arXiv. (2024). Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores. https://arxiv.org/html/2502.20364v1
17. Prompt Engineering Guide. (2023). Tree of Thoughts (ToT). https://www.promptingguide.ai/techniques/tot
18. OpenReview. (2023). Large Language Model Guided Tree-of-Thought. https://openreview.net/forum?id=a648X9AoL4

Citations:

1. https://law.stanford.edu/wp-content/uploads/2023/07/White-Paper_Large-Language-Models-as-Tax-Attorneys.pdf
2. https://www.restack.io/p/embeddings-knowledge-embedding-techniques-json-xml-cat-ai
3. https://www.linkedin.com/pulse/transforming-australian-tax-law-research-practical-rag-lingam-vk86c
4. https://learnprompting.org/docs/intermediate/least_to_most
5. https://www.ai-for-devs.com/blog/retrieval-augmentation-for-gpt-4o
6. https://www.ragie.ai/blog/evaluating-ragie-against-legalbench-rag
7. https://github.com/maximoguerrero/PDF-GPT4-JSON
8. https://www.linkedin.com/pulse/how-file-formats-impact-performance-text-generation-llm-shivam-tyagi-wg3xc
9. https://zilliz.com/blog/build-tax-appeal-rag-with-milvus-llamaindex-and-gpt
10. https://www.ibm.com/think/topics/tree-of-thoughts
11. https://arxiv.org/pdf/2406.02746.pdf
12. https://aclanthology.org/2023.nllp-1.13.pdf
13. https://dl.acm.org/doi/10.1109/LES.2024.3450576
14. https://github.com/nathanfhh/Digital-Form-with-GPT4-Vision-API
15. https://www.robertodiasduarte.com.br/en/a-importancia-do-fine-tuning-com-rag-na-area-tributaria-para-contadores/
16. https://devblogs.microsoft.com/azure-sql/embedding-models-and-dimensions-optimizing-the-performance-resource-usage-ratio/
17. https://arxiv.org/html/2502.20364v1
18. https://www.promptingguide.ai/techniques/tot
19. https://openreview.net/forum?id=a648X9AoL4
20. https://www.christopherspenn.com/2024/02/you-ask-i-answer-retrieval-augmented-generation-for-tax-law/
21. https://platform.openai.com/docs/actions/data-retrieval
22. https://arxiv.org/html/2408.10343v1
23. https://www.um.edu.mt/library/oar/bitstream/123456789/131176/1/ERSJ27(B)A72.pdf
24. https://www.restack.io/p/embeddings-answer-embed-xml-in-json-cat-ai
25. https://www.youtube.com/watch?v=k5RrpqItQkQ
26. https://arxiv.org/html/2412.09078v1
27. https://blog.futuresmart.ai/gpt-4-semantic-search-and-vector-databases-revolutionizing-question-answering
28. https://arxiv.org/abs/2408.10343
29. https://arxiv.org/html/2503.03444
30. https://community.openai.com/t/what-is-the-best-type-of-format-to-use-for-uploaded-documents-for-gpts/504365
31. https://natlawreview.com/article/rags-glitches-large-language-models-small-law-practices
32. https://platform.openai.com/docs/guides/embeddings
33. https://arxiv.org/pdf/2306.07075.pdf
34. https://stackoverflow.com/questions/77469097/how-can-i-process-a-pdf-using-openais-apis-gpts
35. https://www.robertodiasduarte.com.br/en/retrieval-augmented-generation-rag-guia-completo-para-contadores/
36. https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/
37. https://myscale.com/blog/maximize-chatgpt-performance-vector-databases-step-by-step-guide/
38. https://community.openai.com/t/gpts-best-file-format-for-knowledge-to-feed-gpts/497368
39. https://visai.ai/blogs/41/NitiBench-RAG-on-legal-domain-might-not-be-easy
40. https://huggingface.co/blog/mteb
41. https://journals.flvc.org/FLAIRS/article/download/135648/139786/260096
42. https://www.promptingguide.ai/pt/techniques/tot
43. https://unimatrixz.com/blog/prompt-engineering-tree-of-thought-and-graph-rag/
44. https://blog.gopenai.com/tree-of-thoughts-prompting-tot-cracking-the-code-to-simplified-understanding-58ff9a3683de
45. https://www.linkedin.com/pulse/layer-thoughts-new-concept-retrieval-prompt-suman-satpathy-wyyxc
46. https://neurips.cc/virtual/2023/poster/72797
47. https://dev.to/stephenc222/how-to-implement-a-tree-of-thoughts-in-python-4jmc
48. https://relevanceai.com/prompt-engineering/master-tree-of-thoughts-prompting-for-better-problem-solving
49. https://arxiv.org/html/2401.14295v3
50. https://aimlapi.com/academy-articles/tree-of-thought-prompting
51. https://dev-kit.io/blog/ai/tree-of-thought-prompting
52. https://www.forbes.com/sites/lanceeliot/2023/09/08/prompt-engineering-embraces-tree-of-thoughts-as-latest-new-technique-to-solve-generative-ai-toughest-problems/
53. https://aclanthology.org/2023.findings-emnlp.835.pdf

Answer from Perplexity: pplx.ai/share