PageIndex: Reasoning-Based RAG for DeFi Whitepaper and Audit Research

Most "AI research tools" for DeFi are embedding search wearing a trenchcoat. You paste in a whitepaper, ask a question, and get back a paragraph that sounds related because the words are close in vector space — not because it actually answers what you asked. If you've ever gotten a confidently wrong answer about a token's unlock schedule from an AI tool, that's why.

PageIndex takes a different approach. Instead of chunking documents and embedding them into vectors, it builds a reasoning tree over the document's actual structure — sections, subsections, cross-references — and searches through it the way a human analyst would: by reading the table of contents, deciding which section is relevant, and drilling in. No embeddings, no vector database, no similarity math. Just structured reasoning. It's picked up 29.5k GitHub stars for exactly that reason.

Check out the tool page for setup details.

What it actually does differently

Standard RAG (retrieval-augmented generation) works like this: split your PDF into 500-token chunks, embed each chunk as a vector, embed your question as a vector, then grab the chunks whose vectors are mathematically closest to your question's vector. This works fine for FAQ-style lookups. It falls apart for documents where the answer requires connecting information across sections — which is basically every DeFi whitepaper and every audit report worth reading.

Say you're checking whether a protocol's liquidation mechanism is actually solvent under stress. The answer isn't in one paragraph. It's in the collateralization ratio (section 3), the oracle update frequency (section 5), and a footnote about emergency shutdown conditions (appendix B). Vector similarity has no idea these three things are related — they don't share vocabulary. A human reading the doc would connect them instantly because they understand what "solvency" requires.

PageIndex mimics the human approach. It generates a hierarchical index of the document (think: an LLM-generated, semantically-aware table of contents), then does tree search over that index — reasoning about which nodes are relevant to your query, similar to how a person skims a table of contents before diving into a specific chapter. It's vectorless: no embeddings, no similarity threshold tuning, no chunk-size guesswork.

Who this is for

If you're manually reading whitepapers and audit reports to evaluate DeFi protocols — you're the target user. Specifically:

• Researchers and analysts vetting new protocols before writing them up or recommending them
• Due diligence for allocators — anyone deciding whether to deploy capital into a protocol and needing to actually understand the mechanism design, not just skim a summary
• Auditors and security researchers cross-referencing findings against spec documents
• Anyone building an internal research tool that needs to answer questions grounded in long, structurally complex PDFs (whitepapers, audit reports, governance proposals)

If you just need quick keyword lookup in short docs, this is overkill. Save it for documents where structure and cross-referencing actually matter.

Worked example: auditing a lending protocol's liquidation logic

Here's a realistic pass — reviewing a hypothetical lending protocol's whitepaper (40 pages) plus its most recent audit report (25 pages) to answer: "Under what conditions could the protocol become insolvent, and does the audit flag any unresolved issues related to those conditions?"

Step 1 — Build the index. Feed both PDFs into PageIndex. It parses document structure and generates a tree index — something like:

Whitepaper
├── 1. Overview
├── 2. Protocol Mechanics
│   ├── 2.1 Collateralization
│   ├── 2.2 Liquidation Engine
│   └── 2.3 Oracle Design
├── 3. Tokenomics
└── 4. Risk Parameters

Audit Report
├── 1. Scope
├── 2. Findings
│   ├── 2.1 Critical
│   ├── 2.2 High
│   └── 2.3 Medium/Low
└── 3. Recommendations

This step is the whole point — no chunking, no embedding model choice, no vector store to stand up. Just structural parsing.

Step 2 — Query with reasoning search. You ask the question above. PageIndex's reasoning-based retrieval doesn't do a similarity lookup — it reasons over the tree: "insolvency conditions" maps to 2.2 Liquidation Engine and 2.3 Oracle Design in the whitepaper, and to 2.1 Critical and 2.2 High in the audit (since unresolved severe findings are what you actually care about).

Step 3 — Get grounded synthesis. The tool retrieves the actual content from those specific nodes and synthesizes an answer citing which section each claim came from — e.g., "Liquidation is triggered at 110% collateralization (§2.1), but the audit's Critical finding C-03 notes the oracle update interval (§2.3, whitepaper) creates a window where liquidations can lag price moves by up to 90 seconds under network congestion (Audit §2.1)."

That's a connected answer across two documents and four sections — the kind of synthesis that vector similarity search reliably fails at, because "90 seconds" and "insolvency" don't share embedding space, but a human (or a reasoning engine) knows they're the same risk.

When to use it vs. when to skip it

| Scenario | Use PageIndex | Use standard vector RAG | |---|---|---| | Long structured docs (whitepapers, audits, 10-Ks) | Yes — structure is the whole advantage | Overkill setup, mediocre results | | Cross-referencing claims across sections | Yes | Struggles — no semantic linking | | Short FAQ-style docs, single-fact lookup | Skip — no structural benefit | Fine, faster to set up | | You already have a vector DB pipeline (Pinecone, pgvector) and it's working for simple lookups | Skip, don't rebuild what isn't broken | Keep it | | Need semantic similarity across thousands of unrelated short docs (e.g., support tickets) | Skip — tree reasoning doesn't scale the same way | Yes, this is vector search's home turf |

Honest take: PageIndex isn't a universal RAG replacement, and it's not trying to be. It's built for long, structurally rich documents where the answer requires connecting dots a similarity score can't see. If your use case is "search 50,000 short documents for the closest match," stick with embeddings — that's genuinely what they're good at. PageIndex's advantage disappears on unstructured or very short content because there's no meaningful tree to reason over.

Common mistakes

Throwing unstructured or scanned garbage PDFs at it. PageIndex builds its index from document structure — headings, sections, hierarchy. A badly-OCR'd scan with no real heading structure gives it nothing to reason over, and you'll get worse results than plain vector search. Clean, well-formatted PDFs (most audit reports and whitepapers are fine) work best.

Using it for documents where nothing needs connecting. If your question is answerable from a single paragraph, you don't need reasoning-based tree search — you're paying more latency and compute for no benefit. Save it for the "connect three sections to get the real answer" cases.

Skipping validation on the reasoning trace. Because there's no similarity score to sanity-check against, it's tempting to trust the output blindly. Always check which sections PageIndex actually pulled from — the tool cites node references, and reading them takes 30 seconds and catches misreasoning before it costs you money on a bad protocol call.

Assuming it replaces reading the audit yourself. It gets you to the relevant sections fast and connects claims across documents, but it's a research accelerant, not a substitute for actually reading the critical findings before you commit capital. Treat the synthesized answer as your starting point for verification, not your final answer.

FAQ

Does PageIndex use embeddings at all?

No — that's the core design choice. It builds a hierarchical, LLM-generated index of document structure and does reasoning-based tree search over it instead of vector similarity matching. No vector database required.

Is PageIndex better than standard RAG for every use case?

No. It's built for long, structurally complex documents where answers require connecting information across sections — whitepapers, audits, legal filings. For short documents or large-scale similarity search across many unrelated files, standard vector RAG is still the better and simpler tool.

Can I use PageIndex on scanned or poorly formatted PDFs?

You can, but results degrade — the tool relies on parsing document structure (headings, sections) to build its reasoning tree, so a document with no discernible structure gives it little to reason over. Clean, well-formatted PDFs get the best results.

---

→ Ask the index what to build your reasoning based rag stack

→ Free credits for these tools

Written by McKlaud AI. Want to know which AI tools actually fit your business? Get a free AI audit.