Reasoning-Based RAG Is a Better Crypto DD Tool Than Vector Search

@jetpippo's tweet nailed the problem: most crypto DD stacks vector-embed whitepapers, chop tokenomics tables into meaningless chunks, then wonder why retrieval sucks. He found a vesting cliff on page 40 that his old setup missed entirely. That's not a bad prompt or a small model — that's the retrieval architecture failing at the one job it had.

Here's why it happens, and why reasoning-based indexing (the approach behind PageIndex) handles crypto docs better than the vector-search stack everyone defaults to.

Why vector search breaks on tokenomics docs

Standard RAG does three things: split the document into fixed-size chunks (usually 300-800 tokens), embed each chunk into a vector, then retrieve the top-k chunks whose embeddings are closest to your query's embedding.

This works fine for prose — blog posts, FAQs, narrative docs where meaning lives in sentences. It falls apart on the documents crypto DD actually depends on:

• Tables get shredded. A vesting schedule with columns for allocation, TGE unlock %, cliff length, and linear release period gets sliced mid-row when it crosses a chunk boundary. The chunk that contains "12-month cliff" might not contain "team allocation," so the two facts never end up in the same retrieved chunk.
• Semantic similarity isn't the same as relevance. Your query "what's the cliff for team tokens" embeds close to any chunk that talks about vesting in general — marketing copy about "long-term alignment," a risk-disclosure paragraph, a competitor comparison — not necessarily the specific appendix table with the actual number.
• Structure is invisible. A human skimming a whitepaper uses the table of contents: jump to Tokenomics, then Vesting, then the appendix table. Vector search has no concept of "this document has sections" — it just has floating chunks in embedding space.
• Numbers don't embed well. "40%," "12 months," "Q3 2027" — these carry almost no semantic signal on their own. A chunk full of numbers can sit far from a natural-language query in vector space even when it's exactly the answer.

That's how you end up with a DD report that confidently summarizes "reasonable vesting terms" while missing the 6-month cliff buried in Appendix C.

How reasoning-based RAG actually differs

PageIndex skips embeddings entirely. Instead of chunking the document, it builds a hierarchical tree index — essentially a machine-readable table of contents with section boundaries, subsections, and page references, the same structure a human would build by skimming the doc once.

When you query it, an LLM reasons over that tree top-down: "does the answer to this question live under Tokenomics? Under Vesting Schedule? Under the Team Allocation subsection?" It navigates like a person flipping to the right chapter, not like a search engine matching keywords in a haystack. Because it reasons over structure rather than similarity scores, it can follow cross-references ("see Appendix A") and land on a table that has zero semantic overlap with your query wording but is exactly where the answer lives.

The tradeoff: it's slower and more expensive per query than a vector lookup, because you're paying for LLM reasoning steps instead of a cheap cosine-similarity scan. For a 40-page whitepaper that's a non-issue. For a 500-document corpus you're querying thousands of times a day, it adds up.

Worked example: finding a vesting cliff PageIndex-style

Say you're doing DD on a new L2 token and you've got a 52-page PDF: whitepaper + tokenomics appendix combined, structure roughly:

1. Executive Summary
2. Protocol Overview
3. Token Utility
4. Tokenomics
   4.1 Total Supply & Allocation
   4.2 Vesting Schedule
5. Team & Advisors
6. Risk Factors
Appendix A: Vesting Table (by cohort)
Appendix B: Audit Summary

Step 1 — Index the doc. Upload the PDF. Instead of chunking it into ~150 embedded fragments, the tool parses the doc's actual structure (headings, page breaks, table boundaries) into a tree: each node is a section with its page range and a short LLM-generated summary of what's in it.

Step 2 — Ask the real question. Your query: "What's the cliff period and unlock schedule for the team allocation?"

Step 3 — Watch it reason, not match. Instead of scoring 150 chunks for similarity, the model walks the tree: "Token Utility (3) — not it. Tokenomics (4) — plausible, check children. 4.1 Total Supply — has allocation percentages but no cliff info. 4.2 Vesting Schedule — mentions cliffs generally, references Appendix A for exact figures. Appendix A: Vesting Table — this is page 40, has the cohort-by-cohort breakdown." It follows the "see Appendix A" cross-reference the way you would.

Step 4 — Get the actual number. The answer comes back: "Team allocation (18% of supply): 12-month cliff, then 24-month linear unlock, per the cohort table on page 40." That's the exact fact @jetpippo's tweet was pointing at — the thing an embedding-based top-5 retrieval pass would've had maybe a coin-flip's chance of surfacing, because "12-month cliff" sitting alone in a table row doesn't semantically resemble a natural-language question.

When to use which

| | Vector search RAG | Reasoning-based RAG (PageIndex) | |---|---|---| | Best for | Narrative docs, FAQs, chat transcripts, large corpora queried constantly | Structured docs: whitepapers, tokenomics appendices, audit reports, legal terms | | Cost per query | Cheap (embedding + similarity lookup) | Higher (LLM reasoning per query) | | Handles tables/numbers | Poorly — chunks split rows, numbers embed weakly | Well — treats the table as a structural unit | | Speed at scale | Fast, scales to millions of docs | Slower, better suited to targeted DD than mass indexing | | Setup | Standard, lots of tooling (pgvector, Pinecone, etc.) | Newer, fewer off-the-shelf integrations |

If you're building a chatbot over your Discord history or a support FAQ, vector search is still the right default — it's cheap and good enough. If you're doing due diligence on a specific token's whitepaper before allocating capital, reasoning-based retrieval is worth the extra latency and cost. The whole point of DD is not missing the thing on page 40.

A sane middle ground some teams run: vector search for broad "what does this project do" questions across a big doc library, reasoning-based retrieval as the second pass specifically for tokenomics, vesting, and legal sections where a missed fact is expensive.

Common mistakes

• Trusting a RAG summary without checking the source page. Reasoning-based retrieval is much better, not infallible. Always verify the cited page/section against the actual PDF before you make a decision on it — especially for cliff dates and unlock percentages.
• Using vector search for the whole DD pipeline because it's what you already have wired up. It's fine for "summarize the protocol's use case," bad for "what exact percentage unlocks at TGE." Match the tool to the question type instead of forcing one pipeline to do both.
• Feeding it scanned/image-only PDFs and expecting structure detection to work. Reasoning-based indexing depends on being able to parse headings and tables. If the whitepaper is a scanned image with no text layer, OCR it first or you'll get the same garbage-in-garbage-out problem as any other retrieval method.
• Assuming a bigger top-k fixes vector search's blind spots. Pulling the top 20 chunks instead of top 5 doesn't fix the fact that vector similarity doesn't understand document structure — it just means you're paying to send more irrelevant chunks to the model and burying the real answer deeper in the context.

FAQ

Does reasoning-based RAG replace vector search entirely?

No. It's a better fit for structured, high-stakes documents like whitepapers and tokenomics appendices, but vector search is still cheaper and perfectly adequate for large, narrative-heavy corpora like chat logs or FAQs where you're not hunting for one precise buried fact.

Can I use this on a project's audit report, not just the whitepaper?

Yes — audit reports are actually a great fit, since they're heavily structured (findings by severity, section by contract) and the specific finding you care about is often buried in an appendix rather than the executive summary.

Why did my old vector-search setup miss a number that was clearly in the document?

Because vector similarity matches meaning, not facts. A table row like "Team: 12mo cliff" carries little semantic signal on its own, so it can rank far below chunks that just talk about vesting in general terms — even though those chunks don't contain the number you actually needed.

→ Ask the index what to build your crypto research stack

→ Free credits for these tools

Written by McKlaud AI. Want to know which AI tools actually fit your business? Get a free AI audit.