AI search & RAG

Most companies do not need a smarter model. They need a model that can see their own material.

The generic knowledge inside a frontier LLM is enormous and almost entirely irrelevant to the question your support agent is being asked right now. That question is about your refund policy, your SKU numbering, the clause in this customer's contract that was negotiated eighteen months ago by someone who no longer works there. No amount of model capability retrieves a document the model has never seen.

Retrieval-augmented generation is the plumbing that closes that gap. You take the question, find the handful of passages from your own corpus that actually bear on it, put those passages in front of the model, and ask it to answer from them. Done properly, it turns a general-purpose model into something that behaves like the most thorough person on your team — one who has read every ticket, every contract, every internal wiki page nobody has opened since 2022.

Done badly, it produces a system that is confidently, fluently wrong, which is considerably worse than having nothing at all.

Why naive RAG gives wrong answers

Almost every failed RAG project I have looked at failed in the same three places. None of them are the model's fault.

Chunking. The corpus gets split into fixed-size pieces — 500 tokens, 1,000 tokens, whatever the tutorial said — with no regard for what the document actually is. A contract gets cut mid-clause, so the retrieved passage contains the exception without the rule. A support runbook gets cut between the symptom and the fix. A table gets separated from its header row, and the model reads a column of numbers with no idea what they measure. The retrieval step then works perfectly and hands the model a fragment that cannot support a correct answer.

Good chunking is document-aware. Contracts split on clause boundaries and carry their section heading and parent agreement as metadata. Support tickets stay whole, because a ticket is already the right unit of meaning. Long-form documentation splits on headings with a short parent-context prefix prepended to each chunk so a passage retrieved in isolation still knows what page it came from. Tables get extracted and serialized separately rather than being sliced by a character counter. This is unglamorous work and it moves answer quality more than any other single decision in the pipeline.

No re-ranking. Vector similarity is a first-pass filter, not a judgment of relevance. Embeddings capture rough topical proximity, which means the top result for "what's our SLA for enterprise customers" may well be a passage that talks a great deal about enterprise customers and never mentions an SLA. The standard fix is a two-stage retrieval: pull a wide candidate set — thirty, fifty passages — using the cheap vector search, then run those candidates through a cross-encoder re-ranker that scores each one against the query directly and keeps the best handful. It costs more per query and it is usually the difference between a system people trust and one they quietly stop using.

Hybrid retrieval belongs in the same conversation. Dense vectors are poor at exact-match terms: part numbers, error codes, proper nouns, the internal project name nobody outside the company has ever written down. Running keyword search (BM25) alongside vector search and fusing the two result sets recovers precisely the queries that embeddings are worst at, and those queries are disproportionately the ones your users care about.

No evaluation loop. This is the one that kills projects. The system gets built, it demos well against five questions someone made up in a meeting, it ships, and from that point on nobody has any idea whether it is getting better or worse. Every change — a new chunking strategy, a different embedding model, a prompt tweak — is a coin flip made on vibes.

What a real evaluation process looks like

Evals are not a phase at the end. They are the instrument panel, and you build them before you start tuning.

Start with a golden set: 100 to 200 real questions, drawn from actual support tickets, actual sales calls, actual internal Slack, each paired with the passage or passages that genuinely answer it. Real questions matter more than the count — invented questions are always cleaner and better-formed than the ones your users actually type, and a system tuned against clean questions falls over on messy ones.

Then measure the two stages separately, because they fail for different reasons:

Retrieval quality. For each question, is the correct passage in the retrieved set at all? Track recall at k (did we find it in the top 10?) and a rank-sensitive measure like MRR or nDCG (how close to the top did it land?). If recall is low, no amount of prompt engineering will save the answer — the model was never shown the right material. This is where chunking and hybrid search get fixed.

Answer quality. Given the retrieved passages, is the generated answer correct, and is it grounded in those passages rather than in the model's own priors? Faithfulness — every claim in the answer traceable to a retrieved chunk — is the metric that catches hallucination, and it is best scored by an LLM judge with a strict rubric, spot-checked by a human on a sample.

Add refusal behavior to the rubric explicitly. A RAG system that answers every question is broken. When the corpus does not contain the answer, the correct output is "I don't have that" plus a route to a human — and you should have questions in your golden set whose right answer is exactly that, so the metric rewards it.

Then wire the whole thing into CI. Every change to chunking, embeddings, prompt, or retrieval parameters runs the golden set and reports the delta. Once that loop exists, improving a RAG system stops being guesswork and becomes ordinary engineering.

Production telemetry closes the circle: log the query, the retrieved chunk IDs, the answer, and whatever feedback signal you can capture. Questions that retrieve nothing useful are a map of the holes in your corpus, and they are usually the most valuable output of the whole system in the first month.

Pinecone or pgvector

The vector database question gets far more attention than it deserves, and the honest answer for most projects is: use the database you already have.

pgvector is a Postgres extension. If your application data already lives in Postgres — and it usually does — pgvector means your embeddings live beside your rows, with the same backups, the same access control, the same transactions, and no second system to operate or pay for. You can filter by tenant, by document permissions, by date, in the same query that does the similarity search, which is a genuine advantage over bolting metadata filtering onto an external store. For a corpus in the range of a few hundred thousand to a few million chunks, on managed Postgres with an HNSW index, this is a solved problem and the right default.

Pinecone and the other managed vector databases earn their place when scale or operational load makes Postgres awkward: very large corpora, aggressive latency targets under high query concurrency, or a team that would rather not own index tuning and rebuild strategy. You pay for that in money and in a second system with its own consistency story — your source documents and their embeddings can now drift apart, and keeping them in sync becomes your problem.

The decision rule I use: start on pgvector, instrument retrieval latency and recall, and move only when a measurement — not a projection — says you have to. Migrating a retrieval layer later is a contained job, far cheaper than running infrastructure you did not need for a year. Keep the retrieval interface behind a small abstraction and the swap is a few days of work.

Embeddings deserve more thought than the database does. The embedding model determines what "similar" means for your corpus, and re-embedding is the expensive operation — changing models means reprocessing everything — so pin the model version, store it alongside each vector, and treat an upgrade as a planned migration with an eval run attached.

What I actually build

A RAG engagement with me is a pipeline, not a demo: ingestion and document-aware chunking for your specific formats, an embedding and indexing layer on pgvector or a managed store, hybrid retrieval with a re-ranking pass, a grounded generation step with citations back to source documents, refusal behavior when the corpus comes up empty, and the eval harness that lets you keep improving it after I am gone.

Where the answer needs to trigger an action rather than just be read — book the appointment, open the ticket, update the record — retrieval becomes one tool among several inside an agent. Where the corpus lives in Salesforce, NetSuite, a helpdesk, or a document store rather than in a folder someone can hand you, the ingestion side becomes an integration problem before it is a retrieval one, and that is usually where the real timeline lives.

The rate is $1,000 a day, flat. No retainer, no discovery phase you pay for twice. A working retrieval pipeline over a real corpus, with evals you can run yourself, is typically a small number of weeks rather than a quarter — and the eval harness is the part that makes it yours.

If you have a body of knowledge your team keeps re-reading by hand, that is the thing worth pointing this at first.