RAG implementation product manager

Every RAG project has the same arc: a spectacular week-one demo on twenty hand-picked documents, followed by a slow slide into "why did it answer that?" as the real corpus arrives. The gap between demo RAG and production RAG is a hundred product decisions about retrieval, evaluation, and failure handling that someone has to own. I own them.

Decisions I've made with production consequences

I built WisOwl AI's retrieval and matching stack myself (embeddings, FAISS indexes, Supabase pgvector) for a system where retrieval quality directly decides whether a recruiter trusts the product enough to keep using it. 10,000+ jobseekers and 10+ recruitment-firm and startup customers have stress-tested those decisions. The transferable lessons:

• Chunking is a product decision. How you split documents encodes what questions the system can answer. Chunk boundaries that ignore document structure produce answers that are correct sentences from the wrong context, which is the failure that erodes trust fastest.
• Retrieval evaluation comes before generation evaluation. If the right passage isn't in the context window, the model never had a chance. I build the retrieval eval first (recall against a golden query set), because it localizes every future quality debate to the correct layer.
• Hybrid beats pure-vector more often than the hype admits. Exact identifiers, names, and codes need lexical search; concepts need embeddings. Most production systems I've touched ended up hybrid with a reranker.
• The "I don't know" path is a feature. A RAG product that answers confidently at low retrieval confidence is a liability machine. Thresholds, source citations, and graceful refusal get designed into the UX before the first embarrassing screenshot.

How a RAG implementation engagement works

Three ways in: a RAG audit (one to two weeks: retrieval metrics, chunking review, eval coverage, failure UX, ranked fixes), implementation PM (I run the build with your engineers from corpus strategy through launch), or rescue (the build shipped, users are complaining, and nobody can say which layer is failing; I instrument, localize, and re-sequence). All three start with the same 30-minute conversation.

Frequently asked questions

Our RAG answers are inconsistent. Model problem or retrieval problem?

Measure before switching models: in most systems I audit, the majority of bad answers trace to retrieval: the right passage never reached the context. A one-week retrieval eval against a golden query set settles the question cheaply.

Which vector database should we use?

The boring answer that's usually right: pgvector if you're already on Postgres and your scale allows it. Fewer moving parts, real transactions, one backup story. Dedicated engines earn their operational cost at serious scale or latency demands. I've run both FAISS and pgvector in production and will show you the decision math rather than a vendor preference.

How much does a RAG feature cost to run?

Modeled per query: embedding costs, retrieval compute, and generation tokens (the dominant term, driven by how much context you stuff). Caching, context pruning, and model routing routinely cut serving costs by half, so I treat them as part of the product design.

Do we need fine-tuning or is RAG enough?

For knowledge that changes, RAG; for style, format, or narrow-task behavior, consider fine-tuning; often the honest answer is better retrieval plus better prompting and no fine-tune at all. Run the eval first, and the question becomes an experiment instead of a debate.

Related pages