Jeff Huber from Chroma put the problem bluntly: "Don't ship RAG. Ship retrieval." That sounds like a semantic fight until your product starts returning the wrong source, the prompt looks fine, and nobody can tell whether the failure came from the chunker, the embedder, the index, the reranker, or the LLM.
A RAG pipeline is not the product. Retrieval is the product when users pay you to find the right answer from the right source at the right time. If you cannot evaluate that pipeline, you do not own the most important part of your production AI system.
That is the line we use at appssemble. Rent retrieval when it is internal tooling and the corpus is small. Own retrieval when the answer quality, compliance posture, latency, or iteration speed decides whether the product works.
A RAG pipeline you cannot evaluate is rented confidence
The easy version of RAG is seductive: upload files, call an API, get citations back. Managed file search can parse, chunk, embed, index, and retrieve for you, which is exactly what you want when you are validating a workflow with no real traffic.
The problem starts when the first production failure shows up. End-to-end correctness tells you the answer was wrong, but it does not tell you which stage broke. Hamel Husain's split is the right one: the retriever and generator fail independently, so you evaluate them independently.
If the managed service hides chunk-level traces, embedding versions, retrieval candidates, reranker scores, and metadata filters, your team is debugging in the dark. You can change the prompt and hope. You cannot prove why recall dropped from 0.92 to 0.74 after a chunking change.
The failure is usually the chunker, not the model
Most teams spend too much time tuning the synthesis prompt because that is the surface they can see. Jason Liu's warning matches what we see in AI rescues: the answer layer gets blamed while the retrieval layer quietly feeds it the wrong context.
In most production RAG failures we see, the first problem is upstream: parsing, metadata, chunk boundaries, embedding versions, or ranking. The LLM is often just the part that makes the bad context visible. That should change how you budget engineering attention.
Chunking is not a vibe. Start with a measurable baseline: 512-token chunks with 10-20% overlap, metadata-aware context like title, section, date, and source URL, then test it against your real questions. For larger corpora, retrieve a broad 200-300 candidate set, rerank down to roughly 20, and assemble context from evidence you can inspect.
Imagine you are the CTO of a legal assistant with 60,000 contracts, clause libraries, and policy documents across five tenants. A generic upload-and-query setup may answer the demo question correctly, but it can still miss a jurisdiction filter, bury the relevant clause at position 15, and synthesize from the wrong version of the template. With owned retrieval, you can write a gold set for those exact failures and stop them from shipping twice.
Own the pipeline when retrieval is the product
We own the retrieval pipeline when RAG is not a side feature. A clinical lookup tool, legal assistant, finance support product, or customer-facing knowledge system cannot treat retrieval as somebody else's black box.
Ownership gives you four things that matter in production AI. 1. Eval speed: you can change the chunker, rerun 100-300 labeled examples, and know whether Recall@5 or MRR@3 moved. 2. Hybrid retrieval: you can combine BM25 or tsvector with dense embeddings and reciprocal rank fusion.
3. Metadata filtering: tenant, document type, date, language, jurisdiction, and permission filters live next to the vector search. 4. Compliance control: you can decide where embeddings live and how long source data stays there.
The quality lift is not theoretical. Anthropic's contextual retrieval benchmarks show contextual embeddings reducing retrieval failures by 35%, contextual embeddings plus BM25 reducing them by 49%, and adding reranking reducing them by 67%. These are benchmark results, not magic settings. The point is that every stage is measurable: contextualization, BM25, candidate count, reranker, and final context assembly.
This is also why our AI service treats retrieval evaluation as part of the product, not a QA step at the end. The eval set is versioned. The retrieval traces are inspectable. Your engineer can show which documents were retrieved, why they ranked, and which change made them disappear. The pipeline can be changed without asking a vendor to expose one more knob.
Rent it when retrieval is not the product
Owning the pipeline is not a religion. Renting wins when the retrieval problem is small, temporary, or not differentiating.
If you are pre-PMF, have fewer than 10,000 documents, serve one tenant, and only need "search our docs" inside an internal workflow, managed retrieval is a rational move. You ship faster, learn what users ask, and avoid building infrastructure before the product has earned it.
The same goes for teams without the bandwidth to maintain the system. The build-versus-buy research cited a rough breakpoint at 3 dedicated ML engineers. Below that, managed RAG often wins on time-to-value, especially when there is no residency requirement and no heavy PII.
The decision is not "build because engineers like control." The decision is cost per correct answer. Even without a long-context premium, sending 1M tokens for every question can be two orders of magnitude more expensive than sending a 5,000-10,000 token evidence pack, before latency and inspection enter the discussion. If the whole corpus fits inside a cheap managed workflow and quality is not the product, rent it and move on.
pgvector is often the right rescue stack
For Lovable, Bolt, and v0 rescues, our default is usually Postgres plus pgvector, often on Supabase or a Hetzner-hosted Postgres setup. The reason is boring: the app already has Postgres. Adding pgvector is one extension, not a new vendor, a new API key, a new failure domain, and a second source of truth.
The benchmark numbers in the research make the choice easier under the right constraints. In vendor-run pod-era benchmarks, Supabase reported pgvector HNSW delivering 1185% more QPS than Pinecone s1 at matching accuracy. Tiger Data reported pgvector plus pgvectorscale with 28x lower p95 latency, 16x higher throughput at 99% recall, and 75% lower monthly cost versus Pinecone s1.
Those numbers do not mean Pinecone is bad, and they do not prove Postgres wins every workload. Pinecone's current architecture is serverless, while those comparisons were against older pod-era tiers. Under 50M vectors, a Pinecone tier can be cheaper once you price in DevOps time.
The killer feature for Postgres in a rescue is not that Pinecone lacks filters. Pinecone supports metadata filters. The advantage is operational: tenant IDs, permissions, dates, source records, and vectors can live in one transactional system your team already operates.
For a rescue, fewer moving parts matter. A vibe-coded backend that already uses Supabase does not need Pinecone, LangChain, a mystery chunker, and three places to debug one answer. It needs raw SDK calls, Postgres, pgvector, clear ingestion code, and an eval suite that runs before changes merge.
LangChain is useful until it slows the rescue down
We do not treat LangChain as a villain. It is useful for prototypes, learning, and glue when the abstraction matches the job.
The problem appears when iteration speed becomes the job. In many rescues, the stack is Lovable backend, LangChain wrapper, Pinecone index, and OpenAI calls stitched together by prompts. The founder asks why answers are wrong, and the team first has to understand five abstraction layers before it can inspect one retrieval candidate.
LangChain v1 and LangGraph are more focused than the 2023-era framework many teams adopted. The question is still the same: does the abstraction expose the retrieval evidence, or does it hide the part you need to debug?
At that point, we usually remove the framework from the critical path. Most retrieval products need source parsing, chunking, embeddings, metadata filters, hybrid search, reranking, context assembly, and a model call. None of that requires a heavy abstraction when the priority is to see and test each step.
The rule is simple: keep the tool if it makes the pipeline easier to change. Remove it if the team spends more time translating requirements into framework concepts than improving retrieval.
Own the retriever, own the iteration speed
The AI Communication Platform we shipped runs real-time voice and video translation with sub-500ms latency across six platforms. That work reinforced the same principle we use for retrieval: when a system sits on the product's critical path, we own the evals, the traces, and the parts that decide whether it works. You can read more about that work in the AI Communication Platform case study.
For retrieval products, ownership means the chunker is yours, the metadata model is yours, the vector database choice is yours, the reranker is swappable, and the eval set runs on every meaningful change. That is vector database ownership in practice. Not architecture purity, just faster correction when the product is wrong.
If you are the founder, ask for one screen: the query, the top 20 chunks, their scores, the source IDs, and the eval case that catches the failure. If the team cannot show that, they do not own retrieval yet.
Managed RAG is fine when it gets you to the next learning milestone. It is a liability when your users are paying for retrieval quality and your team cannot explain why the right document did not show up.
Own the retriever. Own the eval. Own the iteration speed. Evaluating whether to build or buy your RAG pipeline? Book a call.