RAG for small teams: a knowledge base in two weeks

Most small teams do not have a knowledge problem. They have a finding problem. The answer to "what is our refund policy for annual plans" exists somewhere, in a Google Doc, a Notion page, a Slack thread from March, or the head of the one person who has been there longest. Retrieval-augmented generation, or RAG, is a good fit for this, and it does not require a data platform or a six-month roadmap. Here is the two-week plan we use for teams of five to fifty people, and the choices that decide whether anyone trusts the result.

Week one: sources, chunking and a baseline

The first two days are not engineering. We sit with the team and collect the 40 to 60 questions they actually ask each other, along with where the correct answer lives. This list becomes the evaluation set, and it also tells us which sources matter. In most engagements, three sources cover 80% of the answers, and the rest are noise that would make retrieval worse.

Days three to five are ingestion. Our default stack is intentionally plain:

• Connectors for the two or three systems that matter, usually a docs tool, a shared drive and a helpdesk, synced nightly with deletions honored.
• Chunking by structure, splitting on headings and list boundaries rather than fixed token counts, with the page title and section path prepended to every chunk.
• Hybrid search, combining keyword and vector retrieval, because product names, SKUs and policy codes are exactly the terms embeddings handle poorly.
• A Postgres database with a vector extension. For under a million chunks, a separate vector database adds operational cost without a measurable quality gain.

By Friday we run the question set and record a baseline. A first pass typically answers 55% to 65% of questions correctly. That number is useful precisely because it is unflattering.

Week two: quality, citations and access control

The second week is where trust gets built. We work through the failures from the baseline one category at a time. The usual culprits are predictable: outdated documents outranking current ones, answers split across two chunks, and questions that use internal slang the documents never mention.

The fixes are equally predictable:

1. Add a freshness signal to ranking and archive documents the team confirms are obsolete.
2. Retrieve neighboring chunks when a hit lands mid-section, so answers are not cut off.
3. Maintain a short synonym list the team can edit, mapping internal jargon to documented terms.
4. Add a reranking step on the top 30 results, which typically lifts accuracy by eight to twelve points on its own.

Every answer shows its sources as links, and the assistant is instructed to say it does not know when retrieval comes back weak. Teams forgive "I could not find that" far more readily than a confident wrong answer.

The feature that earns trust is not a better model. It is a citation the reader can click and verify in five seconds.

Access control is the other week-two task, and it is not optional. If a document is restricted to finance in the source system, it must be restricted in the index. We store permission groups on every chunk and filter at query time using the signed-in user's groups. Syncing permissions nightly is acceptable for most teams; for HR or legal content we sync on change.

What good looks like at the end

By the end of week two, a well-scoped project typically reaches 85% to 90% correct on the evaluation set, with the remainder mostly answered as "not found" rather than wrong. The assistant lives where the team already works, usually a Slack or Teams app plus a simple web page, and every answer has a thumbs-up and thumbs-down button that feeds a weekly review.

Running costs for a team of 30 are usually between $60 and $200 a month in model and hosting fees, depending on question volume. The more important cost is ownership. Someone on the team needs to spend 30 minutes a week reviewing downvoted answers and fixing the source document, not the prompt. When that habit sticks, quality keeps rising; when it does not, the assistant slowly drifts as documents go stale.

Where small teams go wrong

We see the same few mistakes repeatedly when teams try this on their own or inherit a stalled project:

• Indexing everything. More sources means more near-duplicate, outdated content competing for the top results. Start narrow and expand only when a question fails for lack of a source.
• Skipping the evaluation set. Without it, every change is judged by vibes, and a change that fixes one demo question silently breaks five others.
• Treating it as a chatbot project. The work is mostly information hygiene. Consolidating three conflicting refund policies into one page does more for accuracy than any prompt.
• Building agents too early. Once the knowledge base is trusted, it is a strong foundation for an agent that takes actions. Before that, adding tools just multiplies the ways things can go wrong.

That last point matters. The support agent we described in our write-up on resolving 62% of tickets was built on the same retrieval layer, and it only worked because the knowledge base had already been cleaned up and measured for months.

Is RAG the right tool for you

RAG is the right choice when answers exist in documents and change more often than you would want to retrain a model. It is the wrong choice when the answer requires live data from a system of record; there, an LLM integration that calls your API directly is simpler and more accurate. Many teams end up with both, and that is fine as long as each is measured on its own.

If you want the two-week version done for you, our RAG knowledge base engagement follows exactly the plan above and ends with a handover so your team owns the code, the index and the evaluation set.

Start with your 50 questions

The fastest way to know whether this will work for your team is to write down the questions people ask most and where the answers live. Send us that list through our contact form and we will come back within a day with a fixed-price quote and an honest read on the accuracy you can expect.