We have a working store: a typed API and a React frontend. The last piece is the one that drew me to the project — a chatbot that answers questions about the store, and only the store.

This is Part 3 of 3:

• Part 1 — The backend
• Part 2 — The frontend
• Part 3 — The RAG chatbot (you’re here)

The problem with just asking an LLM

If you hand a raw language model the question “Do you have linen trousers and how much are they?”, it has no idea what’s in your catalog. It’ll either refuse or, worse, invent a confident answer. It also has no boundaries — ask it about the weather and it’ll happily oblige, which is not what you want from a shop assistant.

RAG — Retrieval-Augmented Generation — fixes both. The idea is two steps:

1. Retrieve the pieces of your own data most relevant to the question.
2. Generate an answer using only those pieces as context.

The model stops guessing and starts reading from your data. The trick is step 1: how do you find the “relevant pieces”?

Embeddings: turning text into searchable vectors

An embedding is a list of numbers (a vector) that captures the meaning of a piece of text. Texts about similar things land close together in that number-space; unrelated texts land far apart. So “find relevant text” becomes “find the nearest vectors” — a math problem a database can solve.

The neat part of this project is that the vectors live in the same Postgres database as everything else, thanks to the pgvector extension. The embeddings get their own column:

class DocChunk(Base):
    __tablename__ = "doc_chunks"
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    content: Mapped[str] = mapped_column(Text, nullable=False)
    source: Mapped[str] = mapped_column(String(50), default="")     # "product" | "faq"
    embedding: Mapped[list[float]] = mapped_column(Vector(1536), nullable=True)

The 1536 is not arbitrary — it’s the number of dimensions OpenAI’s text-embedding-3-small model produces. The column width and the model have to agree.

Building the knowledge base

Before the bot can answer anything, we turn the store’s data into embeddings. One chunk per product (its name, category, description, and price) plus a handful of hand-written FAQ entries for things the catalog can’t answer — sizing, shipping, returns:

def index() -> int:
    db = SessionLocal()
    db.query(DocChunk).delete()         # full rebuild — no stale chunks left behind

    products = db.query(Product).order_by(Product.id).all()
    texts = [_product_text(p) for p in products] + FAQS

    vectors = _embed(texts)             # one batched API call, not one per item
    # ...store each (text, vector) pair as a DocChunk...
    db.commit()

Two small efficiencies. The whole batch is embedded in a single API call — cheaper and faster than looping. And indexing is a full rebuild (delete then re-insert), so when a product changes or is removed, no orphaned chunks linger to mislead the bot. You re-run it whenever the catalog changes:

python -m app.rag

Answering a question

Now the live endpoint. When a question comes in, four things happen.

1. Embed the question — with the same model used for indexing. This matters: vectors are only comparable if they were produced by the same model, into the same space.

qvec = client.embeddings.create(model=EMBEDDING_MODEL, input=question).data[0].embedding

2. Retrieve the nearest chunks by cosine distance. pgvector adds a <=> operator for exactly this; SQLAlchemy exposes it as cosine_distance:

distance = DocChunk.embedding.cosine_distance(qvec)
rows = db.execute(
    select(DocChunk.content, distance.label("dist"))
    .order_by(distance)        # closest first
    .limit(TOP_K)              # keep the top 5
).all()

3. The guardrail — this is the part that keeps the bot honest. Cosine distance runs from 0 (identical meaning) to 2 (opposite). On-topic questions match a chunk closely; “what’s the weather?” doesn’t come close to anything in the store. So if even the best match is too far away, we decline instead of guessing:

if not rows or rows[0].dist > DISTANCE_THRESHOLD:   # 0.75, tuned by trial
    return ChatOut(reply="I'm the shop assistant, so I can only help with our "
                         "T-shirts, trousers, sizing, orders, and shipping.")

Retrieval isn’t just how the bot finds answers — it’s also the boundary that defines what it’s allowed to answer at all.

4. Generate, scoped to the retrieved context. The system prompt sets the rules, the retrieved chunks go in as context, and the question comes last:

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},                    # "answer ONLY from context…"
    {"role": "system", "content": "Context:\n" + "\n---\n".join(chunks)},
    {"role": "user", "content": question},
]
reply = client.chat.completions.create(model=CHAT_MODEL, messages=messages)\
    .choices[0].message.content

A deliberate detail: the retrieved context goes in a system message, separate from the user’s input. It makes it harder for someone to talk the bot out of its rules by typing “ignore your instructions” — the rules and the data aren’t mixed in with untrusted input.

The endpoint returns the reply and the chunks it used as sources, so you can always see what the answer was based on. For a system that’s supposed to be grounded, that transparency is worth the few extra bytes.

The order you run things

Because each layer feeds the next, first-time setup has a sequence:

# 1. enable the vector extension in Postgres (once)
psql -c 'CREATE EXTENSION IF NOT EXISTS vector;'
# 2. seed the catalog
python -m app.seed
# 3. build embeddings from that catalog
python -m app.rag
# 4. ask it something
curl -X POST localhost:8000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "Do you have linen trousers and how much are they?"}'

Skip step 1 and the embedding column can’t be created. Skip step 3 and the bot has nothing to retrieve, so it declines everything. The dependency chain is the whole architecture in miniature.

What I took away

The surprise was how little “AI” code this needed. The model is two API calls — one to embed, one to chat. The intelligence lives in the boring parts around it: keeping the embedding spaces aligned, a distance threshold to stay on-topic, and a prompt that’s strict about only using what was retrieved. RAG isn’t a model you train; it’s a retrieval problem with a language model on the end. Get the retrieval right and the rest mostly takes care of itself.

That wraps the series — a small, honest, end-to-end app: a database, an API, a UI, and an assistant that knows its limits.