RAG, with nothing hidden

Retrieval-augmented generation is usually a black box: a document goes in, an answer comes out, and the interesting middle — chunking, embeddings, similarity search, the context budget, the tool loop — is server-side and invisible. This page is my vllm-chatgpt backend ported to run entirely in your browser, with every step drawn on screen.

Two models load on demand. Embeddings use Xenova/all-MiniLM-L6-v2 (~30 MB) via transformers.js — the same model services/embeddings.py uses. Generation uses a small WebGPU model (~0.9 GB, desktop Chrome/Edge). The context-budget and retrieval panels work without the big model — that's where the ported logic lives.

The pipeline

Six steps, none of them magic.

In the original stack these are FastAPI routes over PostgreSQL + pgvector. Here each one runs client-side: pdf.js for ingestion, a JS array for the vector store, cosine similarity for search.

1 Ingest & chunk

documents.py reads the PDF and splits it into overlapping word windows.

chunk_size=800, overlap=200

2 Embed

Each chunk → a 384-dim vector with all-MiniLM-L6-v2.

embed_texts(chunks) → float32[384]

3 Store

Vectors go to pgvector in prod; here, a JS array persisted to IndexedDB.

db.chunks[] · {text, vec}

4 Retrieve

Embed the query, rank chunks by cosine similarity, keep the top-k.

cos(q, cₖ) → top-k

5 Fit the budget

context.py: sliding window + drop-oldest until under the token limit.

apply_context_window(msgs)

6 Generate / call tools

vllm.py: the model answers, or emits a tool call and loops on the result.

tool_calls → result → re-call

Ingest & retrieve

Ask a question; watch which chunks come back, and how sure the index is.

A small corpus about in-browser ML ships with the page (that's the committed corpus the search tool below uses too). Add your own by dropping in a PDF — it's parsed, chunked and embedded locally; nothing leaves the tab. Scores are raw cosine similarity, so the RAG step is visible, not implied.

or add a PDF top-k Idle — using a lexical fallback until the embedder loads.

Ask the corpus

The query is embedded and compared to every chunk. Bars are cosine similarity (0–1).

The generated answer will appear here. “Retrieve + answer” loads the WebGPU model (~0.9 GB) the first time.

The context budget

Every turn you add is a turn something else falls out.

This is services/context.py ported almost line-for-line — pure logic, so it runs with no model at all. System messages are always kept. Of the rest, only the last N survive the sliding window; if that still blows the token budget, the oldest are dropped one at a time until it fits. Push the budget down or add turns and watch messages fall out of context in real time.

Token budget 260 tokens Sliding window N 6 messages

What actually reaches the model

Kept messages are solid; struck-through & faded ones were dropped by compaction. Token counts are approximate (≈ 4 chars/token).

Show the ported logic (Python → JS)

The tool loop

The model doesn't know 19² — but it knows how to ask.

The backend's browsing tools need a server (and CORS would block them here), so instead I define two real local tools with JSON schemas: a calculator and a corpus_search over the committed corpus. The loop is the genuine article from vllm.py: the model emits a tool_call, we run the tool, append the result, and call again — until it answers without a tool.

Tool schemas

Exactly what the model is told it can call.

Run the loop

Needs the WebGPU model. Try a math question or “what is MiniLM's dimension?”.

Idle.