How Polaris search actually works

Chunking, the stage everyone skips

Retrieval quality is decided before a single vector exists. Split a document badly and no amount of ranking saves you, because the right answer is now half in one chunk and half in another.

Polaris splits on the document's own structure first. A Markdown file is parsed into sections by its heading hierarchy, and a section that fits in the budget stays whole, because a section is already the author's own answer to "what belongs together?". Only when a section overruns does the cascade kick in: split on paragraph boundaries, and if a single paragraph is still too long, on sentence boundaries. Never mid-sentence, and never mid-word.

• Budget: 450 tokens, roughly 1800 characters, tunable
• Overlap: 200 characters between adjacent chunks, so a definition that lands near a boundary appears on both sides of it
• Floor: anything under 50 characters is merged back into the previous chunk. A three-word fragment embeds to noise

Every chunk keeps its heading context, the full path down the heading tree such as Guide > Authentication. That string is not decoration. It is what tells you where an answer came from, and it earns its keep again in the ranking stage further down.

Embedding, on your CPU

The default model is nomic-embed-text-v1.5, about 137 MB, running through ONNX on the CPU. It is downloaded once and cached in a user-global directory, so a second project reuses it rather than fetching its own copy. Two alternatives ship: all-minilm-l6-v2 at ~23 MB when you want small, and mxbai-embed-large-v1 at ~670 MB when you want accurate.

Three details in that stage matter more than the model choice:

• Task prefixes, applied automatically. Nomic wants search_document: on stored text and search_query: on queries, and it degrades measurably without them. Getting this wrong is the single most common way a local RAG setup quietly underperforms, so Polaris does it for you and does not expose the choice
• Matryoshka truncation. Nomic's 768 dimensions are trained so the first N are independently meaningful, so Polaris keeps the first 512 and drops the rest. Two thirds of the storage and distance-computation cost for a quality difference you have to squint at
• L2 normalisation. Every vector is scaled to unit length on the way in, which makes cosine similarity identical to a dot product at query time. Free accuracy, cheaper arithmetic

One SQLite file, two indexes

Chunks live in an ordinary table. Two virtual tables sit on top of it. A sqlite-vec table holds the embeddings as float[512] with a cosine distance metric for KNN. An FTS5 table indexes the content and the heading context for BM25, configured to read its content from the chunks table rather than storing a second copy of your documentation.

The whole thing is one file next to your repo. No vector database to run, no service to keep alive, no schema spread over two systems that can drift apart. Deleting the index is rm polaris.db.

Query time: two searches, one ranking

A query is embedded once, then two independent retrievals run against the same chunks. Vector KNN finds what the query means. BM25 finds what the query says. Both pull a candidate pool three times larger than the requested top_k, because everything downstream needs room to reorder.

You want both, and the reason is the two questions that break each one. Ask "how do we handle expired sessions" and BM25 finds nothing useful, because the doc says "token lifetime". Ask for RRF_K and the vector side shrugs, because an exact identifier carries almost no semantic signal. Hybrid search is not a hedge, it is coverage of two distinct failure modes.

One deliberate asymmetry: if the full-text side errors or the FTS table is empty, Polaris treats it as an empty result list rather than a failed query. A malformed query degrades to vector-only search and still answers. It does not hand your agent an error.

Fusing two rankings that share no scale

Now you have two ranked lists whose scores mean nothing to each other. A cosine similarity of 0.82 and a BM25 score of 7.3 cannot be averaged, and normalising them per query just invents a relationship that is not there.

So Polaris throws the scores away and keeps the ranks. Reciprocal Rank Fusion gives each chunk 1 / (k + rank) in each list and adds the two terms, with k = 60. A chunk in both lists gets both terms and rises above one that only either side liked, which is exactly the behaviour you want: agreement between two different notions of relevance is the strongest signal available. The constant flattens the curve so rank 1 beats rank 2 without steamrolling it.

Then the heading context comes back. If query terms of three characters or more appear in a chunk's heading path, it gets a small additive bonus, scaled by the fraction of terms that matched. The default is 0.05 against raw fused scores in the 0.01 to 0.09 range, so it is a tiebreaker, not a thumb on the scale. A section literally titled "Authentication" should win a question about authentication, and before this boost it sometimes did not.

The last stage is about not repeating yourself

The top three chunks by score are frequently three neighbouring chunks of the same section, which is the worst possible use of an agent's context budget: three slots spent on one answer, and the caveat two sections away never shown.

Maximal Marginal Relevance picks results greedily, subtracting from each candidate its similarity to what has already been chosen. Polaris runs it at lambda 0.7, weighted towards relevance but willing to pass over a near-duplicate for something that adds information. Finally scores are normalised so the top hit reads 1.000 and everything else is relative to it, which is far easier to eyeball than a raw fused score.

Every constant is a config value

Nothing above is hard-coded. Drop a polaris.toml in the project root, or at ~/.config/polaris/ for every project, and override what you need. Values are validated before the model or the database is opened, so a bad setting is a clear error at startup rather than strange results an hour later.

max_chunk_tokens     = 450    # ~1800 chars per chunk
chunk_overlap_chars  = 200    # carried across chunk boundaries
model_id             = "nomic-embed-text-v1.5"
embedding_dim        = 512    # Matryoshka truncation from 768
rrf_k                = 60     # rank-fusion constant
heading_boost        = 0.05   # additive; 0.0 disables it
mmr_lambda           = 0.7    # 1.0 = pure relevance, 0.0 = pure diversity
mmr_candidate_multiplier = 3  # candidate pool = top_k x 3

Two of these are load-bearing: model_id and embedding_dim are recorded in the database, and changing either requires a full re-index. Polaris checks them when it opens the file and refuses with an explanation rather than silently comparing vectors from two different models.

What is not in the pipeline

Cross-encoder reranking is the obvious next stage, and we have a design for it. A cross-encoder scores the query and the chunk together rather than embedding them separately, which catches nuance a bi-encoder misses on ambiguous natural-language queries. It is specced, the model is chosen, and it is not shipped.

The reason is query-time compute. Every stage above is cheap enough to feel instant on a CPU, and a reranker is the first thing that puts real inference in the hot path of a search your agent runs dozens of times a session. It goes in when the quality gain is worth the latency on a laptop, and not before. We would rather ship a fast pipeline with a known ceiling than a slow one with a marginally better ranking.