We build the corpus from which an assistant is allowed to answer, with rules about what goes in, how it is split, how it is searched, and what happens when the source changes.
An assistant that answers customers is only as good as the corpus it reads from. We handle the corpus: extraction from documents and the site, splitting into fragments, indexing, retrieval, the human approval gate and refresh. It includes the answer to the question of when semantic search is not worth it.
Already builtThere are proprietary implementations for each stage, but their status differs and must be stated plainly. What runs with proprietary vectors: the memory of an internal board used daily, with a `vector(1536)` column, an `ivfflat` index on cosine distance, a similarity threshold of 0.35 and a loop that re-indexes every 30 minutes only what has changed. What runs with vendor vectors: the knowledge base of the voice agents, synchronised to the vendor retrieval engine with 500-sized fragments and 100 overlap. What runs in production **without** vectors: the knowledge base of the text assistant platform — extraction, splitting, per-assistant configuration and hybrid retrieval are written and work, but the indexing path writes the vector column as `null`, so the hybrid score degrades to lexical, and the code says this itself through the `keyword-only` label. What is written and still not turned on in production: the proprietary `pgvector` pipeline of the voice platform, whose migration bears the header note “LOCAL ONLY — do not push to prod”. A service is sold with this map on the table, not without it.
An assistant that talks to clients does not "know" anything. At each question, someone first looks up a few text fragments and puts them in front of it, and it forms the answer from them. The quality of the answer is decided almost entirely in that retrieval stage, not in the choice of the model. That is why this service is about corpus and retrieval, not about the model.
The work has four parts that break independently of one another. Extraction: from which files and which pages we extract usable text, and what we do when we cannot. Splitting: how large the fragments are, how much they overlap, where a sentence is cut. Retrieval: how the fragments are chosen for a particular question, how many, and by what criterion. Refreshing: what happens when the source document changes. A system can be impeccable in three of them and useless because of the fourth.
There is one extra stage, which many skip and which changes the result completely: the approval gate. A corpus that grows by itself, from conversations, quickly ends up containing answers that no one has validated. The loop we built goes the other way: when the agent says "I do not have this information", the question is recorded with the status "pending", a person receives a notification and responds, and the answer moves to the status "approved" and only then enters the corpus. Knowledge grows from real gaps, with a person in the chain.
And the unpopular part: semantic search is not always the right choice. For a public in-house platform we deliberately refused it. The corpus is the platform’s own text, split into short passages that each carry the page they come from, marked through word overlap. No vectors, no vector store, no extra network request — and, more importantly, no source outside the repository. The assistant can only repeat what is already written on the site, and any answer is checked on a page written by a human. This is the entire anti-invention mechanism.
What it covers
The work, by component
We extract text and state clearly when we cannot
From plain text, markdown, logs, CSV, JSON, DOCX and PDF. DOCX is unpacked as an archive and the XML document inside is read; PDF goes through the standard extractor with layout preserved, and if that is missing there is a fallback that reads text operators directly from the file. The important case is when it does not work: a scanned PDF, which is an image, does not contain selectable text, and the system returns an explicit message — upload DOCX or text, or run the file through optical recognition first. We do not have optical recognition implemented and we do not pretend we do.
We split into fragments with overlap, cutting between sentences, not through them
The default values we use: 500 lexical units with 50 overlap for the voice pipeline, 1,200 characters with 150 overlap for the text one; on the voice supplier side, 500 with 100. The overlap exists because a sentence cut in half between two fragments becomes unusable for both. The sentence separator is written for the languages we work with: it recognises end of sentence followed by a Latin **or Cyrillic** capital letter, including Romanian diacritics. A separator written for English cuts poorly in Romanian and even worse in Russian.
We index with vectors where we have started it, and say where we have not
The default representation model is `text-embedding-3-small`, with 1,536 dimensions, requested through our model gateway, not directly from the supplier — this means the supplier can be changed without touching the knowledge base code. In the base, the column is of vector type with an `ivfflat` index on cosine distance. There is a fallback strategy written in the migration: if the vector extension is not available, the schema is created at any time with a trigram index on text, and retrieval switches to lexical similarity. It is not hypothetical — it was written precisely because rights to the extension are not guaranteed on every installation.
Hybrid retrieval, with the weights visible
The final score is 0.7 from vector similarity plus 0.3 from word matching, with the lexical part normalised. A set of candidates is scored and the top fragments are retained, and the context sent to the model cuts each fragment to a maximum length, so a single long document does not take up all the space. The configuration is per assistant, by default: retrieval on or off, maximum number of fragments per question (default 5, limited between 1 and 20), similarity threshold, search type — semantic, lexical or hybrid — and the weight of the lexical part.
Approval gate: knowledge grows from gaps, with a human in the loop
When the agent answers that it does not have the information, the question is written to a table with the status “pending”, marked with the channel it came from — voice or widget. A human receives the notification and replies directly to the message; a webhook takes over the reply, moves the entry to the “approved” status, and from there it enters the agent’s context. The possible statuses are four: pending, approved, ignored, duplicate — and duplicates are linked to the original question by a reference, so that ten phrasings of the same gap do not pile up. Delivery to the agent is done once per approved entry.
Refreshing: what happens when the source changes
On addition, the SHA-256 fingerprint of the content is calculated; if the text is identical to something already indexed, it is reused and a new vector representation is not paid for. When the document changes, its fragments are deleted in full and rewritten — complete replacement per source, not partial update, because a partial update leaves orphan fragments from the old version behind, and those are the ones that produce answers contradicted by the site. The source has a visible cycle of statuses: pending, processing, reprocessing, ready or failed, and reindexing can be requested specifically, per source.
We do not re-index what has not changed
In a loop that runs every thirty minutes over a list that only grows, recalculating the representation for an untouched element is a recurring payment for a vector that could not have moved. Our loop compares the time of the last touch with the time of the last indexing and skips what is unchanged. Representations are requested in batches, not one by one. These are cost details, but the monthly cost of a knowledge base is decided almost entirely here.
A connected catalogue is not an uploaded list
The difference is not in format, it is in time. An uploaded list is true at the moment of upload: price, stock and new products remain frozen until someone remembers to upload again — and usually it turns out that no one remembers, when a customer asks about a price from three months ago. A connected catalogue is read at the moment of the question, from the source that keeps the record. What we have built in this direction: an extractor that recognises product collections in JSON or CSV by the usual keys and, when the structure is unusual, asks a model with the obligation to answer strictly in a given form; and a crawler that reads the price from the live page of the shop. The best route remains the shop’s own structured interface, where it exists — but that is checked per shop, not assumed.
Crawling a site: map first, exploration second
The sitemap is sought first, in the usual naming variants, and nested maps are unfolded over two levels. Only if there is no sitemap does it move to breadth-first exploration, strictly on the same origin, with a configurable page limit (default 50, capped at 200). The strategy actually used is reported back — map or exploration — because from it one can read whether a poor result comes from the site or from the method. The site’s exclusion file is respected.
What it looks like
The path, step by step.
01
We establish what the assistant is allowed to know, before any import
The list of permitted sources and, equally important, the list of those rejected, with the reason. An internal purchase price file, a contacts export, an email archive — these are exactly the things that end up in the corpus “so it has context” and then become a reply to a client. We deliver the source inventory with a decision and reason for each, plus the rule about what must never be included.
02
We import, split and check what came out by reading the fragments themselves
Extraction, overlapping splitting, indexing. The verification stage is not skipped: random fragments are read from the resulting corpus. There appear tables turned into strings of joined words, scanned PDFs that produced zero text, and navigation menu pages brought in as standalone documents. We deliver the corpus plus the report on what could not be extracted and why.
03
We tune retrieval on real questions, not on easy questions
A set of questions is built from what customers actually ask, including badly written phrasings and those in two mixed languages, and the number of fragments, the threshold and the weight of the lexical part are adjusted. The aim is not for every question to receive an answer, but for questions with no coverage in the corpus to receive “I do not have this information”. We deliver the question set with the results and the final configuration.
04
We put in the approval gate and the growth loop
Recording unanswered questions, notification to the approver, the response path, moving into the corpus and linking duplicates. We deliver the functional loop and the written instruction for the approver — including what is not copied from the client's question.
05
We hand over the refresh procedure, with the name of the person who runs it
Which source is refreshed, how often, who presses it and how it is seen that it happened. Here we state the limitation openly: on the platforms we operate, reindexing is a requested action, not an automatically scheduled one. A procedure without a person's name against it is a procedure that is not carried out.
The path of a question through the system: approved sources enter extraction and splitting, the fragments reach the index; for the question, candidates are scored — vector and lexical — and only a few fragments reach the context; and when nothing is found, the question exits on the human approval branch and returns to the corpus.
The data
What we touch, where it lives, and how long it stays
The questions anyone with a data protection officer would ask — asked here before they do.
What goes into the corpus
Only what is explicitly approved: uploaded documents, selected pages, question-and-answer pairs, handwritten texts. The sources are typed — text, document, site, question-and-answer pairs — and have a declared purpose: products, documentation, frequently asked questions or others. Each source has its own switch, and the switch is checked at every retrieval query, not just in the interface — a switched-off source cannot reach the context even by mistake.
Where the fragments and their representations sit
In the database of that platform, alongside the rest of the client's data, not in a separate external search service. For PostgreSQL installations, in separate tables with the vector extension explicitly enabled; for the text assistant platform, in MySQL, with the representation in a JSON column and the distance calculation done in the application. The second model works at small scale and does not scale to hundreds of thousands of fragments — that is an architectural limitation, stated as such.
Personal data in conversations
The corpus should not contain personal data, but the learning loop can bring it in, because a client's question is text written by the client. That is why entry into the corpus goes through human approval: that is where filtering takes place. The operational rule that goes with this: whoever approves answers receives the explicit instruction not to copy into the corpus names, telephone numbers or order details from the original question.
What is kept about queries
There is a retrieval query log table, with the scores obtained, retrieval time and number of fragments returned. It is the tool used to answer “why did it say that” — without it, adjusting thresholds becomes guesswork. The log is kept as long as it is useful for tuning and falls under the platform's retention periods, not under a separate regime.
What does not exist and is stated as such
There is no application of an expiry term to documents: the field is declared in the schema, but nothing reads it, so a document does not disappear from the corpus on its own. There is no automated scheduled reindexing in the client platforms — refresh is requested explicitly, per source. There is no re-ranking with a dedicated model and no lexical scoring of the BM25 type; what we call “hybrid” is a weighted sum between word matching and vector distance, and that wording is the correct one.
A case
A knowledge base that grew from the questions the agent did not know how to answer
The situation
A restaurant chain with delivery had a voice agent on the phone and an assistant on the web page. The initial corpus had been uploaded once, at launch. The problem was not that the agent was wrong, but that it often said “I do not have this information” — to perfectly reasonable questions that no one had anticipated when the corpus was written. Each such answer was a lost conversation, and their list did not exist anywhere.
What we built
We turned the lack into a system input. When the agent answers that it does not have the information, the question is written to a table with the status “pending” and with the channel it came from — voice or widget. The responsible person on the client's side receives a notification on the messaging channel they already use, with the instruction to reply directly to the message. A webhook picks up the response, moves the entry to the “approved” status, and from there the answer enters the agent's context. Different phrasings of the same gap are linked to the original entry by a reference, and delivery to the agent is done once per entry, so an approved answer is not reinjected on every cycle.
What came out
The corpus ceased to be a file loaded at launch and became a list of real gaps, ordered by clients, not by our assumptions. Whoever approves sees exactly what was asked and in which channel, so they write the answer in the form in which it will be used. There are four states — pending, approved, ignored, duplicate — because “ignored” is a legitimate outcome: some questions should never receive an automatic answer.
What the case does not say
The loop works exactly as well as the person in it works. If nobody answers the notifications, the “pending” entries pile up and the agent continues not to know — the mechanism makes the lack visible, it does not solve it on its own. The second limitation, equally important: the implementation was wired for this client's flow, with the table, notification and injection written for it. As a generic function, available from the interface for any account, it does not yet exist.
Questions
What people ask us before they call
Why does the assistant answer incorrectly if the information is in the uploaded documents?
Almost always because the correct fragment was not retrieved, not because the model is “bad”. The frequent causes, in the order in which we check them: fragments that are too large, where the useful information is lost among other things; cutting through the middle of the sentence, without overlap; too few fragments brought into context; or a corpus in which the same information appears in three different versions and the older one scores better. The query log, with the scores obtained, shows which of them it is — without it, one guesses.
Do you really use semantic search everywhere?
No, and it is more honest to say exactly where. With our own vectors: the memory of an internal board used daily, on a vector column with a cosine distance index and a similarity threshold of 0.35. With supplier vectors: the knowledge base of the voice agents, synchronised to the supplier's retrieval engine. **Without** vectors, today, in production: the knowledge base of the text assistant platform — the indexing path writes an empty vector column, so retrieval runs lexically, and the code labels its own module as `keyword-only`. There is also a fully written in-house pipeline, with the migration marked “not to be published in production” until the rights on the vector extension are confirmed. That is the real map.
What happens when we modify an already indexed document?
The source fragments are deleted entirely and rewritten from the new version. We chose full replacement instead of partial updating because the hard part is not adding, but removing: fragments left over from the old version continue to be retrieved and produce answers that the client contradicts with the front page. On addition there is also a content fingerprint check — if the text is identical, nothing is recalculated and nothing is paid for in vain.
Does it update itself when the site or catalogue changes?
Not automatically, in the platforms we operate today. Reindexing is requested explicitly, on the source, through a dedicated action. There is a single exception, in an internal system of ours, where a loop runs every thirty minutes and re-indexes only what has changed. Automatic scheduling can be built and it is a separate stage; what we do not do is leave the impression that it already exists. There is also an expiry field declared in the schema that nothing reads — so a document does not leave the corpus on its own.
What is the difference between a connected catalogue and an uploaded product list?
The moment of truth. The uploaded list is correct on the day it is uploaded; from the next day, the price, stock and new products diverge silently, and the divergence is discovered when a client asks for a price that no longer exists. The connected catalogue is read at the moment of the question, from the system that keeps the records, so it cannot fall behind. The cost is different: a dependency on the availability of that system appears, and it must be decided what the assistant says when the source does not respond — the correct answer being “I cannot confirm the price now”, not the last known value.
Can you index scanned PDFs?
Not directly. A scanned PDF is an image in a PDF wrapper and does not contain selectable text; our extractor detects this and returns an explicit message instead of indexing an empty document — the worst possible result being a source marked “ready” that contains nothing. The solution is optical recognition before import, or providing the document in DOCX or text. We do not have optical recognition implemented in the current chain.
Does it work in Romanian and Russian?
Yes, and there is a detail that matters more than it seems. Fragmentation cuts between sentences, and our rule recognises the end of a sentence followed by a Latin capital letter **or** a Cyrillic one, including Romanian diacritics. A separator written for English does not recognise the start of a sentence in Cyrillic and produces stuck fragments, with a direct effect on retrieval quality. For the voice side, the representation model used by the supplier is multilingual.
When is semantic search NOT worth it?
When the corpus is small, stable and written by you — for example the content of your own site. In our own public platform we deliberately chose retrieval by word overlap over the platform's own text, split into short passages that each carry the source page. No vectors, no vector store, no extra network request and, crucially, no source outside the repository. The assistant can only repeat what the site already says, and each answer can be checked on a page written by a person. When the main goal is not to invent anything, that beats semantic search.
Are you keeping us captive in your platform?
The sources are yours and remain in their original form; the fragments and representations are stored in your installation’s database, not in an external search service to which only we have access. The representation model is requested through its own model gateway, so changing provider does not affect the knowledge base code. The vector representations, however, are tied to the model that produced them: changing the model requires reindexing the entire corpus. It is a computational operation, not a free one, and it is better known at the start than found out halfway through.
What the statements above rest on (29 sources)
29 of them are code and files from our repositories. We don't publish their name or line: together, on a single page, they would describe too precisely how systems that aren't only ours are built. We go through them with you, in the repository, on request — verification stays possible, it just happens in a conversation.