Expertise · Custom AI Models
The work has four parts: we measure candidate models on your data, choose between configuration, document search and fine-tuning, check the data usage rights, then keep consumption under budget per key. We have not trained any model in production so far, and we say that before you ask.
Already builtThe measurement and operation side is built and running, with results kept: an in-house test bench that compared eight speech recognition systems on the same 200 Romanian utterances, with confidence intervals and pairwise differences, plus a breakdown of voice latency into six segments that add up to the total with a deviation of under one millisecond. The operation side is an in-house gateway with 44 configured models, per-project keys with a model allowlist and budget, and a daily consolidation of consumption by user × key × model. The part that we did NOT do — the actual fine-tuning — is written as such: the pipeline is prepared and costed, but it has never run on GPU. The service remains “delivered” because the work we sell is informed selection and operation, not training.
Most “custom model” projects do not need a trained model. They need the right model, with the right instruction, over the right data, with a per-call cost that someone watches. The work starts with the question few people ask: what does success look like, measured how, on which set of cases? Without that answer, any model comparison is a matter of taste.
We measure ourselves, not the vendors’ figures. When we needed to know what best recognises spoken Romanian, we ran eight systems on the same 200 utterances, with the same normaliser, with bootstrap confidence intervals and pairwise differences. The results were uncomfortable: the system we used in production had a 26,26% word error rate, and a small 110 million parameter model, run on processor, had 6,83%. We also published the unpleasant part — on Romanian accuracy we lose by a wide margin to commercial vendors.
The second part of the work is choosing between three paths that are often confused: model configuration (instruction, tools, parameters), search in your documents, and fine-tuning. The order in which we try them is not a preference, it is economy: the first two are reversible in an afternoon, the third needs data, rights, GPU and a before-and-after measurement. We recommend fine-tuning only when we have proof that the first two are not enough.
The third part, the one that remains after launch: per-call cost. Models go through a proprietary gateway, where each key has a model allowlist, budget and period. Consumption is consolidated daily by user, key and model, which means the question “why did the bill go up this month” has an answer on rows, not on assumptions.
What it covers
A case set from your reality, a numeric criterion and a fixed normalisation method set down in writing. For speech we use word error rate with our own normaliser; for text responses, schema compliance plus a common-sense check plus similarity to a reference answer. Without this step, the rest of the work cannot be verified by anyone.
The same utterances, the same machine, one model loaded at a time, each in its own process. We report bootstrap confidence intervals and pairwise differences, not just the mean — because two models can have different means and still be statistically tied. We had exactly that case: two systems at 66 out of 66 pairwise comparisons, that is, a tie, although the mean table suggested a winner.
In a voice path we measured six segments separately — turn decision, speech recognition, the gate, time to the model’s first text, the first synthesis block, the queue and transport. The check that matters: the sum of the six gives the total time to first sound with a deviation under one millisecond on each valid turn. From there it is clear where the problem is: in the measured case, 68.7% of the time was waiting for the model’s first text, not speech recognition, which took 2.9%.
Document search is implemented and running: hybrid search, 70% semantic and 30% word match, similarity threshold 0,70, five fragments per question, with a per-query journal that keeps scores and timings. We propose fine-tuning only with proof that the first two are not enough — and with the budget and rights on the table before, not after.
It is not a formality; it is what stops projects. A corpus of 1,746 hours of Romanian that we had taken into account turned out to be licensed for non-commercial use — therefore unusable for a commercial model. A set of 270,946 examples had the code under a permissive licence, but the data with no licence declared at all. And a voice provider’s usage policy explicitly forbids training a model on its output. Each source is checked on its own page, not on what is written in an article.
Own gateway with 44 configured models, each with token cost and context limit in the configuration. The project key has a model allowlist, budget and period, and can be rotated while keeping the history. Consumption is consolidated daily by user × key × model. Routing uses the least busy strategy, two retries and a maximum time of 120 seconds.
Reasoning-type models produce internal steps that the cost-tracking platform does not see, but the provider bills them. The practical consequence: the gross budget set per key must be lower than the desired ceiling, divided by a reasoning factor. If you do not make this correction, the key looks within budget and the invoice is not.
We have a functioning local chain: speech recognition with a 0.6 billion parameter model on CPU and voice synthesis with a 99 million parameter model, also local. In a radio monitoring product, transcription is done locally, and the gateway version exists in parallel — so the comparison between local and hosted is made, not assumed. What leaves the machine is text, not audio.
A weekly regression suite on fixed cases, which compares schema compliance, passes a common-sense test and measures similarity with reference answers. A model that changes under you — and does change — is visible in the table, not in customer complaints.
What it looks like
The cases come from your reality, not from a general test set. The numeric criterion, the normalisation method and the threshold from which the result is acceptable are fixed. We deliver: the case set, the written criterion and the script that calculates it.
The candidate models run on identical data, with a confidence interval. We deliver: the table with all tested systems, including those rejected and the reason, plus the exact reproduction command. A model we rejected for Romanian had a 99.69% error rate because it slipped into Italian — Romanian was not among its declared languages. That result is in the table.
Configuration, document search or fine-tuning — with the reason, the cost and what is lost by choosing it. We deliver: the reasoned decision, plus the measurement that supports it. If the recommendation is "we do not train anything", we write it just as explicitly.
The project key is issued with budget, period and allowlist of permitted models, with the correction for reasoning tokens applied. We deliver: the key, the consumption dashboard and the threshold from which someone receives an alert.
The same cases, the same criterion, on the production model. We deliver: the before/after comparison and the list of changes with the reason for each. A recurring regression bank catches changes in the behaviour of the provider’s model.
The data
The questions anyone with a data protection officer would ask — asked here before they do.
A case
The situation
We had a speech recognition system running for Romanian and an open question: is it the best available, or just the first one we integrated? There was no in-house measurement, only figures published by providers, measured on other languages and other sets.
What we built
We built a bank: 200 Romanian utterances from a public set, the same seed for selection, 35.1 minutes of audio, the same normaliser for all systems, confidence interval by bootstrap, differences calculated pairwise. Eight systems, each loaded alone, in its own process, on the same machine. We wrote in the report both the machine configuration and the exact reproduction command.
What came out
The production system had a 26.26% word error rate. A 110 million parameter model, run on CPU with a language model alongside it, had 6.83%. A highly praised model came out at 99.69%, because it drifted into Italian — Romanian was not among its declared languages. And two systems that seemed different by mean turned out statistically tied in the pairwise comparison. We published the full table, including the row that contradicted our choice.
What the case does not say
The measurement was made on a public reading set, not on real phone calls, with noise and overlaps. A good result on this set does not guarantee the same result on the phone. That is why, for a client, the bank is rebuilt on its recordings — otherwise we measure something other than what matters to them.
Questions
No. No fine-tuned or trained model of ours is running in production, and there is no saved checkpoint in our repositories. What we do have: the training pipeline written down to the exact command for starting the GPU machine, the budget calculated for three scenarios, the data prepared and the licences verified — and a written list of what must be decided before the first run. We prefer to say that rather than sell a competence we have not exercised.
Because the work that delivers the result in most cases is not training. It is knowing which model does the job on your data, with what latency and at what cost per call — and that requires a measurement almost nobody does. One example from our work: a single wrong default value in a model configuration (a repetition penalty set to 1.2 instead of 1.0) cost 4.3 percentage points of error. Training was not needed, but measurement was.
Start with retrieval, almost always. It is reversible, it updates by changing a file, and it can show the source of the answer. Fine-tuning changes the model’s behaviour in ways you cannot inspect, requires labelled data, usage rights, and a measurement before and after. We recommend fine-tuning when we have proof that the first variant is not enough — for example, when we showed that an offline model does not become a streaming model just through configuration: by narrowing the attention window without retraining, the error rose from 8.81% to 22.26% and then to 48.95%.
No, and no honest provider can, because accuracy depends on your data. What we guarantee is the method: we measure on your cases, show the confidence interval, and say when the difference between two options is not statistically significant. We had a case where a result published by the authors of a model was not reproduced by us — they reported 20.70%, we measured 26.68% on the same model. We wrote that we do not know why, instead of choosing the convenient figure.
It depends on the route chosen, and it is a decision, not an accident. Through the gateway, the request reaches the model provider. If that is not acceptable, we have the local chain working: speech recognition and speech synthesis run on your machine, where only text leaves the machine, never audio. In a radio monitoring product, local transcription and gateway transcription are both implemented, so we can show the trade-off between them with figures.
It happens. That is why the work includes a regression bank on fixed cases, run periodically, which compares schema compliance, passes a common-sense test and measures similarity to reference answers. And that is why integration goes through the gateway: the model changes from one field, not by rewriting the code.
The project key has an allowlist of models, a budget and a period, and it can be rotated while keeping the history. Consumption is consolidated daily by user, key and model. One trap we correct from the start: reasoning models produce internal steps that the tracking system cannot see, but the provider bills them — so the gross budget on the key is set lower than the desired ceiling, divided by a reasoning factor. Anyone who does not make the correction sees the key within budget and the invoice above it.
Because “public” and “commercially usable” are different things, and the difference is discovered late and at high cost. Three examples from our checks: a large Romanian corpus, licensed non-commercial, therefore excluded for a commercial product; a set of hundreds of thousands of code examples under a permissive licence but with data that had no declared licence at all; and a voice provider’s usage policy, which explicitly forbids training a model on its output. The check is done on the source page, not on what an article repeats.
They do not train foundation models today and do not promise a proprietary model in production. They do not guarantee percentages of improvement. They do not compare a model with the competition based on the competition’s published figures — if we did not run the comparison ourselves, we say it is a figure stated by them, not one measured by us.
26 of them are code and files from our repositories. We don't publish their name or line: together, on a single page, they would describe too precisely how systems that aren't only ours are built. We go through them with you, in the repository, on request — verification stays possible, it just happens in a conversation.
Tell us about your process. Together we decide what is worth building, what we can connect, and how we check the result.