AI
AI development, measured before it goes live.
Assistants, agents and automated workflows built on your own content. Retrieval that cites its sources, guardrails that survive a hostile user, and an evaluation set that gives you a number before launch instead of an opinion after it.
The gap
What changes between a demo and a system.
A demo has to work once, for somebody who wants it to work. A production system has to work for somebody trying to break it, at eight in the morning, at volume, while the model provider is having a bad day. Four things change, and all four are engineering problems rather than model problems.
The inputs stop being friendly
Real users paste in half a document, ask something out of scope, switch language mid-sentence, or try to talk the system into ignoring its instructions. A demo never sees any of that.
Being wrong starts to cost something
A confident wrong answer to a customer, a regulator or an operations team is a different object from a confident wrong answer in a meeting. The system needs to know when it does not know, and decline.
Someone becomes accountable for the output
Three weeks later somebody asks why it said that. Without logged inputs, retrieved sources and the version of everything involved, there is no answer to give.
The model underneath moves
Providers deprecate versions, change prices and adjust behaviour. If you cannot re-measure, you cannot safely upgrade, and you end up pinned to a model that is getting more expensive and less supported.
Scope
What the work covers.
Most engagements are one of these with a piece of another. We will tell you which parts your problem does not need.
Assistants
Question answering over your own material, with citations back to the source document and a refusal path when the material does not contain the answer.
Agents
Systems that use tools and take actions, with a written boundary between what the agent may decide alone and what it must hand to a person, plus the audit trail for both.
Process automation
Extraction, routing, classification and approval across work that was never designed to be machine-read. The exception paths get designed, because in most operations the exceptions are the process.
Retrieval engineering
The part that decides whether any of this works: how documents are split, what gets indexed, how a query is expanded, and how the right passage reaches the model when the obvious search misses it.
Model engineering
Selection against your task rather than a leaderboard, prompt and context design, fine-tuning only where it earns its cost, and inference economics worked out before the first invoice.
Integration
Putting the output where the work already happens, in the systems people already have open, rather than behind a new chat window nobody opens twice.
Evaluation
A number before the launch.
Most AI projects have no written definition of working, which is why they argue about whether they are finished. We build the definition first: a set of real questions with agreed answers, scored automatically, run on every change. It is the least glamorous part of the work and the one that decides whether the thing ships.
The questions come from your material
Real questions people actually ask, collected from support queues, email and the people who do the job. Questions invented by the engineering team measure the engineering team.
Someone who would know agrees the answers
A subject expert signs off what a correct answer looks like, including the cases where the correct answer is that the system should not answer at all.
Scored on more than resemblance
Correctness, whether the answer is grounded in a retrieved source, whether it refused when it should have, and whether the format is usable by whatever consumes it.
Run on every change
Model, prompt, chunking, retrieval, tools. Nothing ships that moves the number down, which also means an upgrade is a measurement rather than a leap of faith.
Handed over with the system
You keep the set and the score. If you replace us, you can still tell whether the next team made it better or worse.
Your data
Where your content goes, and where it does not.
This is usually the first question an enterprise asks, and it deserves specifics rather than a paragraph of reassurance. Every one of these is a decision made with you, written down, and visible in the configuration.
What the model sees
Only the passages a query actually needs, not the corpus. Narrowing that is a retrieval problem, and solving it well reduces both exposure and cost.
Training
Deployments are configured so your content is not used to train a third-party model, and we show you the setting rather than describing it.
Residency
Where inference runs and where the index lives. If either has to stay in a particular jurisdiction, that constrains the model choice, and it is better to know at design time than after a procurement review.
Retention
What is logged, for how long, and who can read it. Prompt and response logs are how you answer questions later, which also makes them something to govern.
Access
If a document is restricted in your systems, retrieval has to respect that. Permissions are enforced at query time rather than assumed at index time.
Deletion
Removing a document means removing it from the index and the caches as well as the store. An embedding built from deleted content is still derived from it, and the design has to account for that.
Delivery
How a project runs.
Five phases. The evaluation set is built in phase two, before the system it will judge.
Scoping
Which task, whose content, who the users are, and what working would have to mean for this to be worth doing.
Baseline
The evaluation set, agreed and scored against whatever happens today, even if today is a person reading documents. Without a baseline an improvement cannot be claimed.
Build
Retrieval, prompts, tools and guardrails, developed against the set from the first day rather than demonstrated at the end.
Harden
Adversarial testing, refusal behaviour, failure under load and when a provider degrades, and cost per query at the volume you expect.
Operate
Monitoring, drift, cost, and re-measurement whenever the model or the corpus moves. We stay on it, which is why the set is built to outlive the launch.
Questions
The ones worth asking first.
Will it make things up?
Any language model can produce a confident wrong answer, and anyone who tells you their system cannot is selling something. What can be engineered is how often, and what happens when it does: answers grounded in retrieved passages, citations you can open, and a refusal path for questions the material does not cover. Then you measure the rate rather than assert it, which is what the evaluation set is for.
Which model do you use?
Whichever one measures best on your task, and we build so it can be changed. Tying an application to one provider is how teams end up unable to move when pricing, availability or behaviour changes. Model choice is a decision inside the project, revisited when the numbers say so.
Do we need to fine-tune?
Usually not, and rarely first. Most problems that look like they need a fine-tuned model are retrieval problems: the right passage was never put in front of the model. Fine-tuning earns its cost for format, tone and narrow classification, and it is worth doing once there is a set to prove it helped.
Can it work in Arabic?
Yes, and it has to be built for it rather than translated into it. Tokenisation, chunking, query expansion and evaluation all behave differently in Arabic, and a retrieval pipeline tuned on English will quietly underperform on an Arabic corpus. The evaluation set is built in the language the users will actually type in, including when that is both.
What does it cost to run?
It depends on the task, the corpus and the volume, so a number quoted before any of those are known is a guess. What we can do early is model it: tokens per query, retrieval calls, cache behaviour and the expected volume, worked out in scoping so the running cost is a decision rather than a surprise on the first invoice.
What happens when the provider changes the model?
You re-run the set. That is the whole reason it exists. A version deprecation becomes a measured comparison and an upgrade decision, instead of a week of everyone arguing about whether the assistant feels worse.
Tell us what you are trying to build.
Describe the task and who has to trust the output. If the honest answer is that a model is the wrong tool for it, you will hear that on the call.
info@eorbitt.com · We reply within one business day.