AI Rescue & Optimisation

Your AI proof of concept worked. Production is another story.

We diagnose AI implementations that have become unreliable, expensive, slow or difficult to scale — and tell you which layer the problem is actually in before anyone starts rebuilding it.

AI systems rarely fail loudly. They drift. The answers are mostly fine, so nobody raises it. The invoice creeps up. The response time slides past what the interface can tolerate. Someone finds a confidently wrong answer and trust starts to erode — and because there is no evaluation set, nobody can say whether it is getting better or worse.

This page is for systems that already exist. If you are still trying to get something into production for the first time, AI Product Engineering is the closer fit.

What people call us about

• Answers are inconsistent between runs
• The system states things that are simply not true
• Retrieval returns the wrong passages
• RAG works on the test corpus and not on the real one
• Monthly API cost has doubled with no change in usage
• Responses are too slow for the product they sit in
• Quality got worse after a provider updated their model
• It cannot handle the volume now going through it
• Nobody can tell whether last week's changes helped
• There is no logging, so failures cannot be reconstructed
• Security or legal has raised questions nobody can answer
• It works for one team and fails for another

What we examine

A diagnostic review works through the stack in order, because a fault in an early layer makes everything downstream look broken:

How a review runs

1. 1Understand the intent — What the system is supposed to do, for whom, and what “good” would look like. Surprisingly often this has never been written down.
2. 2Measure the present state — Build an evaluation set from your real data and establish a baseline for quality, latency and cost per request.
3. 3Work down the stack — Data, chunking, retrieval, prompts, model, workflow, guardrails, infrastructure — isolating which layer the failures come from.
4. 4Report with evidence — What is wrong, what it is costing, what we would change, in priority order, with effort estimates. You can act on it with us or without us.
5. 5Fix and re-measure — Each change scored against the same baseline, so improvement is demonstrated rather than asserted.
6. 6Leave it observable — Tracing, cost attribution and a regression suite, so the next change does not need another review.

What usually turns out to be wrong

• Retrieval, not the model. The right passage exists in the corpus and never reaches the prompt.
• Chunking that ignores document structure. Fixed-size splits that cut tables, clauses and headings in half.
• One prompt doing five jobs. Decomposing it usually improves quality and cost at the same time.
• Context bloat. Sending far more than the model needs, which raises cost and latency and often lowers accuracy.
• The largest model everywhere. Including on steps a small, fast model handles perfectly.
• No evaluation. So tuning is driven by whichever example someone looked at most recently.
• No guardrails. Nothing validates the output before it is shown or acted on.

We operate an AI-assisted product ourselves

We do not only build software for clients. Ausvanta is our own compliance and operations platform for Australian NDIS providers — rostering, GPS time tracking, incident management, claiming and audit evidence. We designed it, built it, and continue to operate it for paying customers in a regulated sector. That means releases, uptime, support load and production incidents are our problem too, not just a handover document. See Ausvanta.

Who we usually do this for

Australian software companies and mid-market organisations with an AI feature already in front of users or staff, and a technical team who need an independent read on why it is not behaving. We work alongside your engineers rather than around them — the goal is that your team can maintain and extend what we leave behind.

Where this sits in our services

• AI & Machine Learning — the retrieval, evaluation and model work itself
• Quality Assurance — regression testing, including for non-deterministic behaviour
• Cloud & DevOps — scaling, caching, monitoring and cost visibility
• Technology Consulting — if the question is whether to continue with the approach at all

Questions we get asked

Do we need to fine-tune our model?

Probably not, and we will not start from that assumption. Fine-tuning is expensive to set up, creates ongoing work every time the base model changes, and does not add knowledge — which is what most teams are actually trying to achieve when they reach for it. In the systems we review, poor output is far more often caused by retrieval returning the wrong passages, prompts that are ambiguous, data quality, or a workflow asking the model to do too much in one step. We establish which layer is failing first. Where fine-tuning genuinely is the answer — a consistent output format, a specialist classification task with good labelled data — we will say so, with the evidence.

Why are our answers inconsistent?

Usually one of four things. Retrieval is non-deterministic or returns different context for similar questions. The prompt leaves room for interpretation, so the model resolves the ambiguity differently each time. Temperature and sampling settings are higher than the task needs. Or the task itself is under-specified — the model is being asked to make a judgement you have not defined. All four are diagnosable, and none of them are fixed by changing model.

Our AI costs are out of control. Where does the money go?

Almost always context size. Long prompts, entire documents passed instead of relevant passages, conversation history resent on every turn, and retrieval configured to return far more than the model needs. After that: using a large model for steps a small one handles fine, no caching of repeated work, and retry logic that silently multiplies calls on failure. We measure cost per request before changing anything, because it is common for one endpoint to account for most of the bill.

Can you fix the latency without losing quality?

Often, yes. The usual gains come from running independent steps in parallel rather than in sequence, streaming the response so the interface feels immediate, using a smaller model for intermediate steps, caching retrieval results, and removing steps that are not earning their time. Where a genuine trade-off exists we will show you both options measured, rather than choosing for you.

How do you know whether a change actually helped?

We build an evaluation set from your real inputs and expected outputs, then score every change against it. Without that, AI tuning is anecdote — someone tries a new prompt, the next three answers look better, and the change ships. Most of the teams we review have no evaluation set, and creating one is usually the highest-value thing we do first.

Will you have to rebuild our system?

Rarely. Most of what we find is fixable in place: retrieval configuration, chunking strategy, prompt structure, model routing, caching, workflow decomposition and instrumentation. We recommend a rebuild only when the architecture cannot support what the system now needs to do, and we will show you the measurements that led to that conclusion.

Building something new rather than fixing something live? Start at AI Product Engineering →