Human-in-the-loop evaluation
Step 1
Model output
Summaries and UK court-specific document formats
Step 2
Expert evaluation
UK legal experts review outputs against market-specific standards
Step 3
Documented correction
Experts record where an output misses the standard
Step 4
Feedback into retraining
Correction signal becomes input to later model improvement
The next model iteration returns to expert evaluation.
A Canadian legal tech startup built a bespoke LLM trained on Canadian legal data. The model was producing strong results in its home market, and leadership wanted to expand into the UK. The challenge was that UK legal case judgments required a different approach: the model needed to ingest UK-specific inputs, extract the relevant information, and produce summaries and court-specific document formats that met the standards of a UK legal audience. The UK team had been doing this manually, and the goal was to replace that process with something systematic and autonomous that could scale. The work was also happening against a difficult backdrop. Lawyers across North America were being sanctioned for submitting AI-generated briefs with hallucinated citations, and trust in AI legal tools was at a low point. Any expansion into a new market had to be defensible, not just functional.
I built the process to extend the model to UK legal data, defining the output types the new market required and the criteria the model needed to meet to produce them reliably. I designed and implemented a human-in-the-loop feedback process so that UK legal experts could evaluate outputs against market-specific standards and feed that signal back into retraining. The loop was structured so that expert corrections informed each round of retraining, rather than requiring the same manual fixes indefinitely.
The expansion opened a new revenue stream. Outputs that had always required manual lawyer time were produced by the model at scale, reviewed through the evaluation process, and delivered with a defensible quality standard. The human-in-the-loop process became the trust infrastructure the market required, not an afterthought added once something went wrong.