Scalize Systems
HomeServicesHow I WorkCase StudiesResources & TemplatesAboutContact
HomeServicesHow I WorkCase StudiesResources & TemplatesAboutContact
All case studies
Legal TechnologyLLM ExtensionHuman-in-the-Loop Evaluation

Expanding an AI Product into a New Market

Setting
Canadian legal technology startup (client anonymized)
My role
Built the process to extend the company’s LLM to UK legal data, including the human-in-the-loop evaluation process
Operating challenge
Replace a manual UK process with a systematic one that met UK legal standards while trust in AI legal tools was low
Outcomes
  • •Opened a new revenue stream
  • •Model-produced outputs reviewed through a defensible expert evaluation process

Human-in-the-loop evaluation

  1. Step 1

    Model output

    Summaries and UK court-specific document formats

    ↓
  2. Step 2

    Expert evaluation

    UK legal experts review outputs against market-specific standards

    ↓
  3. Step 3

    Documented correction

    Experts record where an output misses the standard

    ↓
  4. Step 4

    Feedback into retraining

    Correction signal becomes input to later model improvement

↺The next model iteration returns to expert evaluation.

The human-in-the-loop evaluation process described in this case study. Expert corrections were fed back as input to retraining, so manual review informed subsequent improvements instead of being repeated indefinitely.

The problem

A Canadian legal tech startup built a bespoke LLM trained on Canadian legal data. The model was producing strong results in its home market, and leadership wanted to expand into the UK. The challenge was that UK legal case judgments required a different approach: the model needed to ingest UK-specific inputs, extract the relevant information, and produce summaries and court-specific document formats that met the standards of a UK legal audience. The UK team had been doing this manually, and the goal was to replace that process with something systematic and autonomous that could scale. The work was also happening against a difficult backdrop. Lawyers across North America were being sanctioned for submitting AI-generated briefs with hallucinated citations, and trust in AI legal tools was at a low point. Any expansion into a new market had to be defensible, not just functional.

My contribution

I built the process to extend the model to UK legal data, defining the output types the new market required and the criteria the model needed to meet to produce them reliably. I designed and implemented a human-in-the-loop feedback process so that UK legal experts could evaluate outputs against market-specific standards and feed that signal back into retraining. The loop was structured so that expert corrections informed each round of retraining, rather than requiring the same manual fixes indefinitely.

The result

The expansion opened a new revenue stream. Outputs that had always required manual lawyer time were produced by the model at scale, reviewed through the evaluation process, and delivered with a defensible quality standard. The human-in-the-loop process became the trust infrastructure the market required, not an afterthought added once something went wrong.

Explore the Operating Diagnostic
Scalize Systems — Build, Scale, Optimize

© 2026 Scalize Systems LLC. All rights reserved.

ServicesHow I WorkCase StudiesResources & TemplatesAboutContact
katie@scalizesystems.comKatie Robblee on LinkedIn