Human-in-the-loop evaluation
Step 1
Model output
Summaries and UK court-specific document formats
Step 2
Expert evaluation
UK legal experts review outputs against market-specific standards
Step 3
Documented correction
Experts record where an output misses the standard
Step 4
Feedback into retraining
Correction signal becomes input to later model improvement
The next model iteration returns to expert evaluation.
A Canadian legal tech startup built a bespoke LLM trained on Canadian legal data. The model was producing strong results in its home market, and leadership wanted to expand into the UK. The challenge was that UK legal case judgments required a different approach: the model needed to ingest UK-specific inputs, extract the relevant information, and produce summaries and court-specific document formats that met the standards of a UK legal audience. The UK team had been doing this manually, and the goal was to replace that process with something systematic and autonomous that could scale. The work was also happening against a difficult backdrop. Lawyers across North America were being sanctioned for submitting AI-generated briefs with hallucinated citations, and trust in AI legal tools was at a low point. Any expansion into a new market had to be defensible, not just functional.
I built the process to extend the model to UK legal data, defining the output types the new market required and the criteria the model needed to meet to produce them reliably. I designed and implemented a human-in-the-loop feedback process so that UK legal experts could evaluate outputs against market-specific standards and feed that signal back into retraining. The loop was structured so that expert corrections informed each round of retraining, rather than requiring the same manual fixes indefinitely.
The expansion opened a new revenue stream. Outputs that had always required manual lawyer time were produced by the model at scale, reviewed through the evaluation process, and delivered with a defensible quality standard. The human-in-the-loop process became the trust infrastructure the market required, not an afterthought added once something went wrong.
Before: one record per carton
Carton: Headboard
No link to the other cartons
Carton: Footboard
No link to the other cartons
Carton: Rails
No link to the other cartons
After: one connected product
Bed frame
One product record linked to every carton
Box 1 of 3: Headboard
Visual identifier on the packaging
Box 2 of 3: Footboard
Visual identifier on the packaging
Box 3 of 3: Rails
Visual identifier on the packaging
Complete shipment identified and validated before it ships
The supply chain industry operates on a 1:1 data model: one product, one shipment record, one carton, one identifier. That structure is the global standard, and it made it structurally impossible to coordinate multibox products at scale. Consider a bed frame: the headboard, footboard, and rails each ship in a separate box, each box could come from a different fulfillment center, and the rails might fit ten other beds by the same supplier while the headboard is sold separately on its own. Nothing on the outside of any box indicated what was inside or what it needed to arrive with. Individual cartons do not carry relationships to one another within the international supply chain, and every major retailer, including Amazon, Target, and Walmart, faced the same constraint. None of them had found a way around it. The result was the same across the industry: incomplete deliveries, customers unpacking what arrived only to find an entire box missing, and no systematic fix in sight.
I was the only person working on this from a technical capacity at Wayfair, and I built it from scratch. I redesigned the data architecture from a 1:1 model to a many-to-one structure, making it possible to associate multiple cartons with a single product and track each one individually through the supply chain. I worked with suppliers to define and document the carton relationships for every multibox product, building a process where each carton was assigned a visual identifier on the outside of the packaging so fulfillment center staff could verify at a glance that they had box one, box two, and box three before a shipment left the building. Getting there required coordinating across eight siloed divisions within Wayfair to update supplier-facing systems, internal fulfillment center infrastructure, and the scanning hardware on the warehouse floor. Suppliers needed to understand the problem, buy into the solution, and complete the carton mapping work for their product catalogs. After the first wave, the ongoing process became lightweight: only new products required updates.
The architecture made it possible to pick and validate complete multibox products before they shipped, eliminating the structural reason incomplete deliveries happened. Within the first year, supplier participation grew by 30%, customer lifetime value increased by 15%, and the program drove a $48M revenue surge. Over three years, the program increased revenue by $100M in total, including that first-year $48M. By year three, multibox shipping costs had dropped by 40%. Shipping cost reductions totaled more than $40M. What had been an unsolvable coordination problem across the entire industry became a solvable operational one, because the data model finally reflected how the products were actually built.
Before: brands spread across four legacy systems
Brand A
On one of 4 legacy systems
Brand B
On one of 4 legacy systems
Brand C
On one of 4 legacy systems
100+ brands
On one of 4 legacy systems
Routine change (new promotion or landing page)
MarketingEngineering queueLive
After: one shared platform
Brand A
Brand B
Brand C
100+ brands
Shared acquisition platform
Routine change
MarketingCMSLive
Endurance International Group grew through an acquisition model, owning more than 100 brands serving small and mid-sized businesses, including Constant Contact and Domain.com as well as a number of hosting platforms such as HostGator and Bluehost. Each acquired brand needed its own customer acquisition flow, but no shared platform existed to support them. Each of the 100+ brands ran on one of four legacy systems built independently over time, with no common architecture for promotions, domain search, or payments. Marketing depended on engineering for basic updates like a new promotion or a landing page change, which meant slow iteration across a portfolio that needed to move fast. The legacy systems also carried accumulated vulnerability surface from years of independent, uncoordinated builds.
I led the organization's first greenfield platform build: a white-label customer acquisition platform designed to serve all 100+ brands from shared infrastructure. My team was the first at the organization to move to the cloud, and I oversaw that migration on AWS, then built best practice playbooks from that work for cloud infrastructure and AWS use across the organization. I owned the product strategy and roadmap, defining and delivering A/B testing, promotions, semantic domain search, Payment-as-a-Service, and a bespoke CMS that gave marketing direct control over campaigns and landing pages without routing every change through engineering. Alongside the new build, I decomposed the legacy systems underneath it and rebuilt them on a microservice infrastructure, making the platform extensible to any brand the organization acquired going forward. I also created the company's first multi-location annual planning process, coordinating delivery across seven geographic locations.
The platform eliminated the dependency that had kept marketing waiting on engineering for routine changes, giving flagship brands direct control over their acquisition flows. Conversion rates improved 30% and CSAT gained 30% for those brands. Operating costs fell 25%. Replacing four different legacy systems, built on outdated monolithic architecture, meant engineering teams no longer had to maintain old code just to keep the lights on. And because my team released through CI/CD on AWS and monitored its own releases, system administrators no longer had to push code to bare-metal servers and oversee each launch. The annual planning process improved on-time delivery by 20% across all seven geographic locations. The platform became core infrastructure supporting a $1B annual revenue target across the brand portfolio.
Before
VSTO product
Headed for deprecation, no CI/CD, couldn't support the features law firms requested
What changed
Diagnosis and recommendation
Leadership gap and outdated architecture identified; modern rewrite recommended
Product management team
Product manager, product owner, and UX designer hired; voice of the customer program
Clearer workflow
Definitions of done, success criteria, and a move from Scrum to Kanban
New engineering leadership
Scalable, cloud-based microservice architecture
Results
The company built a litigation research and document drafting platform designed to bridge the gap between online legal research and document assembly. The original product was written in VSTO (Visual Studio Tools for Office), a Microsoft technology that was headed for deprecation. The codebase was cumbersome, and VSTO's implementation limits made continuous integration and continuous delivery (CI/CD) impossible. The team couldn't build the features law firms were requesting most, because the technology wouldn't support them. The architecture kept the product from being improved at all, kept the organization from expanding into new markets, and was incapable of scaling beyond its current state. Technical leadership hadn't kept pace with modern architecture or languages like React, so the organization stayed on outdated technology instead of building toward the future. The product's design and onboarding also meant customers took longer than they should to reach first value.
I diagnosed the core issues: technical leadership was lacking, and the architecture was outdated and couldn't scale. Drawing on my experience, I recommended rewriting the product in a modern language on a scalable architecture that would support practices like A/B testing and a more modern approach to software development. I also coached the CEO on product development best practices and on using customer data to plan new features and functionality. I created a voice of the customer program so the team spent its time on features that were both marketable to other firms and responsive to current customers' needs. I oversaw the creation of a product management team, hiring a product manager, a product owner, and a UX designer. I redesigned how the team worked. I introduced definitions of done and business success criteria so everyone understood the objective before work began, which made planning more efficient. Because success and testing criteria were defined before development started, tickets were completed faster and there were fewer rounds of back-and-forth between QA and developers. I also moved the team from Scrum to Kanban, so it could move with more agility and respond continuously as customer feedback updated priorities.
The company brought in new engineering leadership, and together we built a more efficient, scalable, cloud-based microservice architecture. The full product rewrite was delivered on time and $150K under budget. The team reached full CI/CD, and release frequency improved 30x. Through improved design and onboarding, customers achieved first value 50% faster than with the previous product. The new architecture also allowed the organization to expand into a new market.