Read summarized version with
About project
Working time:
Since 2025
Industry:
Healthcare technology
The service:
Agentic AI
Overview
The customer is a New York healthcare technology company whose voice agents call insurers and patients on behalf of outpatient clinics, handling eligibility checks, claim status and appointment follow-ups. As more clinics joined, the number of prompt, model and tool changes shipped each month grew with them.
The Challenge
The voice agents improved every week, but nobody could say whether a given change made them better. Releases were checked by listening to a handful of calls, and regressions were found by clinic staff reading call notes.
1. No Shared Definition of a Good Call
Engineers, operations and clinic success teams judged calls differently, so quality debates had no reference.
2. Manual Spot Checks
Each release was validated on a few hand-picked calls that rarely covered the cases that broke.
3. Silent Cost and Latency Drift
Model changes moved cost per call and response latency without anyone noticing until the monthly bill.
4. Low-Confidence Outcomes
Ambiguous call outcomes were written into clinic systems as if they were certain.
Technology & Approach
We built an evaluation and release discipline around the agents instead of around individual prompts: a labelled evaluation set from real calls, graders that score every release, and continuous scoring of live traffic.
Implementation
Phase 1: Evaluation Set and Quality Definition
Drew 1,200 de-identified call transcripts from production and past failures, agreed with operations what a correct outcome is for each call type, and labelled them.
Phase 2: Graders and Call-Outcome Classifier
Combined exact checks for structured fields, rule-based checks for required statements and model-based grading with Amazon Bedrock evaluations, and trained a call-outcome classifier that labels every call with a confidence score.
Phase 3: Release Gate in CI/CD
Every prompt, model or tool change runs the graded suite in the pipeline, and a release is blocked if any call type scores below the previous version.
Phase 4: Production Monitoring
A daily sample of live calls is scored, with Amazon CloudWatch alerts on quality, latency and cost per call, and low-confidence outcomes go to a review queue before they reach clinic systems.
Customization & Client Adaptation
The evaluation design followed how clinics actually use the agents, call type by call type.
Careful Data Handling
Transcripts used for evaluation are de-identified and kept in the customer’s AWS account, encrypted with AWS KMS, with access logged in AWS CloudTrail.
Thresholds per Call Type
Eligibility, claim status and scheduling calls each have their own pass thresholds, so a gain in one cannot hide a regression in another.
People Where It Matters
Outcomes below the confidence threshold go to clinic operations staff, and their corrections flow back into the evaluation set.
Cost as a Release Metric
Every release report shows cost and latency per call next to quality, so model choices are made with all three in view.
If you find this case interesting, we recommend taking a closer look at
The Results
Releases stopped being guesses. The team now ships prompt and model changes weekly instead of monthly, and no release since the gate went live has scored below the version before it.
Long-Term Value
- A shared definition of a good call, written as tests every release must pass.
- An evaluation set that grows each quarter from new failures and reviewer corrections.
- Model upgrades judged on quality, latency and cost together before they ship.
Business Impact
- Regressions are caught in the pipeline instead of by clinic staff.
- Low-confidence call outcomes reach a person before they reach clinic systems.
- Cost per call is tracked per release, with alerts when it drifts.
