What changed

Across the team, engineers reported getting back roughly 15 hours a month each that had previously gone to investigation overhead — time that went back into feature work and into reviewing the agent's draft PRs, which took meaningfully less time than writing the fix from scratch. About 40% of incoming bugs ended up with a usable draft PR attached before a human even looked at the ticket. The team's median time from "bug filed" to "fix merged" dropped accordingly, not because the agent replaced engineering judgment, but because it removed the part of the work that was mostly about finding things, not deciding things.

What made it hard

Early on, the agent opened a draft PR for almost every bug, including several it hadn't actually reproduced — those drafts were mostly noise, and a couple of engineers started auto-closing them without reading. We tightened the policy so a PR only opens once reproduction confidence crosses a threshold; below that, it posts the investigation summary and stops. That change is most of the gap between the ~40% draft-PR rate and the much larger share of tickets that got a useful investigation summary.

What changed

Time spent reconstructing deal and portfolio history before meetings dropped by roughly 80% — a half-day of digging became a sourced answer checked in minutes, and quarterly portfolio reviews became working sessions instead of data-gathering exercises. New associate ramp-up on the existing portfolio dropped from several weeks to under a week, and on at least two occasions the agent surfaced a prior pass memo on a company that resurfaced with a new round — including a reminder of a concern that, in one case, turned out to still be live.

What made it hard

The hardest part wasn't retrieval — it was conflicts. A founder's deck would say one ARR number, the fund's own tracking spreadsheet would say another, and an old email thread would have a third, all 'correct' at different points in time. Early answers just picked whichever source had been modified most recently, which was sometimes the stale one. We ended up having the agent surface all three with dates and let the partner pick, rather than try to silently resolve it — slower, but it's the version the team actually trusts.

What changed

Live answer rate during peak windows went from roughly 35% to about 95% — nearly every call is now answered immediately, including after-hours and weekend calls that previously went to voicemail. FNOL intake that used to wait until the next business day now happens at the moment of the call, routine policy and coverage questions are resolved without staff involvement, and appointment scheduling that took two or three days of phone tag now happens on the first call. Every call being transcribed has also given the team a searchable record of what policyholders actually call about — which already surfaced a recurring coverage question now addressed in renewal materials.

What made it hard

The hardest part was the handoff line, not the call-answering itself. In the first week, the agent tried to talk a clearly upset policyholder through a coverage dispute instead of bringing in a person right away — technically correct information, wrong moment for it. We spent the next three weeks listening to recordings and pulling the escalation trigger earlier: any sign of frustration, ambiguity, or a request for 'someone,' and it hands off immediately, even mid-sentence.

What changed

Every one of the roughly 40,000 agent actions per month across the three systems is now logged, permissioned, and attributable — the governance layer's audit log became the primary evidence in the group's next compliance review, closing what had been an open question on the prior one. Scoping the patient-Q&A assistant's access down also stopped it from occasionally surfacing clinical detail staff didn't need. Under the billing assistant's two-tier model, about 70% of its write actions are low-risk reschedules that execute directly and get logged, while billing-code and cross-provider changes always route to a staff approval queue — which caught two proposed billing-code changes a reviewer corrected before they applied — and the patient messaging review queue gave the clinics their first real visibility into how much staff actually edit the agent's drafts.

What made it hard

Migrating the patient-Q&A assistant's access was the trickiest piece, because scoping it down broke a few workflows the front-desk team had quietly come to depend on — staff who'd gotten used to asking it things like medication history would suddenly get 'I don't have access to that.' We spent a week going back through real questions with the front-desk team to sort which of those needed a different, narrower field added to the scoped set versus which ones genuinely shouldn't have been answerable by the assistant in the first place.

What changed

Within a month, the agent was handling roughly 25 requests a week directly in Slack — quick lookups now resolve in under a few minutes instead of minutes-to-days, and background tasks like draft PRs or doc edits show up in-thread within hours instead of 'whenever someone gets to it.' The bigger shift was behavioral: because asking stopped costing anyone else their time, people started raising the small things they'd previously just lived with, and the visible thread history has already prompted two recurring requests to get fixed at the source instead of handled over and over.

What made it hard

The early failure mode wasn't capability, it was tone and scope. In week one, the agent would sometimes answer a vague request ('can someone look at the signup numbers?') by guessing what was wanted and posting a full report nobody asked for — technically responsive, but it cluttered the thread and occasionally answered the wrong question. We added a quick clarifying-question step for ambiguous requests, which slowed down the fastest lookups slightly but cut down on 'that's not what I meant' replies a lot. The other early issue was ownership confusion — a couple of times, someone assumed the agent had handled a request when it had actually declined it as out of scope. Declines are now posted explicitly, with a reason, rather than silently skipped.

What changed

The agent now picks up every alert the moment it fires, pulls account history, KYC profile, related-party data, and prior alerts across the bank's core systems, and assembles an investigation packet with a recommended disposition — clear, escalate, or file a SAR — along with the specific data points behind that recommendation. About half of all alerts are now low-risk enough that an analyst can confirm the agent's 'clear' recommendation in a couple of minutes rather than rebuilding the case from scratch, which cut the alert backlog from roughly 600 to about 120 within two months. The other half — the genuinely ambiguous or higher-risk alerts — now get the analyst's full attention from the start, because the agent has already done the data-gathering, and the team filed its first SAR sourced from an alert that would previously have sat in the backlog for over a week.

What made it hard

The early version of the agent leaned heavily on transaction velocity and amount thresholds to score risk, which meant it routinely flagged 'escalate' on accounts that were simply high-volume by nature — small business merchant accounts processing dozens of transactions a day as a matter of course. Analysts quickly learned to ignore 'escalate' on those account types, which defeated the point. We rebuilt the scoring to compare each account against its own historical baseline and its account-type peer group rather than a single global threshold — about three weeks of analyst feedback to get right, but it's most of why the agreement rate between the agent's recommendation and the analyst's final disposition climbed from roughly 55% to over 85%.

What changed

The agent now reviews every inbound NDA and vendor contract against the legal playbook the moment it arrives. When a document matches the playbook's approved positions, the agent clears it and notifies the requester directly — no attorney involved. When something doesn't match, the agent drafts redlines against the specific deviating clauses and routes the document to an attorney with those deviations flagged, so review starts at 'here's what's different' instead of 'read the whole thing.' About 70% of inbound NDAs now clear the same day, most within a couple of hours, and the attorneys' time on routine document review dropped from roughly 12 hours a week to about 3 — time that went back into the contracts that actually need negotiation. Sales and procurement noticed the change before legal did: NDA turnaround stopped being something they had to plan around.

What made it hard

The hardest part was teaching the agent the difference between a clause that was worded differently but meant the same thing, and one that was actually a substantive deviation. Early on, it flagged a lot of documents as 'off-playbook' simply because a counterparty had rephrased a standard confidentiality term or used a different definition that amounted to the same scope — technically different text, same legal effect. That drove the off-playbook rate to nearly half of all documents in week one, barely better than the old process. We spent about two weeks going through flagged documents with the attorneys, building out a library of 'equivalent phrasing' for the playbook's core clauses, before the in-playbook clearance rate stabilized at around 70%.