OAK (Output Audit & Knowledge): AI-Powered Error Resolution

Clinical data programmers use ROAK to generate R code for RAW-to-SDTM transformations, and that code can fail at runtime.

• Many errors need knowledge of ROAK's internals, so programmers often had to ask the core ROAK team.
• That caused delays, repeated support work, and a dependency on a few experts.

A curated rulebook of known error patterns and their resolutions, by ROAK version, plus the programmers' error logs (up to 50 per request).

LangSmith tracing and evaluation, feedback reports saved to S3, and the programmer's approval or rejection of each answer.

Retrieve, then verify:

1. An exact-match search against the rulebook in OpenSearch. A hit returns the known resolution with no LLM call.
2. Otherwise, an Azure OpenAI embedding drives a k-NN semantic search. Gemini 2.5 Flash then checks the top candidates against the error's structure (prefixes, function names, placeholder values).

The LLM never writes a fix: every resolution comes from the curated knowledge base, which keeps hallucination risk low.

• A Python orchestration layer on AWS Lambda behind Bedrock AgentCore Gateway.
• Parallel batches (ThreadPoolExecutor, up to 10 workers).
• ROAK-version filtering, and retries with exponential backoff.
• JWT-based user identification, DocumentDB audit logging, LangSmith tracing, S3 feedback reports.
• A human-in-the-loop approval step.

Status: Deployed internally

In production at ZS (internal) on AWS Lambda.

• Known errors are resolved immediately by deterministic matching, and the LLM is used only when meaning matters.
• Programmers depend less on the core ROAK team for recurring errors.
• The team's knowledge gets reused, traceability improves through audit logs, and debugging becomes self-service.

• OAK can only resolve errors the rulebook already knows. New error types still go to the core team, and the rulebook needs ongoing curation.