200+ AI Agents for Clinical Data Extraction:
From Pilot to HIPAA-Compliant Production

The Challenge

Our client — a mid-size pharmaceutical company running drug discovery and clinical trials — had 20 separate manual workflows for extracting structured data from research documents: trial reports, lab results, regulatory filings, and clinical notes. Twelve full-time employees were spending 60% of their working hours on copy-paste extraction tasks.

The pain was compounding: document volume was growing 40% year-over-year as the pipeline expanded, while headcount was frozen. The team was falling further behind each quarter. Manual errors in extraction were also creating downstream issues in regulatory submissions, requiring expensive re-review cycles.

The constraint: every solution had to be HIPAA-compliant, with full audit trail, no data leaving the AWS environment, and documented evidence for SOC2 Type II review. Their previous attempt at automation using a SaaS NLP vendor had stalled on compliance review for 11 months.

Our Approach

System Architecture

The full architecture runs entirely within the client's AWS VPC. No data egresses to external services. All model inference runs through Amazon Bedrock with VPC endpoints — no public internet traffic.

  ┌─────────────────────────────────────────────────────────────────┐
  │  DOCUMENT INGESTION                                             │
  │  S3 Bucket (encrypted, VPC-only) ──► Bedrock Knowledge Base    │
  └────────────────────────┬────────────────────────────────────────┘
                           │ Retrieval
  ┌────────────────────────▼────────────────────────────────────────┐
  │  ORCHESTRATOR AGENT (Claude 3.5 Sonnet via Bedrock)             │
  │  - Classifies document type                                     │
  │  - Routes to appropriate sub-agent                              │
  │  - Tracks extraction confidence                                 │
  └──┬──────────┬──────────┬──────────┬──────────┬─────────────────┘
     │          │          │          │          │
  ┌──▼──┐  ┌───▼───┐  ┌───▼───┐  ┌──▼───┐  ┌───▼──────────────┐
  │Doc  │  │Regex  │  │Clin-  │  │Valid-│  │Audit             │
  │Class│  │Extract│  │NLP    │  │ation │  │Logger            │
  │-ify │  │-or    │  │Agent  │  │Agent │  │(CloudTrail+DDB)  │
  └──┬──┘  └───┬───┘  └───┬───┘  └──┬───┘  └───────────────────┘
     └─────────┴──────────┴─────────┘
                           │
  ┌────────────────────────▼────────────────────────────────────────┐
  │  DynamoDB — Agent State + Extraction Results                    │
  └────────────────────────┬────────────────────────────────────────┘
                           │
  ┌────────────────────────▼────────────────────────────────────────┐
  │  REPORT GENERATOR AGENT                                         │
  │  - Assembles structured output                                  │
  │  - Confidence scoring                                           │
  │  - Routes <85% confidence to human review queue               │
  └────────────────────────┬────────────────────────────────────────┘
                           │
                   ┌───────▼──────────┐
                   │ S3 Output Bucket  │
                   │ (structured JSON) │
                   └──────────────────┘

  Step Functions Express Workflows orchestrate the entire pipeline.
  All agent-to-agent calls are synchronous invocations via Bedrock.
      

What We Built

• Bedrock Knowledge Base — 180K clinical documents ingested, chunked, and embedded using Amazon Titan Embeddings v2. OpenSearch Serverless as the vector store. Hybrid retrieval (semantic + keyword).
• 6-agent taxonomy — Document Classifier, Regex Extractor, Clinical NLP Agent, Validation Agent, Audit Logger, Report Generator. Each agent has a distinct system prompt, tool set, and temperature configuration.
• Step Functions orchestration — Express Workflows with parallel branches for sub-agents, Wait states for human review, automatic retry with exponential backoff on Bedrock throttling.
• Bedrock Guardrails — PII redaction (SSN, DOB, patient IDs), denied topics policy, content filters (medium sensitivity), custom word policies for regulatory terminology.
• CloudTrail audit log — Every agent invocation captured: timestamp, agent ID, input hash (SHA-256), output hash, confidence score, user who triggered, document ID. Immutable S3 log bucket with Object Lock.
• DynamoDB state store — Per-document extraction state, agent output storage, human review queue, confidence scoring history.
• Human-in-the-loop queue — SQS queue + Lambda notifier for outputs with confidence <85%. Reviewers see agent output + source document snippet side-by-side in a minimal internal UI.
• CloudWatch dashboards — Per-agent invocation count, latency, confidence score distribution, human review queue depth, daily cost per agent.

Compliance & Governance

HIPAA Controls

• All data remains within the client's AWS account and VPC — zero egress to third-party services
• Bedrock VPC endpoints — model inference never traverses public internet
• PII redaction via Bedrock Guardrails applied before any logging
• S3 Object Lock on audit log bucket — WORM compliance, immutable for 7 years
• AWS KMS customer-managed keys on all storage (S3, DynamoDB, CloudWatch)
• IAM least-privilege: each agent Lambda role has only the permissions it needs; no wildcard resource ARNs

SOC2 Evidence

• CloudTrail logs for all API calls — forwarded to the client's SIEM
• AWS Config rules for continuous compliance monitoring (encryption at rest, VPC isolation, public access blocks)
• GuardDuty enabled on the account for anomalous API activity detection
• Quarterly access reviews automated via IAM Access Analyzer

Results

Client Perspective

Related Resources