HealthSteward — Technical Design Document
A point-in-time architecture snapshot, written up for anyone who wants to go deep on how it's built. Re-written on architectural shifts, not on every change; see the note at the bottom of this page.
Problem & motivation
Patients managing fragmented care — multiple specialists, no shared record system — carry an unpaid coordination job: remembering what changed since the last visit, which labs are pending, what to raise with which doctor. This falls hardest on people with the least capacity to carry it (mid-flare, mid-crisis), and nobody on the clinical side owns it.
Most tools treat this as a records problem: store the documents, make them searchable, maybe chat with them. HealthSteward treats it as a coordination problem instead — ingest documents providers already give you, track what's changed and what's still open, and turn that into something concrete for the next visit. Locally, by default, because a tool holding this much health history shouldn't require sending it to a server to be useful.
What it does
Health profile management
Conditions (ICD-10 coded), medications, doctors, appointments in one place.
AI visit preparation
An agentic loop drafts personalized questions for an upcoming visit — pulling relevant history and calling tools before finalizing. Runs fully local via Ollama by default; Claude API and other OpenAI-compatible providers are opt-in.
After-visit summary parsing
Upload the PDF from a visit. Parsed locally, reviewed by you, then applied to your profile.
Proactive action items
Follow-ups to book, labs to get done, referrals to schedule — surfaced at the moment of engagement and again until resolved.
Goals
- Generate genuinely useful, specialty-relevant visit-prep questions from the patient's own data
- Keep health data local by default; anonymize anything that must leave the machine
- Turn parsed AVS data into closed-loop action, not just storage
- Degrade gracefully — a failing LLM call should never block the user from getting something useful
Explicit non-goals
- Not a clinical decision-support tool. Generated questions are prompts for a conversation with a real clinician — not diagnostic or treatment guidance. No automated clinical-safety validation exists or is planned; human review is the only safety mechanism today.
- Not multi-user yet. Family sharing is deferred pending a decision, not built toward.
- Not HIPAA-scoped. Personal/family use, not a covered entity.
- No A/B testing, canary rollout, or drift-monitoring infrastructure. Single-user local app, no population to canary against — a deliberate scope cut, not an oversight.
Tech stack
FastAPI + SQLAlchemy (async) + SQLite, migrations via Alembic
React 19 + TypeScript + Tailwind CSS + Vite
Pluggable backend — Ollama (llama3.2 by default) locally, or Claude API (Sonnet), or any OpenAI-compatible provider — for visit prep's tool-use loop, switchable at runtime from Settings. llama3.2 specifically for reliable multi-turn tool-calling, since this loop is conversational.
Ollama for two different local jobs, on two different models: qwen2.5:7b for AVS PDF parsing (single-shot text→JSON extraction, where Qwen2.5 at this size is generally considered strong at structured output) — and context-selection's Stage 2 relevance scoring, which reuses whichever model ollama_model is set to (llama3.2 by default, the same model the agentic loop uses), not a dedicated scoring model. Both fit the same 8GB-RAM/4-bit-quantized ceiling DEC-009 established. The AVS parser's model choice isn't benchmarked — no documented comparison against alternatives (issue #59).
System Design
Two LLMs, two trust boundaries. Almost everything runs inside the dashed local boundary below; the only thing outside it is an optional external LLM call, made only after anonymization.
Data flow
Only if LLM_PROVIDER=claude or custom — anonymized context and tool results only. Response returns to Orchestrate; Serve continues either way.
Two trust boundaries
Ollama, local
PDF parsing always runs on a local model, and visit prep's agentic loop does too by default (DEC-016). Raw records never touch the network — enforced by a hard localhost-only safety check in the Ollama client, not just convention.
Pluggable backend (Claude or any custom provider)
Only ever receives already-anonymized data, when opted into. Anonymization happens before the agentic loop starts, and every tool result fed back into the loop is anonymized the same way, regardless of backend.
Reliability design
The agentic tool-use loop is fallback-not-hard-failure by construction, three layers deep: if it can't converge within agent_max_turns (default 6) or a backend produces a malformed tool call, it falls back to the pre-existing single-shot prompt-in/JSON-out call; if that fails too — or anything else in prepare_visit() raises, LLM-related or not — an outer catch-all returns a fixed set of generic questions rather than an error. That hardcoded fallback, not the single-shot call, is the real floor: no functional regression is possible by design, only degraded quality on failure, and prepare_visit() never raises to its caller.
Deep Dives
Expand any section — these are written to be answerable as "walk me through how X works" in a technical conversation.
AVS PDF parsing +
Section-routing architecture in src/parsers/: rather than sending the whole document to an LLM, the text is split into named sections and each one is handled by whichever strategy actually fits it — deterministic regex where the shape of the data is predictable (a medication list, a header line), a focused local-LLM call only where it isn't (freeform vitals prose, referral mentions).
SectionSplitter (src/parsers/agent/section_splitter.py) is what turns raw PDF text into the named chunks the router hands off — a table of regexes (SECTION_HEADERS) matched against the text to find where each section starts, covering two structurally different export formats side by side:
| Format | Example header pattern | Section key |
|---|---|---|
| Sutter Health AVS | Today'?s\s+Visit | todays_visit |
| Sutter Health AVS | (?:Tests\s+ordered|New\s+orders\s+ordered|Lab\s+order) | lab_orders |
| Tebra clinical note | ^Assessment$ | assessment |
| Tebra clinical note | ^Vitals\b | vitals_section |
- Order matters —
SECTION_HEADERSis a list, not a dict; earlier patterns are matched first when two headers could plausibly overlap on section boundaries. - Truncated to
MAX_SECTION_CHARS = 2000chars per section — this, not just the section-routing split itself, is what keeps each LLM call small and focused rather than re-feeding a bloated chunk of the document. - Cached after first access —
.get()/.all_sections()only run the regex table once per document, even though multiple router stages (e.g.vitalspulls fromtodays_visit,vitals_section, andphysical_exam) request overlapping sections. - A section can feed more than one router stage, and a stage can pull from more than one section — the split and the routing are deliberately decoupled layers;
SectionSplitterdoesn't know which stage strategy is deterministic vs. LLM, it just answers "give me the text under header X."
The router's strategy per section is a fixed table, decided up front by a human reading real Sutter Health and Tebra export samples — not inferred at runtime:
| Section | Strategy | Why |
|---|---|---|
| patient_provider | deterministic | Header line has a fixed shape: date, facility, phone. |
| medication_changes | deterministic | Structured med-change section, cross-referenced against the medication list. |
| follow_up | deterministic | Regex for "recheck/follow-up in N weeks/months", target date computed from the visit date. |
| upcoming_appointments | deterministic | Semi-tabular block keyed off month abbreviations, walked line by line. |
| diagnoses | deterministic → llm | Regex looks for an Assessment section formatted as Condition (ICD-code); only calls the LLM if that section is missing — the one runtime (not format-time) branch in the pipeline. |
| vitals | llm | Buried in freeform "Today's Visit" / physical-exam prose, no fixed shape. |
| lab_orders | llm | Same — unstructured "tests ordered" narrative text. |
| notes | hybrid | Deterministic extraction first, LLM supplements, results de-duped by word-overlap before merging. |
| referrals | llm | Referral mentions are scattered through instructions/plan/impression text. |
extract_diagnoses() regex-matches Condition (ICD-code) pairs. No LLM call for this section at all.
Regex returns None (not []) — the router reads that as "use the LLM" and routes to the diagnoses llm config instead.
Each llm/hybrid section only receives the specific chunk(s) of text it needs (from SectionSplitter, capped at 2000 chars each) — never the full document — via a focused system prompt (src/parsers/agent/prompts.py) and a single direct_chat() call (src/parsers/agent/ollama_chat.py).
- Local-only, enforced not just documented —
ollama_chat.pyhas a safety check that blocks any non-localhost URL, so a misconfiguration can't silently route PHI externally. - Two document formats, one pipeline — the deterministic regexes carry parallel fallback chains for Sutter Health AVS exports and Tebra clinical-note exports, since the two look structurally different (e.g. header parsing tries the Sutter header-line pattern first, then falls back to Tebra's "Visited on:" / "MRN :" patterns).
- Review before apply — parsed items are always presented for user confirmation with per-section checkboxes; nothing is auto-applied to the profile.
- Deduplication — diagnoses dedupe by name on apply (existing conditions get ICD-10 backfilled if missing); medication stops match by name.
4-stage context selection +
Before generating visit-prep questions, relevant past-visit history is selected via a 4-stage pipeline in src/utils/context_selection.py:
More than 5 visits remain after stage 1, and Ollama is available. Each is scored 1–10; ≥7 survives, plus the pinned visit regardless of score.
≤5 visits remain, or Ollama is unavailable — stage 3 packs by recency instead of score, since none exists.
- Rules-based filter (instant, free) — always include same-doctor's last visit and PCP/Internal Medicine visits; apply a specialty-relevance mapping (e.g. Endocrinology ↔ Cardiology, Nephrology); exclude doctors flagged out of prep context. The same-doctor visit is pinned — its id is tracked through stages 2 and 3 so it can't be capped, score-filtered, or budget-dropped later, unlike every other candidate.
- Local LLM relevance scoring (Ollama) — only runs if more than 5 visits remain after stage 1; scores each 1-10, keeps ≥7 (pinned visit kept regardless). Candidates beyond
context_stage2_max_candidates(default 15, the pinned visit exempt) are cut before scoring even happens — one sequential, unbatched Ollama call per candidate makes this a real cost, not just a formality. - Token budget check — packs visits by priority: pinned first, then stage 2's relevance score (highest first), then recency as a fallback when no score exists (stage 2 was skipped). Visits that don't fit are dropped, not summarized — see the callout below.
- Anonymize — the only stage where data crosses into what the external-capable backend will see.
Stage 2's relevance scoring deliberately runs on raw, pre-anonymization text — it's a local-only call, so there's no privacy reason to anonymize before it, and doing so first would degrade the local model's ability to judge relevance from real names/context.
Stage 3 doesn't summarize — visits that don't fit the token budget are dropped (priority-ordered, with the count surfaced on ContextSelectionResult and logged), not condensed. That's deliberate: a lossy or hallucinated summary from a small quantized local model is a worse failure mode for medical context than an honestly-truncated list. Separately, context_stage2_max_candidates (default 15) is a placeholder, not a measured value, and stage 2 scores one candidate per Ollama call with no batching — issue #56 tracks grounding the cap in real latency and evaluating batched scoring.
PII anonymization +
Hybrid approach in src/utils/anonymization.py: structured fields get deterministic handling (patient name omitted entirely — not substituted, just never included in AnonymizedProfile; DOB → exact age; doctor name → "your [specialty]"; prescribing doctor → "Prescribing physician"; doctor phone/email dropped, clinic kept), free text goes through regex (phone/email/SSN patterns) plus spaCy NER for names.
- Documented as best-effort on free text, not a guarantee — NER can miss names in unusual phrasing. This is stated explicitly rather than implied, since silently overstating a privacy guarantee is worse than none.
- The free-text date regex is deliberately over-inclusive — it redacts anything MM/DD/YYYY-shaped, not just birthdates, so a visit date mentioned inside notes prose gets swept up too. Same safe-by-default philosophy as this doc's other fail-open designs: better to over-redact than risk a real date slipping through.
- Applies uniformly regardless of provider — even when running fully local Ollama (where anonymization is technically unnecessary since nothing leaves the machine), the same code path is used for consistency and to avoid two divergent implementations.
- Tool results are anonymized too, not just the initial context — a result from
lookup_past_visitsmid-loop goes through the sameAnonymizerbefore re-entering the conversation.
Agentic tool-use loop +
_run_agentic_loop in src/agents/visit_prep.py: send context + tool specs to the backend → if it returns tool calls, execute each against the DB, anonymize the result, append to the conversation, repeat → if it returns final text with no tool calls, that's the answer → if agent_max_turns is exhausted first, raise and let the caller fall back to single-shot.
Model returns plain text before the turn cap. Logged to ConversationLog, parsed as the final JSON questions.
Parse error or turn cap hit → caught, and prepare_visit() silently falls back to a single-shot call.
On-demand structured lookup — dosage, frequency, purpose, side effects — for one or all current medications. Not a real interaction checker; that's a separate, bigger feature (issue #24).
Deeper visit-history query beyond what's already in the context, filterable by specialty/keyword. Windowing to "since last visit with this provider" is scoped separately (issue #21).
- Why bounded, not open-ended: an unbounded loop against a small/unreliable local model can spin. A hard turn cap makes worst-case latency and cost predictable.
- Why Ollama as default, not Claude API (DEC-016, superseding DEC-009's default): the dev machine (Apple M3, 8GB RAM) can only run 4-bit quantized 7-8B models, and small quantized models produce unreliable tool-calling — malformed JSON, wrong tool selection, non-convergence — which is still true and exactly why the fallback-to-single-shot behavior below matters. But defaulting the primary, most-used flow to an external API sat awkwardly next to the project's local-first pitch. Claude (and any custom OpenAI-compatible provider) remain fully supported, now as explicit opt-ins from Settings rather than the default.
- Two tools today, deliberately bounded scope for v1 — a real drug-interaction checker and a user-facing pause-to-ask-clarifying-questions flow were explicitly descoped as separate, bigger features.
- This loop's fallback isn't the last line of defense — if single-shot generation fails too (or anything else in
prepare_visit()raises), an outer catch-all returns a fixed set of generic questions rather than an error. See Reliability Design in System Design for the full three-layer chain.
Specialty-aware prompting +
Before DEC-011, visit prep generated the same questions regardless of which specialist the appointment was with — it would suggest discussing a dermatology cream with a cardiologist. Fixed with two system prompt templates (SYSTEM_PROMPT_TEMPLATE, SYSTEM_PROMPT_GENERIC in src/agents/visit_prep.py) plus data enrichment that tags conditions and medications with the specialty that actually manages them.
| ICD-10 prefix/range | Mapped specialty |
|---|---|
| E08–E13 | Endocrinology (diabetes) |
| I00–I99 | Cardiology |
| L00–L99 | Dermatology |
| E28 | Endocrinology + Gynecology (e.g. PCOS) |
- ICD-10 → specialty mapping (
ICD10_SPECIALTY_MAP, prefix and range matching via_icd10_to_specialties()) — exact prefixes checked before ranges, so a specific code like E28 doesn't get swallowed by the broader E20–E35 range. - Medication → prescriber specialty tagging — matches a medication's free-text
prescribing_doctoragainst the profile'sDoctorrecords to label it (e.g. "prescribed for Dermatology"), so the model can tell which meds are actually this specialist's business. - Clinic-name inference fallback (
_infer_specialty_from_clinic()) — keyword-matches the clinic name (e.g. "Sutter Endocrinology") whendoctor.specialtyis null. - The prompt itself draws the line explicitly: don't suggest unrelated-specialty medications, but do surface real cross-condition interactions relevant to this specialty (its own worked example: Hashimoto's and PCOS interact hormonally, which matters for an endocrinologist even if a gynecologist made the original diagnosis).
Pluggable LLM backend +
src/agents/llm_backend.py: an LLMBackend ABC with three implementations — ClaudeBackend (wraps AsyncAnthropic, uses Claude's native tools= parameter), OllamaBackend (calls Ollama's /api/chat with OpenAI-style function-calling tool specs), and CustomOpenAICompatibleBackend (DEC-016 — any endpoint speaking OpenAI's /chat/completions + tool-calling format: OpenAI, OpenRouter, Groq, Together, a self-hosted vLLM/LM Studio server, etc.). All three normalize into the same ToolCall/LLMTurnResult shape, so _run_agentic_loop never branches on which provider it's talking to — it just calls backend.call(...) and reads a uniform result.
- One canonical tool spec, reused wire format:
src/agents/tools.pydefinesTOOL_SPECSonce, thenclaude_tools()adapts it for Claude andollama_tools()covers both Ollama and the custom provider — Ollama's/api/chatalready mirrors OpenAI's function-calling shape, so the custom backend needed no third adapter. - One classification, shared by both call sites:
uses_openai_style_wire_format(provider)is the single source of truth for the Ollama/custom-vs-Claude split, used by bothget_llm_backend()(which class to instantiate) andget_tools_for_provider()(which tool-spec shape to build). Two independent== "ollama"checks could drift out of sync if a provider were added later; one shared function can't disagree with itself. - Why this matters beyond the agentic loop: it's what makes
LLM_PROVIDER=ollamaa real, fully-local default rather than a stub — the same tool-use loop, same tools, same anonymization boundary, just a different backend underneath. - Selected via a factory,
get_llm_backend(settings), keyed offsettings.llm_provider. DEC-016 also consolidated dispatch insidevisit_prep.pyitself — it previously checkedllm_provider == "ollama"via string equality in three separate places; all three now route throughget_llm_backend()/get_tools_for_provider(). - Runtime-editable, not just env-configured:
settings.llm_provideris now overlaid from a DB-backedAppSettingssingleton row (src/services/settings_service.py), so switching providers from the Settings page takes effect on the next request — no.envedit or restart needed.
Proactive action items & nudging +
Closes the loop between "document parsed" and "patient acts on what the doctor ordered." Two surfaces: a post-AVS action panel (fires at the moment of highest engagement, right after confirming a parsed document) and a persistent "Needs Attention" overview section (catches items accumulated from past visits, always visible).
- Snooze (1w/2w/1m) and completion state persist via
snoozed_until/completed_atonFollowUp/LabOrder/Referral, plus aNudgeStatetable for computed nudges that have no natural row to attach state to (e.g. "past-due appointment"). - Five computed nudge types drive the overview section: past-due appointments (
scheduled_datepassed, status stillscheduled), upcoming appointments without prep (scheduled within the next 30 days, noVisitPreprow yet), completed visits with no AVS uploaded within 14 days of the appointment date, adverse vitals trends (meaningful directional change in weight/BMI/blood pressure/heart rate across ≥2 recorded visits), and follow-ups, lab orders, and referrals that remain active (not yet completed) since they were logged. All five respect per-item snooze state so a dismissed nudge doesn't resurface until it expires. - Scheduled push notifications for genuinely disengaged patients — reaching someone who isn't currently in the app — were considered and explicitly deferred; it's new infrastructure (a scheduler + notification channel) that arguably conflicts with the local-first architecture, not a pure addition on existing data like the two shipped surfaces are.
Data model & schema design +
src/data/models.py: HealthProfile at the root, with Condition, Medication, Doctor, and Appointment hanging off it. AVS-parsed data nests under Document — Vitals (1:1), LabOrder/Referral/FollowUp (1:many) — so every parsed item traces back to the source PDF it came from. VisitPrep and ConversationLog record generated output and (anonymized) LLM call history; NudgeState persists snooze/dismiss state for the nudges that have no natural row of their own.
- UUID primary keys, not auto-increment integers (DEC-004) — a sequential ID leaks how many records exist and lets rows be enumerated by guessing. UUIDs cost a little storage/perf, which is a fine trade for health data specifically.
- Profile-nested API routes (
/api/profiles/{id}/conditions/, etc.) — ownership is enforced by the URL structure itself, not left to an ad-hoc check in each handler. Condition.icd_10was added specifically to enable specialty-aware prompting (DEC-010/DEC-011) — the schema changed because a feature needed it, not speculatively ahead of time.- Async throughout — SQLAlchemy 2.0 async sessions,
aiosqlite, matching the async FastAPI route handlers end to end rather than mixing sync DB calls into an async app.
Decisions & Tradeoffs
A curated set of the calls with real engineering tension — not all 15+ decision-log entries, just the ones worth defending in a conversation. Full log: docs/notes/DECISIONS.md.
Agentic loop framework: native tool use, not Agent SDK or LangGraph
| Option | Why not |
|---|---|
| Anthropic Agent SDK | New dependency and learning curve for a single-agent workflow that doesn't need its structured primitives |
| LangGraph | Explicit state machine is overkill — heavy abstraction for one agent, one loop |
Local Ollama vs. Claude API for the agentic backend
| Option | Tradeoff |
|---|---|
| Local Ollama (7-8B, quantized) | Free, maximum privacy — but only 4-5GB RAM available after OS on an 8GB M3, so only 4-bit quantized models fit, and those produce unreliable tool-calling (malformed JSON, wrong tools, loops) |
| Claude API (Sonnet) | Reliable tool use out of the box, ~$0.03-0.10 per visit prep (~$1/month typical use) — but leaves the local-first ideal for this one call path |
LLM_PROVIDER=ollama with documented reduced reliability. Anonymization made the Claude path privacy-acceptable rather than privacy-ideal. Superseded by DEC-016 (below) — the default flipped, this card's tool-reliability finding did not.Flip the default agentic backend to Ollama; add a custom OpenAI-compatible provider
| Option | Tradeoff |
|---|---|
| Keep Claude as default | Reliable tool-calling out of the box — but sits awkwardly next to a project pitched as local-first when the most-used flow defaults off-device |
| Flip default to Ollama, keep Claude opt-in | Privacy-first positioning holds by default — but more installs hit the small-model tool-calling unreliability DEC-009 documented, leaning harder on the existing single-shot fallback (DEC-013) |
CustomOpenAICompatibleBackend (any OpenAI-compatible endpoint — OpenAI, OpenRouter, Groq, self-hosted, etc.) as explicit opt-ins, switchable at runtime from a new Settings page (DB-backed, no .env edit or restart needed) rather than .env-only. DEC-009's reliability finding is unchanged — this supersedes its default choice, not its technical conclusion.SQLite vs. PostgreSQL
| Option | Tradeoff |
|---|---|
| SQLite | Zero setup, file-based, portable — but single-writer, not ideal if multi-user ever ships |
| PostgreSQL | Production-ready, concurrent — but requires a running server for a currently single-user local app |
UUID vs. auto-increment integer primary keys
| Option | Tradeoff |
|---|---|
| Auto-increment integers | Simple, compact, fast — but leaks record count and lets IDs be enumerated/guessed |
| UUIDs | Larger, marginally slower — but private and trivially portable across databases |
PDF parsing: local-only Ollama vs. cloud OCR/vision
| Option | Tradeoff |
|---|---|
| Claude Vision / cloud OCR | Best accuracy on scanned/image PDFs — but sends raw medical documents to a third party |
| Local Ollama + deterministic parsers | Maximum privacy, section-routing keeps accuracy reasonable on structured sections — but weaker on genuinely unstructured, image-heavy documents |
Agentic loop scope: two read-only tools, not a full feature set
| Descoped | Why |
|---|---|
| Real drug-interaction checker | Needs a licensed external interaction-database API — a separate, bigger feature (tracked as its own issue) |
| User-facing clarifying-question pause | Needs new DB state, a new API endpoint, and new frontend UI to resume a paused conversation (tracked as its own issue) |
Evaluation Plan
None of this is implemented yet — the test suite verifies plumbing (loop convergence, tool execution, anonymization boundaries hold), not output quality. Grounded in the actual pipeline (context_selection.py's 4-stage selector, visit_prep.py's prompt + agentic loop, tools.py's two read-only tools), not a generic eval template. Tracked in issue #29.
Two eval surfaces inside visit prep
Retrieval and generation are separate AI decisions that fail differently — a generation eval can't tell you retrieval missed something, because the LLM never saw it to begin with. They need separate ground truth and separate metrics.
Which past visits/facts reach the prompt. Fails by omission (a relevant visit never reaches the LLM) or dilution (irrelevant history crowds the 2000-token budget).
The questions themselves. Fails by hallucination (states a fact not in the anonymized context) or scope violation (asks the endocrinologist about a dermatology cream, which the system prompt explicitly forbids).
Retrieval eval
Stage 1 is a deterministic function of SPECIALTY_MAPPING + flags — checkable by exact assertion, not fuzzy eval. Stage 2's LLM scoring is where real judgment is needed.
| Judge? | What | Ground truth | Metric | Method |
|---|---|---|---|---|
| deterministic | Stage 1 rules | Pure function of specialty mapping + flags — exact, not fuzzy | Pass/fail assertion | Unit test, no LLM involved (cheap — close this gap first if it isn't covered) |
| deterministic | End-to-end selection (Stages 1-3) | ~30-50 synthetic profiles, human-labeled (target, past_visits[]) → expected_ids |
Recall@selected (primary — Stage 3's token budget makes recall the scarce resource), Precision@selected (secondary) | Deterministic replay of select_context() against fixtures — the full pipeline's output, not Stage 2 in isolation |
| deterministic | Stage-attribution | Same labels as above | Stage-of-death histogram — count of missed gold visits per stage (Stage 1 rule miss / Stage 2 scored low / Stage 3 budget cutoff) | Log which stage discarded each gold visit — turns "recall is 70%" into a root cause |
| local llm ± judge | Stage 2 calibration | Borderline visits with human relevance labels | Score distribution (1-10) for true-relevant vs. true-irrelevant — catches a badly-calibrated llama3.2 before it silently degrades recall |
Offline scoring against real Ollama, or a stronger model (Claude) re-scoring the same visits as a cheap proxy without full human labels |
Generation eval
The system prompt (SYSTEM_PROMPT_TEMPLATE) specifies several independent correctness properties — scoring them as one "quality" number hides which one is actually breaking.
| Judge? | Dimension | Ground truth | Metric | Method |
|---|---|---|---|---|
| det ± judge | Groundedness | The anonymized context actually sent (logged verbatim in ConversationLog) — the safety-critical dimension |
Unsupported-claim rate (unsupported facts / total factual claims) | Cheap pass: extract entities (drug/date/lab names, reusing anonymization.py's regex/NER), verify each appears in the logged context. Deeper pass: NLI-style LLM judge (supported/contradicted/unsupported) for softer inferential claims, scored by a different model than generated it to avoid self-grading bias. |
| deterministic | Specialty scope | The target appointment's specialty vs. each referenced condition/med's tagged specialty | Violation rate (out-of-scope questions / total) | Fully programmatic, no judge needed — reuses med_specialty_map/ICD-10 tags and are_specialties_related() already computed in code |
| llm judge | Relevance/usefulness | No cheap proxy — needs a rubric | 1-5 rubric score, rubric anchored with 2-3 example ratings for cross-run comparability | LLM-judge, supplemented by periodic human spot-check (the dimension most likely to drift from what a judge model rewards) |
| det ± judge | Non-redundancy | The generated question set itself | Duplicate rate (near-duplicate pairs / total questions) | Embedding similarity or judge — cheap to check programmatically |
| deterministic | Format validity | The response schema (JSON, 8-15 questions, one of 5 categories) | Pass/fail | Simple assertion, near-zero cost |
AVS parsing & PII anonymization eval
AVS parsing
Golden-set PDFs, hand-labeled JSON, diffed per field. Only needs to cover the llm/hybrid section-routing rows (vitals, lab orders, notes, referrals, diagnoses-without-Assessment) — deterministic sections are exact-match by construction.
PII anonymization
Synthetic notes with PII injected at known character offsets — recall (missed PII) and precision (over-redaction) are a span-overlap computation, no judge needed. The one component where ground truth is fully knowable.
Harness shape
eval/
fixtures/ # synthetic profiles + appointments + labeled gold context
patient_001/
profile.json
appointments.json
gold_retrieval.json # {target_appointment_id: [expected_visit_ids]}
gold_claims.json # optional: hand-labeled "these facts are grounded"
retrieval_eval.py # replays select_context() — Stage 1 needs no LLM call
generation_eval.py # replays prepare_visit(), captures output
judges/
groundedness_judge.py # NLI-style judge + entity-extraction pass
scope_judge.py # deterministic, no LLM
relevance_judge.py # rubric-based LLM judge
report.py # per-dimension metrics, stage-attribution table
Runs against real ConversationLog entries too, not just synthetic fixtures — every real call is already logged (anonymized content + tokens), so periodically sampling N real conversations through the groundedness/scope judges is a second, zero-new-data-collection eval mode. Regression gate: run the fixture set through old-prompt vs. new-prompt and diff scores per dimension before merging a prompt change — there's no before/after quality comparison today.
Offline vs. online
Single-user local app, not a service with traffic — "online" doesn't mean A/B-testing a population (already a non-goal). Reinterpreted at this project's actual scale:
Offline — the fixture harness
Synthetic labeled data, run whenever prompts or context-selection logic change. This is where recall/precision/hallucination-rate/scope-violation-rate live as repeatable numbers.
"Online" — ConversationLog monitoring
Currently write-only. Add: fallback rate (% of calls hitting single-shot instead of completing the agentic loop — see issue #30), retroactive judge sampling on real logged conversations, and a human-in-the-loop signal — noting a wrong/off-scope/fabricated question and feeding it back as a new regression fixture. The realistic substitute for population-scale online eval at single-user scale.
Build order
One ranked list across all four components, not four separate ones — by value/effort, cheapest and most deterministic first.
#1 (scope checker) is the standout — reuses med_specialty_map/ICD-10 tags already computed in code, so it's near-zero new infrastructure for a rule the system prompt already claims to enforce but never verifies. #4 (PII) is the highest-stakes item in tier 1 despite being just as cheap — it's the actual privacy boundary.
Walkthrough
⚠ synthetic example — fabricated patient, not real dataThere's no captured real-patient output in the repo to pull from (by design — nothing leaves a real user's machine). This walkthrough is a constructed, illustrative example showing what actually happens end-to-end for one visit-prep run.
Input: upcoming appointment + existing profile data
Patient has an upcoming Endocrinology visit. Profile has two conditions with a real cross-specialty interaction worth surfacing: Type 2 Diabetes (E11) and Hashimoto's thyroiditis (E06.3) — both endocrine, but historically diagnosed by different specialists (PCP and an OB-GYN, respectively).
Conditions: - Type 2 Diabetes Mellitus (E11.9) [active] — typically managed by: Endocrinology - Hashimoto's Thyroiditis (E06.3) [active] — typically managed by: Endocrinology Medications: - Metformin 500mg — twice daily [prescribed for Endocrinology] - Levothyroxine 75mcg — once daily, morning [prescribed for Endocrinology] Lab Orders: - HbA1c (ordered 2026-05-02) - TSH (ordered 2026-05-02)
Context selection + anonymization
4-stage pipeline includes the same-doctor's last visit and PCP visits automatically; a past OB-GYN visit surfaces via the Endocrinology ↔ Gynecology specialty-relevance mapping (PCOS/thyroid cross-relevance) rather than being excluded as off-specialty. Before this reaches the LLM, patient name is omitted entirely, DOB → "39 years old", doctor names → "your Endocrinologist".
Agentic loop: a tool call mid-generation
Model calls get_medication_details with no filter to double-check dosing/timing before finalizing a question about levothyroxine-metformin timing overlap — a real absorption interaction worth asking about. Result is anonymized (prescriber name redacted) and fed back into the conversation.
→ tool_call: get_medication_details({})
← result: "- Metformin (500mg) — twice daily
Purpose: Blood sugar control
- Levothyroxine (75mcg) — once daily, morning
Purpose: Thyroid hormone replacement
Prescribed by: your Endocrinologist"
Final output
Loop converges (no further tool calls) within 2 turns. Output parsed as JSON, returned to the UI:
- How does having both Hashimoto's and Type 2 Diabetes affect your target A1c range?
- Is my current thyroid control (TSH) affecting how well Metformin is managing my blood sugar?
- Should I space out Levothyroxine and Metformin — I take them close together in the morning?
- My HbA1c and TSH were both ordered last visit — what results would prompt a dosage change on either medication?
Context summary: "Patient manages co-occurring Type 2 Diabetes and Hashimoto's Thyroiditis, both under active medication management with pending labs to assess control."
Risks & Open Gaps
What's actually unsolved right now — shown deliberately, not glossed over.
Risks
Generated output is health-adjacent guidance. The only safety mechanism is the patient reading it before a real appointment — no automated check exists for a plausible-but-wrong suggestion (e.g. a hallucinated interaction, a misread lab trend). Framed as a permanent human-in-the-loop requirement, not a gap with a "done" state.
Ollama model tags (qwen2.5:7b, llama3.2) are referenced by tag, not pinned digest — a silent upstream re-tag could change parsing/scoring/visit-prep behavior with no code change to explain why, and this now affects the default agentic backend directly, not just an opt-in path. Anthropic API and any custom provider's pricing/availability changes only matter to whoever has opted into them from Settings. Tracked in issue #31.
The specialty-aware system prompts are the actual product behavior, but changes go through normal code review with no dedicated versioning or before/after quality comparison. Low-stakes at current scale; would need a real process if the loop gains more agentic freedom.
Known gaps
Tests mock the LLM and verify the pipeline works — loop convergence, tool execution, anonymization boundaries — not whether generated questions are actually good: relevant, non-redundant, correctly specialty-scoped. Tracked in issue #29.
ConversationLog stores the data, but nothing surfaces how often the loop actually falls back to single-shot. A silently degrading backend would be invisible. Tracked in issue #30.
Backend has 72+ tests; frontend has zero — no framework installed, no test files. Tracked in issue #27.
See all open issues for the full backlog, or start a Discussion if something here isn't clearly a bug or a scoped feature yet.
Glossary
Terms as used specifically in this codebase — some have a more specific meaning here than the general ML/LLM usage.
- Agentic loop
- The bounded tool-use conversation loop in
visit_prep.py— the model can call tools before producing a final answer, up toagent_max_turns, with automatic fallback to single-shot generation if it doesn't converge. - AVS
- After-Visit Summary — the PDF a patient is typically handed at the end of a doctor's visit, containing vitals, orders, diagnoses, and notes.
- Anonymization boundary
- The point in the pipeline (after context selection, before the agentic loop starts) past which no unredacted PII should exist — a hard constraint (DEC-006), not a best-effort nicety.
- Specialty-aware prompting
- System prompts that constrain generated questions to what's relevant to the specific doctor's specialty, using ICD-10 → specialty mapping and medication → prescriber tagging (DEC-011).
- Section-routing
- The AVS parser's architecture: deterministic parsers handle structured PDF sections, local LLM calls handle unstructured ones — routed by section type, not one monolithic LLM call over the whole document.
- Pluggable backend
LLMBackendabstraction (src/agents/llm_backend.py) that normalizes Ollama, Claude API, and any custom OpenAI-compatible provider's tool-calling into one interface, so the agentic loop code doesn't branch on provider (DEC-013, DEC-016).- NudgeState
- DB table persisting snooze/dismiss state for computed action items that have no natural row of their own to attach state to (e.g. "appointment is past-due").
- DEC-XXX
- An entry in
docs/notes/DECISIONS.md's architectural decision log — Date / Topic / Context / Options Considered / Decision / Reasoning / Status.
FAQ
It needs a licensed external service (RxNorm/DrugBank/FDB-class), which is a bigger, separate integration than what shipped with the agentic loop's first version — deliberately descoped rather than blocking the rest of the architecture on it. Tracked as its own follow-up.
Because small local models are known to produce unreliable tool-calling, and the single-shot path already worked before the agentic loop existed. Falling back preserves that guarantee — a user should never get a worse experience than pre-agentic HealthSteward, only a potentially less-refined one on failure.
Not scoped as HIPAA-compliant — it's personal/family use, not a covered entity under HIPAA. The privacy design (local-first, anonymization, no cloud storage) is motivated by good practice for sensitive personal data, not a regulatory compliance target.
Same local Ollama server, different model tags: PDF parsing always uses avs_parser_model (qwen2.5:7b), while relevance scoring reuses whichever model ollama_model is set to (llama3.2 by default — the same one the agentic loop uses). Not a deliberate "pick the best model per job" choice so much as parsing having its own dedicated setting and scoring never getting one — see issue #59. They're kept as distinct call sites because they have different failure-handling needs regardless: parsing failure surfaces to the user for review; scoring failure silently falls back to rules-based selection.
PDF parsing simply can't proceed (surfaced to the user). Context-selection relevance scoring skips stage 2 and falls back to rules-based filtering plus truncation. Visit prep — Ollama by default as of DEC-016 — falls back to its own single-shot path per DEC-013's convergence handling; switching to Claude or a custom provider in Settings avoids the dependency on Ollama for visit prep specifically.