HealthSteward / Technical Design Doc
← project site github
Snapshot as of DEC-016 · 2026-07-09

HealthSteward — Technical Design Document

A point-in-time architecture snapshot, written up for anyone who wants to go deep on how it's built. Re-written on architectural shifts, not on every change; see the note at the bottom of this page.

Problem & motivation

Patients managing fragmented care — multiple specialists, no shared record system — carry an unpaid coordination job: remembering what changed since the last visit, which labs are pending, what to raise with which doctor. This falls hardest on people with the least capacity to carry it (mid-flare, mid-crisis), and nobody on the clinical side owns it.

Most tools treat this as a records problem: store the documents, make them searchable, maybe chat with them. HealthSteward treats it as a coordination problem instead — ingest documents providers already give you, track what's changed and what's still open, and turn that into something concrete for the next visit. Locally, by default, because a tool holding this much health history shouldn't require sending it to a server to be useful.

What it does

Profile

Health profile management

Conditions (ICD-10 coded), medications, doctors, appointments in one place.

Prep

AI visit preparation

An agentic loop drafts personalized questions for an upcoming visit — pulling relevant history and calling tools before finalizing. Runs fully local via Ollama by default; Claude API and other OpenAI-compatible providers are opt-in.

Ingest

After-visit summary parsing

Upload the PDF from a visit. Parsed locally, reviewed by you, then applied to your profile.

Follow-up

Proactive action items

Follow-ups to book, labs to get done, referrals to schedule — surfaced at the moment of engagement and again until resolved.

Goals

  • Generate genuinely useful, specialty-relevant visit-prep questions from the patient's own data
  • Keep health data local by default; anonymize anything that must leave the machine
  • Turn parsed AVS data into closed-loop action, not just storage
  • Degrade gracefully — a failing LLM call should never block the user from getting something useful

Explicit non-goals

  • Not a clinical decision-support tool. Generated questions are prompts for a conversation with a real clinician — not diagnostic or treatment guidance. No automated clinical-safety validation exists or is planned; human review is the only safety mechanism today.
  • Not multi-user yet. Family sharing is deferred pending a decision, not built toward.
  • Not HIPAA-scoped. Personal/family use, not a covered entity.
  • No A/B testing, canary rollout, or drift-monitoring infrastructure. Single-user local app, no population to canary against — a deliberate scope cut, not an oversight.

Tech stack

Backend

FastAPI + SQLAlchemy (async) + SQLite, migrations via Alembic

Frontend

React 19 + TypeScript + Tailwind CSS + Vite

AI (agentic)

Pluggable backend — Ollama (llama3.2 by default) locally, or Claude API (Sonnet), or any OpenAI-compatible provider — for visit prep's tool-use loop, switchable at runtime from Settings. llama3.2 specifically for reliable multi-turn tool-calling, since this loop is conversational.

AI (local)

Ollama for two different local jobs, on two different models: qwen2.5:7b for AVS PDF parsing (single-shot text→JSON extraction, where Qwen2.5 at this size is generally considered strong at structured output) — and context-selection's Stage 2 relevance scoring, which reuses whichever model ollama_model is set to (llama3.2 by default, the same model the agentic loop uses), not a dedicated scoring model. Both fit the same 8GB-RAM/4-bit-quantized ceiling DEC-009 established. The AVS parser's model choice isn't benchmarked — no documented comparison against alternatives (issue #59).

Architecture

System Design

Two LLMs, two trust boundaries. Almost everything runs inside the dashed local boundary below; the only thing outside it is an optional external LLM call, made only after anonymization.

Data flow

Your machine — everything in this box runs locally by default
Ingest
AVS PDFdata/avs/
Ollamalocal, qwen2.5:7b
Review & confirmuser
Select
SQLitepast visits
Context selection4-stage pipeline
Ollama scoringraw, pre-anonymization · llama3.2 default
Anonymize
Deterministic replacementname omitted, DOB → age
Regex patternsphone, email, SSN
spaCy NERnames in free text
Anonymized context
Orchestrate
Agentic tool-use loop≤ agent_max_turns
Query DB, anonymize resulttool calls
⇄ optionally calls out to Claude or a custom provider — see right
Backend
Ollamalocal, llama3.2 default — stays inside this box
Serve
SQLiteVisitPrep
FastAPI
React UI
Claude API or custom provideroptional, leaves this box

Only if LLM_PROVIDER=claude or custom — anonymized context and tool results only. Response returns to Orchestrate; Serve continues either way.

local process local AI call (Ollama) data store anonymization / external boundary

Two trust boundaries

Stays on device — default

Ollama, local

PDF parsing always runs on a local model, and visit prep's agentic loop does too by default (DEC-016). Raw records never touch the network — enforced by a hard localhost-only safety check in the Ollama client, not just convention.

Anonymized first

Pluggable backend (Claude or any custom provider)

Only ever receives already-anonymized data, when opted into. Anonymization happens before the agentic loop starts, and every tool result fed back into the loop is anonymized the same way, regardless of backend.

Reliability design

The agentic tool-use loop is fallback-not-hard-failure by construction, three layers deep: if it can't converge within agent_max_turns (default 6) or a backend produces a malformed tool call, it falls back to the pre-existing single-shot prompt-in/JSON-out call; if that fails too — or anything else in prepare_visit() raises, LLM-related or not — an outer catch-all returns a fixed set of generic questions rather than an error. That hardcoded fallback, not the single-shot call, is the real floor: no functional regression is possible by design, only degraded quality on failure, and prepare_visit() never raises to its caller.

Component internals

Deep Dives

Expand any section — these are written to be answerable as "walk me through how X works" in a technical conversation.

Ingest
AVS PDF parsing +

Section-routing architecture in src/parsers/: rather than sending the whole document to an LLM, the text is split into named sections and each one is handled by whichever strategy actually fits it — deterministic regex where the shape of the data is predictable (a medication list, a header line), a focused local-LLM call only where it isn't (freeform vitals prose, referral mentions).

parse_avs_pdf() — src/parsers/__init__.py
Raw PDF text
SectionSplitterheader regexes
SectionRouterwalks PIPELINE, one section at a time
Ollamallm / hybrid sections only
postprocess_safety()

SectionSplitter (src/parsers/agent/section_splitter.py) is what turns raw PDF text into the named chunks the router hands off — a table of regexes (SECTION_HEADERS) matched against the text to find where each section starts, covering two structurally different export formats side by side:

FormatExample header patternSection key
Sutter Health AVSToday'?s\s+Visittodays_visit
Sutter Health AVS(?:Tests\s+ordered|New\s+orders\s+ordered|Lab\s+order)lab_orders
Tebra clinical note^Assessment$assessment
Tebra clinical note^Vitals\bvitals_section
  • Order mattersSECTION_HEADERS is a list, not a dict; earlier patterns are matched first when two headers could plausibly overlap on section boundaries.
  • Truncated to MAX_SECTION_CHARS = 2000 chars per section — this, not just the section-routing split itself, is what keeps each LLM call small and focused rather than re-feeding a bloated chunk of the document.
  • Cached after first access.get()/.all_sections() only run the regex table once per document, even though multiple router stages (e.g. vitals pulls from todays_visit, vitals_section, and physical_exam) request overlapping sections.
  • A section can feed more than one router stage, and a stage can pull from more than one section — the split and the routing are deliberately decoupled layers; SectionSplitter doesn't know which stage strategy is deterministic vs. LLM, it just answers "give me the text under header X."

The router's strategy per section is a fixed table, decided up front by a human reading real Sutter Health and Tebra export samples — not inferred at runtime:

SectionStrategyWhy
patient_providerdeterministicHeader line has a fixed shape: date, facility, phone.
medication_changesdeterministicStructured med-change section, cross-referenced against the medication list.
follow_updeterministicRegex for "recheck/follow-up in N weeks/months", target date computed from the visit date.
upcoming_appointmentsdeterministicSemi-tabular block keyed off month abbreviations, walked line by line.
diagnosesdeterministic → llmRegex looks for an Assessment section formatted as Condition (ICD-code); only calls the LLM if that section is missing — the one runtime (not format-time) branch in the pipeline.
vitalsllmBuried in freeform "Today's Visit" / physical-exam prose, no fixed shape.
lab_ordersllmSame — unstructured "tests ordered" narrative text.
noteshybridDeterministic extraction first, LLM supplements, results de-duped by word-overlap before merging.
referralsllmReferral mentions are scattered through instructions/plan/impression text.
Diagnoses: Assessment section found

extract_diagnoses() regex-matches Condition (ICD-code) pairs. No LLM call for this section at all.

Diagnoses: no Assessment section

Regex returns None (not []) — the router reads that as "use the LLM" and routes to the diagnoses llm config instead.

Each llm/hybrid section only receives the specific chunk(s) of text it needs (from SectionSplitter, capped at 2000 chars each) — never the full document — via a focused system prompt (src/parsers/agent/prompts.py) and a single direct_chat() call (src/parsers/agent/ollama_chat.py).

  • Local-only, enforced not just documentedollama_chat.py has a safety check that blocks any non-localhost URL, so a misconfiguration can't silently route PHI externally.
  • Two document formats, one pipeline — the deterministic regexes carry parallel fallback chains for Sutter Health AVS exports and Tebra clinical-note exports, since the two look structurally different (e.g. header parsing tries the Sutter header-line pattern first, then falls back to Tebra's "Visited on:" / "MRN :" patterns).
  • Review before apply — parsed items are always presented for user confirmation with per-section checkboxes; nothing is auto-applied to the profile.
  • Deduplication — diagnoses dedupe by name on apply (existing conditions get ICD-10 backfilled if missing); medication stops match by name.
Select
4-stage context selection +

Before generating visit-prep questions, relevant past-visit history is selected via a 4-stage pipeline in src/utils/context_selection.py:

runs once per visit prep, in order
1. Rules-based filter
2. Local LLM scoring
3. Token budget check
4. Anonymize
↺ stage 4 is the only point data crosses into what the external-capable backend will see
Stage 2 runs

More than 5 visits remain after stage 1, and Ollama is available. Each is scored 1–10; ≥7 survives, plus the pinned visit regardless of score.

Stage 2 skipped

≤5 visits remain, or Ollama is unavailable — stage 3 packs by recency instead of score, since none exists.

  1. Rules-based filter (instant, free) — always include same-doctor's last visit and PCP/Internal Medicine visits; apply a specialty-relevance mapping (e.g. Endocrinology ↔ Cardiology, Nephrology); exclude doctors flagged out of prep context. The same-doctor visit is pinned — its id is tracked through stages 2 and 3 so it can't be capped, score-filtered, or budget-dropped later, unlike every other candidate.
  2. Local LLM relevance scoring (Ollama) — only runs if more than 5 visits remain after stage 1; scores each 1-10, keeps ≥7 (pinned visit kept regardless). Candidates beyond context_stage2_max_candidates (default 15, the pinned visit exempt) are cut before scoring even happens — one sequential, unbatched Ollama call per candidate makes this a real cost, not just a formality.
  3. Token budget check — packs visits by priority: pinned first, then stage 2's relevance score (highest first), then recency as a fallback when no score exists (stage 2 was skipped). Visits that don't fit are dropped, not summarized — see the callout below.
  4. Anonymize — the only stage where data crosses into what the external-capable backend will see.

Stage 2's relevance scoring deliberately runs on raw, pre-anonymization text — it's a local-only call, so there's no privacy reason to anonymize before it, and doing so first would degrade the local model's ability to judge relevance from real names/context.

Known gap

Stage 3 doesn't summarize — visits that don't fit the token budget are dropped (priority-ordered, with the count surfaced on ContextSelectionResult and logged), not condensed. That's deliberate: a lossy or hallucinated summary from a small quantized local model is a worse failure mode for medical context than an honestly-truncated list. Separately, context_stage2_max_candidates (default 15) is a placeholder, not a measured value, and stage 2 scores one candidate per Ollama call with no batching — issue #56 tracks grounding the cap in real latency and evaluating batched scoring.

Anonymize
PII anonymization +

Hybrid approach in src/utils/anonymization.py: structured fields get deterministic handling (patient name omitted entirely — not substituted, just never included in AnonymizedProfile; DOB → exact age; doctor name → "your [specialty]"; prescribing doctor → "Prescribing physician"; doctor phone/email dropped, clinic kept), free text goes through regex (phone/email/SSN patterns) plus spaCy NER for names.

  • Documented as best-effort on free text, not a guarantee — NER can miss names in unusual phrasing. This is stated explicitly rather than implied, since silently overstating a privacy guarantee is worse than none.
  • The free-text date regex is deliberately over-inclusive — it redacts anything MM/DD/YYYY-shaped, not just birthdates, so a visit date mentioned inside notes prose gets swept up too. Same safe-by-default philosophy as this doc's other fail-open designs: better to over-redact than risk a real date slipping through.
  • Applies uniformly regardless of provider — even when running fully local Ollama (where anonymization is technically unnecessary since nothing leaves the machine), the same code path is used for consistency and to avoid two divergent implementations.
  • Tool results are anonymized too, not just the initial context — a result from lookup_past_visits mid-loop goes through the same Anonymizer before re-entering the conversation.
Orchestrate
Agentic tool-use loop +

_run_agentic_loop in src/agents/visit_prep.py: send context + tool specs to the backend → if it returns tool calls, execute each against the DB, anonymize the result, append to the conversation, repeat → if it returns final text with no tool calls, that's the answer → if agent_max_turns is exhausted first, raise and let the caller fall back to single-shot.

repeats up to agent_max_turns (default 6)
Model responds
Tool calls?
Execute, anonymize result
Append, loop again
↺ no tool calls in the response → loop exits, that response is the final answer
Converges

Model returns plain text before the turn cap. Logged to ConversationLog, parsed as the final JSON questions.

Fails or times out

Parse error or turn cap hit → caught, and prepare_visit() silently falls back to a single-shot call.

get_medication_details

On-demand structured lookup — dosage, frequency, purpose, side effects — for one or all current medications. Not a real interaction checker; that's a separate, bigger feature (issue #24).

lookup_past_visits

Deeper visit-history query beyond what's already in the context, filterable by specialty/keyword. Windowing to "since last visit with this provider" is scoped separately (issue #21).

  • Why bounded, not open-ended: an unbounded loop against a small/unreliable local model can spin. A hard turn cap makes worst-case latency and cost predictable.
  • Why Ollama as default, not Claude API (DEC-016, superseding DEC-009's default): the dev machine (Apple M3, 8GB RAM) can only run 4-bit quantized 7-8B models, and small quantized models produce unreliable tool-calling — malformed JSON, wrong tool selection, non-convergence — which is still true and exactly why the fallback-to-single-shot behavior below matters. But defaulting the primary, most-used flow to an external API sat awkwardly next to the project's local-first pitch. Claude (and any custom OpenAI-compatible provider) remain fully supported, now as explicit opt-ins from Settings rather than the default.
  • Two tools today, deliberately bounded scope for v1 — a real drug-interaction checker and a user-facing pause-to-ask-clarifying-questions flow were explicitly descoped as separate, bigger features.
  • This loop's fallback isn't the last line of defense — if single-shot generation fails too (or anything else in prepare_visit() raises), an outer catch-all returns a fixed set of generic questions rather than an error. See Reliability Design in System Design for the full three-layer chain.
Specialty-aware prompting +

Before DEC-011, visit prep generated the same questions regardless of which specialist the appointment was with — it would suggest discussing a dermatology cream with a cardiologist. Fixed with two system prompt templates (SYSTEM_PROMPT_TEMPLATE, SYSTEM_PROMPT_GENERIC in src/agents/visit_prep.py) plus data enrichment that tags conditions and medications with the specialty that actually manages them.

ICD-10 prefix/rangeMapped specialty
E08–E13Endocrinology (diabetes)
I00–I99Cardiology
L00–L99Dermatology
E28Endocrinology + Gynecology (e.g. PCOS)
  • ICD-10 → specialty mapping (ICD10_SPECIALTY_MAP, prefix and range matching via _icd10_to_specialties()) — exact prefixes checked before ranges, so a specific code like E28 doesn't get swallowed by the broader E20–E35 range.
  • Medication → prescriber specialty tagging — matches a medication's free-text prescribing_doctor against the profile's Doctor records to label it (e.g. "prescribed for Dermatology"), so the model can tell which meds are actually this specialist's business.
  • Clinic-name inference fallback (_infer_specialty_from_clinic()) — keyword-matches the clinic name (e.g. "Sutter Endocrinology") when doctor.specialty is null.
  • The prompt itself draws the line explicitly: don't suggest unrelated-specialty medications, but do surface real cross-condition interactions relevant to this specialty (its own worked example: Hashimoto's and PCOS interact hormonally, which matters for an endocrinologist even if a gynecologist made the original diagnosis).
Backend
Pluggable LLM backend +

src/agents/llm_backend.py: an LLMBackend ABC with three implementations — ClaudeBackend (wraps AsyncAnthropic, uses Claude's native tools= parameter), OllamaBackend (calls Ollama's /api/chat with OpenAI-style function-calling tool specs), and CustomOpenAICompatibleBackend (DEC-016 — any endpoint speaking OpenAI's /chat/completions + tool-calling format: OpenAI, OpenRouter, Groq, Together, a self-hosted vLLM/LM Studio server, etc.). All three normalize into the same ToolCall/LLMTurnResult shape, so _run_agentic_loop never branches on which provider it's talking to — it just calls backend.call(...) and reads a uniform result.

  • One canonical tool spec, reused wire format: src/agents/tools.py defines TOOL_SPECS once, then claude_tools() adapts it for Claude and ollama_tools() covers both Ollama and the custom provider — Ollama's /api/chat already mirrors OpenAI's function-calling shape, so the custom backend needed no third adapter.
  • One classification, shared by both call sites: uses_openai_style_wire_format(provider) is the single source of truth for the Ollama/custom-vs-Claude split, used by both get_llm_backend() (which class to instantiate) and get_tools_for_provider() (which tool-spec shape to build). Two independent == "ollama" checks could drift out of sync if a provider were added later; one shared function can't disagree with itself.
  • Why this matters beyond the agentic loop: it's what makes LLM_PROVIDER=ollama a real, fully-local default rather than a stub — the same tool-use loop, same tools, same anonymization boundary, just a different backend underneath.
  • Selected via a factory, get_llm_backend(settings), keyed off settings.llm_provider. DEC-016 also consolidated dispatch inside visit_prep.py itself — it previously checked llm_provider == "ollama" via string equality in three separate places; all three now route through get_llm_backend()/get_tools_for_provider().
  • Runtime-editable, not just env-configured: settings.llm_provider is now overlaid from a DB-backed AppSettings singleton row (src/services/settings_service.py), so switching providers from the Settings page takes effect on the next request — no .env edit or restart needed.
Cross-cutting
Proactive action items & nudging +

Closes the loop between "document parsed" and "patient acts on what the doctor ordered." Two surfaces: a post-AVS action panel (fires at the moment of highest engagement, right after confirming a parsed document) and a persistent "Needs Attention" overview section (catches items accumulated from past visits, always visible).

  • Snooze (1w/2w/1m) and completion state persist via snoozed_until/completed_at on FollowUp/LabOrder/Referral, plus a NudgeState table for computed nudges that have no natural row to attach state to (e.g. "past-due appointment").
  • Five computed nudge types drive the overview section: past-due appointments (scheduled_date passed, status still scheduled), upcoming appointments without prep (scheduled within the next 30 days, no VisitPrep row yet), completed visits with no AVS uploaded within 14 days of the appointment date, adverse vitals trends (meaningful directional change in weight/BMI/blood pressure/heart rate across ≥2 recorded visits), and follow-ups, lab orders, and referrals that remain active (not yet completed) since they were logged. All five respect per-item snooze state so a dismissed nudge doesn't resurface until it expires.
  • Scheduled push notifications for genuinely disengaged patients — reaching someone who isn't currently in the app — were considered and explicitly deferred; it's new infrastructure (a scheduler + notification channel) that arguably conflicts with the local-first architecture, not a pure addition on existing data like the two shipped surfaces are.
Data model & schema design +

src/data/models.py: HealthProfile at the root, with Condition, Medication, Doctor, and Appointment hanging off it. AVS-parsed data nests under DocumentVitals (1:1), LabOrder/Referral/FollowUp (1:many) — so every parsed item traces back to the source PDF it came from. VisitPrep and ConversationLog record generated output and (anonymized) LLM call history; NudgeState persists snooze/dismiss state for the nudges that have no natural row of their own.

HealthProfile
↓ has many
Conditionicd_10
Medicationprescribing_doctor
Doctorspecialty
Appointmentprep/visit notes
Document
↓ every parsed item traces back to its source PDF
Vitals1:1
LabOrder1:many
Referral1:many
FollowUp1:many
supporting / computed — no direct FK from the two roots above
VisitPrepgenerated output
ConversationLoganonymized LLM history
NudgeStatesnooze/dismiss state
  • UUID primary keys, not auto-increment integers (DEC-004) — a sequential ID leaks how many records exist and lets rows be enumerated by guessing. UUIDs cost a little storage/perf, which is a fine trade for health data specifically.
  • Profile-nested API routes (/api/profiles/{id}/conditions/, etc.) — ownership is enforced by the URL structure itself, not left to an ad-hoc check in each handler.
  • Condition.icd_10 was added specifically to enable specialty-aware prompting (DEC-010/DEC-011) — the schema changed because a feature needed it, not speculatively ahead of time.
  • Async throughout — SQLAlchemy 2.0 async sessions, aiosqlite, matching the async FastAPI route handlers end to end rather than mixing sync DB calls into an async app.
Alternatives considered

Decisions & Tradeoffs

A curated set of the calls with real engineering tension — not all 15+ decision-log entries, just the ones worth defending in a conversation. Full log: docs/notes/DECISIONS.md.

DEC-009 / DEC-013

Agentic loop framework: native tool use, not Agent SDK or LangGraph

OptionWhy not
Anthropic Agent SDKNew dependency and learning curve for a single-agent workflow that doesn't need its structured primitives
LangGraphExplicit state machine is overkill — heavy abstraction for one agent, one loop
Chose: Claude API's native tool use. Already integrated, no new deps, a simple send/loop/execute pattern that's easy to upgrade to a framework later if the loop ever needs to coordinate multiple agents.
DEC-009

Local Ollama vs. Claude API for the agentic backend

OptionTradeoff
Local Ollama (7-8B, quantized)Free, maximum privacy — but only 4-5GB RAM available after OS on an 8GB M3, so only 4-bit quantized models fit, and those produce unreliable tool-calling (malformed JSON, wrong tools, loops)
Claude API (Sonnet)Reliable tool use out of the box, ~$0.03-0.10 per visit prep (~$1/month typical use) — but leaves the local-first ideal for this one call path
Chose (at the time): Claude API as default, Ollama as an explicit opt-in via LLM_PROVIDER=ollama with documented reduced reliability. Anonymization made the Claude path privacy-acceptable rather than privacy-ideal. Superseded by DEC-016 (below) — the default flipped, this card's tool-reliability finding did not.
DEC-016

Flip the default agentic backend to Ollama; add a custom OpenAI-compatible provider

OptionTradeoff
Keep Claude as defaultReliable tool-calling out of the box — but sits awkwardly next to a project pitched as local-first when the most-used flow defaults off-device
Flip default to Ollama, keep Claude opt-inPrivacy-first positioning holds by default — but more installs hit the small-model tool-calling unreliability DEC-009 documented, leaning harder on the existing single-shot fallback (DEC-013)
Chose: Ollama as default; Claude and a new CustomOpenAICompatibleBackend (any OpenAI-compatible endpoint — OpenAI, OpenRouter, Groq, self-hosted, etc.) as explicit opt-ins, switchable at runtime from a new Settings page (DB-backed, no .env edit or restart needed) rather than .env-only. DEC-009's reliability finding is unchanged — this supersedes its default choice, not its technical conclusion.
DEC-003

SQLite vs. PostgreSQL

OptionTradeoff
SQLiteZero setup, file-based, portable — but single-writer, not ideal if multi-user ever ships
PostgreSQLProduction-ready, concurrent — but requires a running server for a currently single-user local app
Chose: SQLite for the current single-user phase; the SQLAlchemy abstraction keeps a later move to Postgres cheap if/when multi-user sharing (deferred, DEC-001) actually ships.
DEC-004

UUID vs. auto-increment integer primary keys

OptionTradeoff
Auto-increment integersSimple, compact, fast — but leaks record count and lets IDs be enumerated/guessed
UUIDsLarger, marginally slower — but private and trivially portable across databases
Chose: UUIDs. Health data specifically warrants the extra privacy consideration over the minor storage/perf cost.
DEC-005 / DEC-010

PDF parsing: local-only Ollama vs. cloud OCR/vision

OptionTradeoff
Claude Vision / cloud OCRBest accuracy on scanned/image PDFs — but sends raw medical documents to a third party
Local Ollama + deterministic parsersMaximum privacy, section-routing keeps accuracy reasonable on structured sections — but weaker on genuinely unstructured, image-heavy documents
Chose: Local-only Ollama, enforced with a hard safety check, not just a default. For a tool whose entire pitch is "your health data never leaves your machine," an accuracy/privacy tradeoff here isn't really optional.
DEC-013

Agentic loop scope: two read-only tools, not a full feature set

DescopedWhy
Real drug-interaction checkerNeeds a licensed external interaction-database API — a separate, bigger feature (tracked as its own issue)
User-facing clarifying-question pauseNeeds new DB state, a new API endpoint, and new frontend UI to resume a paused conversation (tracked as its own issue)
Chose: ship the smallest version of the loop that's actually useful — two tools buildable entirely from existing schema data, both read-only, both anonymized — rather than block the whole agentic architecture on the bigger features.
Proposed, not built

Evaluation Plan

None of this is implemented yet — the test suite verifies plumbing (loop convergence, tool execution, anonymization boundaries hold), not output quality. Grounded in the actual pipeline (context_selection.py's 4-stage selector, visit_prep.py's prompt + agentic loop, tools.py's two read-only tools), not a generic eval template. Tracked in issue #29.

Two eval surfaces inside visit prep

Retrieval and generation are separate AI decisions that fail differently — a generation eval can't tell you retrieval missed something, because the LLM never saw it to begin with. They need separate ground truth and separate metrics.

Retrieval — ContextSelector, stages 1-2

Which past visits/facts reach the prompt. Fails by omission (a relevant visit never reaches the LLM) or dilution (irrelevant history crowds the 2000-token budget).

Generation — VisitPrepAgent.prepare_visit

The questions themselves. Fails by hallucination (states a fact not in the anonymized context) or scope violation (asks the endocrinologist about a dermatology cream, which the system prompt explicitly forbids).

Retrieval eval

Stage 1 is a deterministic function of SPECIALTY_MAPPING + flags — checkable by exact assertion, not fuzzy eval. Stage 2's LLM scoring is where real judgment is needed.

Judge?WhatGround truthMetricMethod
deterministic Stage 1 rules Pure function of specialty mapping + flags — exact, not fuzzy Pass/fail assertion Unit test, no LLM involved (cheap — close this gap first if it isn't covered)
deterministic End-to-end selection (Stages 1-3) ~30-50 synthetic profiles, human-labeled (target, past_visits[]) → expected_ids Recall@selected (primary — Stage 3's token budget makes recall the scarce resource), Precision@selected (secondary) Deterministic replay of select_context() against fixtures — the full pipeline's output, not Stage 2 in isolation
deterministic Stage-attribution Same labels as above Stage-of-death histogram — count of missed gold visits per stage (Stage 1 rule miss / Stage 2 scored low / Stage 3 budget cutoff) Log which stage discarded each gold visit — turns "recall is 70%" into a root cause
local llm ± judge Stage 2 calibration Borderline visits with human relevance labels Score distribution (1-10) for true-relevant vs. true-irrelevant — catches a badly-calibrated llama3.2 before it silently degrades recall Offline scoring against real Ollama, or a stronger model (Claude) re-scoring the same visits as a cheap proxy without full human labels

Generation eval

The system prompt (SYSTEM_PROMPT_TEMPLATE) specifies several independent correctness properties — scoring them as one "quality" number hides which one is actually breaking.

Judge?DimensionGround truthMetricMethod
det ± judge Groundedness The anonymized context actually sent (logged verbatim in ConversationLog) — the safety-critical dimension Unsupported-claim rate (unsupported facts / total factual claims) Cheap pass: extract entities (drug/date/lab names, reusing anonymization.py's regex/NER), verify each appears in the logged context. Deeper pass: NLI-style LLM judge (supported/contradicted/unsupported) for softer inferential claims, scored by a different model than generated it to avoid self-grading bias.
deterministic Specialty scope The target appointment's specialty vs. each referenced condition/med's tagged specialty Violation rate (out-of-scope questions / total) Fully programmatic, no judge needed — reuses med_specialty_map/ICD-10 tags and are_specialties_related() already computed in code
llm judge Relevance/usefulness No cheap proxy — needs a rubric 1-5 rubric score, rubric anchored with 2-3 example ratings for cross-run comparability LLM-judge, supplemented by periodic human spot-check (the dimension most likely to drift from what a judge model rewards)
det ± judge Non-redundancy The generated question set itself Duplicate rate (near-duplicate pairs / total questions) Embedding similarity or judge — cheap to check programmatically
deterministic Format validity The response schema (JSON, 8-15 questions, one of 5 categories) Pass/fail Simple assertion, near-zero cost

AVS parsing & PII anonymization eval

Ingest

AVS parsing

Golden-set PDFs, hand-labeled JSON, diffed per field. Only needs to cover the llm/hybrid section-routing rows (vitals, lab orders, notes, referrals, diagnoses-without-Assessment) — deterministic sections are exact-match by construction.

Anonymize

PII anonymization

Synthetic notes with PII injected at known character offsets — recall (missed PII) and precision (over-redaction) are a span-overlap computation, no judge needed. The one component where ground truth is fully knowable.

Harness shape

eval/
  fixtures/                    # synthetic profiles + appointments + labeled gold context
    patient_001/
      profile.json
      appointments.json
      gold_retrieval.json      # {target_appointment_id: [expected_visit_ids]}
      gold_claims.json         # optional: hand-labeled "these facts are grounded"
  retrieval_eval.py            # replays select_context() — Stage 1 needs no LLM call
  generation_eval.py           # replays prepare_visit(), captures output
  judges/
    groundedness_judge.py      # NLI-style judge + entity-extraction pass
    scope_judge.py             # deterministic, no LLM
    relevance_judge.py         # rubric-based LLM judge
  report.py                    # per-dimension metrics, stage-attribution table

Runs against real ConversationLog entries too, not just synthetic fixtures — every real call is already logged (anonymized content + tokens), so periodically sampling N real conversations through the groundedness/scope judges is a second, zero-new-data-collection eval mode. Regression gate: run the fixture set through old-prompt vs. new-prompt and diff scores per dimension before merging a prompt change — there's no before/after quality comparison today.

Offline vs. online

Single-user local app, not a service with traffic — "online" doesn't mean A/B-testing a population (already a non-goal). Reinterpreted at this project's actual scale:

Run on-demand

Offline — the fixture harness

Synthetic labeled data, run whenever prompts or context-selection logic change. This is where recall/precision/hallucination-rate/scope-violation-rate live as repeatable numbers.

Passive, on real usage

"Online" — ConversationLog monitoring

Currently write-only. Add: fallback rate (% of calls hitting single-shot instead of completing the agentic loop — see issue #30), retroactive judge sampling on real logged conversations, and a human-in-the-loop signal — noting a wrong/off-scope/fabricated question and feeding it back as a new regression fixture. The realistic substitute for population-scale online eval at single-user scale.

Build order

One ranked list across all four components, not four separate ones — by value/effort, cheapest and most deterministic first.

tier 1 — deterministic, no judge model, build first
1. Scope checkergeneration
2. Format validationgeneration
3. Stage 1 unit testsretrieval
4. PII span-overlapanonymize
5. AVS golden-set diffingest
tier 2 — cheap, no judge, but not a pure assertion
6. Groundedness entity-matchgeneration
tier 3 — needs fixtures and/or a judge model, build last
7. Retrieval recall/precisionretrieval
8. LLM-judge relevancegeneration

#1 (scope checker) is the standout — reuses med_specialty_map/ICD-10 tags already computed in code, so it's near-zero new infrastructure for a rule the system prompt already claims to enforce but never verifies. #4 (PII) is the highest-stakes item in tier 1 despite being just as cheap — it's the actual privacy boundary.

Example run

Walkthrough

⚠ synthetic example — fabricated patient, not real data

There's no captured real-patient output in the repo to pull from (by design — nothing leaves a real user's machine). This walkthrough is a constructed, illustrative example showing what actually happens end-to-end for one visit-prep run.

1

Input: upcoming appointment + existing profile data

Patient has an upcoming Endocrinology visit. Profile has two conditions with a real cross-specialty interaction worth surfacing: Type 2 Diabetes (E11) and Hashimoto's thyroiditis (E06.3) — both endocrine, but historically diagnosed by different specialists (PCP and an OB-GYN, respectively).

Conditions:
- Type 2 Diabetes Mellitus (E11.9) [active] — typically managed by: Endocrinology
- Hashimoto's Thyroiditis (E06.3) [active] — typically managed by: Endocrinology
Medications:
- Metformin 500mg — twice daily [prescribed for Endocrinology]
- Levothyroxine 75mcg — once daily, morning [prescribed for Endocrinology]
Lab Orders:
- HbA1c (ordered 2026-05-02)
- TSH (ordered 2026-05-02)
2

Context selection + anonymization

4-stage pipeline includes the same-doctor's last visit and PCP visits automatically; a past OB-GYN visit surfaces via the Endocrinology ↔ Gynecology specialty-relevance mapping (PCOS/thyroid cross-relevance) rather than being excluded as off-specialty. Before this reaches the LLM, patient name is omitted entirely, DOB → "39 years old", doctor names → "your Endocrinologist".

3

Agentic loop: a tool call mid-generation

Model calls get_medication_details with no filter to double-check dosing/timing before finalizing a question about levothyroxine-metformin timing overlap — a real absorption interaction worth asking about. Result is anonymized (prescriber name redacted) and fed back into the conversation.

→ tool_call: get_medication_details({})
← result: "- Metformin (500mg) — twice daily
  Purpose: Blood sugar control
- Levothyroxine (75mcg) — once daily, morning
  Purpose: Thyroid hormone replacement
  Prescribed by: your Endocrinologist"
4

Final output

Loop converges (no further tool calls) within 2 turns. Output parsed as JSON, returned to the UI:

Condition Management
  • How does having both Hashimoto's and Type 2 Diabetes affect your target A1c range?
  • Is my current thyroid control (TSH) affecting how well Metformin is managing my blood sugar?
Medication Review
  • Should I space out Levothyroxine and Metformin — I take them close together in the morning?
Lab Results & Monitoring
  • My HbA1c and TSH were both ordered last visit — what results would prompt a dosage change on either medication?

Context summary: "Patient manages co-occurring Type 2 Diabetes and Hashimoto's Thyroiditis, both under active medication management with pending labs to assess control."

Honest assessment

Risks & Open Gaps

What's actually unsolved right now — shown deliberately, not glossed over.

Risks

Clinical safety

Generated output is health-adjacent guidance. The only safety mechanism is the patient reading it before a real appointment — no automated check exists for a plausible-but-wrong suggestion (e.g. a hallucinated interaction, a misread lab trend). Framed as a permanent human-in-the-loop requirement, not a gap with a "done" state.

External dependency risk

Ollama model tags (qwen2.5:7b, llama3.2) are referenced by tag, not pinned digest — a silent upstream re-tag could change parsing/scoring/visit-prep behavior with no code change to explain why, and this now affects the default agentic backend directly, not just an opt-in path. Anthropic API and any custom provider's pricing/availability changes only matter to whoever has opted into them from Settings. Tracked in issue #31.

Prompt-change management

The specialty-aware system prompts are the actual product behavior, but changes go through normal code review with no dedicated versioning or before/after quality comparison. Low-stakes at current scale; would need a real process if the loop gains more agentic freedom.

Known gaps

No quality evaluation of generated output

Tests mock the LLM and verify the pipeline works — loop convergence, tool execution, anonymization boundaries — not whether generated questions are actually good: relevant, non-redundant, correctly specialty-scoped. Tracked in issue #29.

No visibility into agentic-loop fallback rate

ConversationLog stores the data, but nothing surfaces how often the loop actually falls back to single-shot. A silently degrading backend would be invisible. Tracked in issue #30.

No frontend test coverage

Backend has 72+ tests; frontend has zero — no framework installed, no test files. Tracked in issue #27.

See all open issues for the full backlog, or start a Discussion if something here isn't clearly a bug or a scoped feature yet.

Reference

Glossary

Terms as used specifically in this codebase — some have a more specific meaning here than the general ML/LLM usage.

Agentic loop
The bounded tool-use conversation loop in visit_prep.py — the model can call tools before producing a final answer, up to agent_max_turns, with automatic fallback to single-shot generation if it doesn't converge.
AVS
After-Visit Summary — the PDF a patient is typically handed at the end of a doctor's visit, containing vitals, orders, diagnoses, and notes.
Anonymization boundary
The point in the pipeline (after context selection, before the agentic loop starts) past which no unredacted PII should exist — a hard constraint (DEC-006), not a best-effort nicety.
Specialty-aware prompting
System prompts that constrain generated questions to what's relevant to the specific doctor's specialty, using ICD-10 → specialty mapping and medication → prescriber tagging (DEC-011).
Section-routing
The AVS parser's architecture: deterministic parsers handle structured PDF sections, local LLM calls handle unstructured ones — routed by section type, not one monolithic LLM call over the whole document.
Pluggable backend
LLMBackend abstraction (src/agents/llm_backend.py) that normalizes Ollama, Claude API, and any custom OpenAI-compatible provider's tool-calling into one interface, so the agentic loop code doesn't branch on provider (DEC-013, DEC-016).
NudgeState
DB table persisting snooze/dismiss state for computed action items that have no natural row of their own to attach state to (e.g. "appointment is past-due").
DEC-XXX
An entry in docs/notes/DECISIONS.md's architectural decision log — Date / Topic / Context / Options Considered / Decision / Reasoning / Status.
Reference

FAQ

Why not just use a real drug-interaction API from day one?

It needs a licensed external service (RxNorm/DrugBank/FDB-class), which is a bigger, separate integration than what shipped with the agentic loop's first version — deliberately descoped rather than blocking the rest of the architecture on it. Tracked as its own follow-up.

Why does the agentic loop fall back instead of just erroring?

Because small local models are known to produce unreliable tool-calling, and the single-shot path already worked before the agentic loop existed. Falling back preserves that guarantee — a user should never get a worse experience than pre-agentic HealthSteward, only a potentially less-refined one on failure.

Is any of this HIPAA-compliant?

Not scoped as HIPAA-compliant — it's personal/family use, not a covered entity under HIPAA. The privacy design (local-first, anonymization, no cloud storage) is motivated by good practice for sensitive personal data, not a regulatory compliance target.

Why two different local models (Ollama for parsing vs. relevance scoring) instead of one shared path?

Same local Ollama server, different model tags: PDF parsing always uses avs_parser_model (qwen2.5:7b), while relevance scoring reuses whichever model ollama_model is set to (llama3.2 by default — the same one the agentic loop uses). Not a deliberate "pick the best model per job" choice so much as parsing having its own dedicated setting and scoring never getting one — see issue #59. They're kept as distinct call sites because they have different failure-handling needs regardless: parsing failure surfaces to the user for review; scoring failure silently falls back to rules-based selection.

What happens if Ollama isn't running at all?

PDF parsing simply can't proceed (surfaced to the user). Context-selection relevance scoring skips stage 2 and falls back to rules-based filtering plus truncation. Visit prep — Ollama by default as of DEC-016 — falls back to its own single-shot path per DEC-013's convergence handling; switching to Claude or a custom provider in Settings avoids the dependency on Ollama for visit prep specifically.