07 / PRODUCTION ENGINEERING

Artifacts, workers, deployment and incidents

Use the resource planner for configuration arithmetic. It cannot estimate latency, runtime RAM savings or infrastructure cost without measurements.

Startup and model lifecycle

1

Launch

python -m api; ddtrace-run if enabled

2

Download

Fetch seven missing S 3 artifacts; overwrite flag controls refresh

3

Preload

Workers >1: master loads cached models before fork

4

Build app

GZip middleware; /api routes; /static mount

5

Warmup

Startup calls cache loader; effective load for single worker

6

Serve

Predict in-process; no per-request S 3 or DB access

functools.cache is process-local. Pre-fork loaded memory is shared via copy-on-write until pages are modified; sharing is not a guarantee of zero per-worker RAM overhead.

Documented

Seven V 1 artifacts · one compatible bundle

Artifact	Filename	Purpose
Diagnosis mapping	diagnosis-dictionary.pkl	ICD-10 lookup
Drug mapping	druglist-dictionary.pkl	Medication lookup
New drug mapping	new-druglist-dictionary.pkl	Additional medication mapping
SHAP	explainer 1.0.pkl	Feature attribution
Model	fall-risk-analysis 1.0.joblib	Random Forest prediction
Order	order-dictionary.pkl	Exact training feature order
Preprocessing	pipeline-preprocess 1.0.pkl	Transform features

S 3 prefix: rcs-ehr-cai-s3-{env}/fallrisk/v1/; local cache api/data/ (prod volume /app/src/api/data). V 2 has separate pretrained/generated paths; its full bundle filenames are not enumerated in this README.

Resource planning lab

Replicas Workers per replica Memory request per pod (GiB)

CPU fixed at documented 2-core request /3-core limit per replica. Memory limit remains 5 GiB for the reference; if request exceeds it, the lab flags an invalid configuration. Workers do not multiply the pod’s requested resources.

Documented

Production deployment snapshot

Property	Value
Pod count / workers	10 /2
CPU request /limit	2 /3 cores per pod
Memory request /limit	4096 /5120 Mi per pod =4 /5 GiB
Total request /limit	20 /30 CPU; 40 /50 GiB
Autoscaling	No HPA configured
Port	settings default 80; local 8009; prod 8080
Ingress timeouts	360 s read/send
Volumes	emptyDir at /tmp and /app/src/api/data
Shutdown	SIGTERM; Gunicorn graceful_timeout 30 s; app shutdown hook is pass
Health	Docker 30 s healthcheck; production probes disabled

Documented

Environment map

Variable	Default / prod / purpose
API_HOST / API_PORT	0.0.0.0; 80 default /8009 local /8080 prod
API_WORKERS_COUNT	1 default; 2 prod; >1 selects Gunicorn
API_RELOAD	false; dev autoreload
API_ENVIRONMENT	dev default; prod/stg/qa/dev
API_LOG_LEVEL	INFO default
API_S 3_BUCKET	Empty default; prod rcs-ehr-cai-s 3-prod
API_FALLRISK_OVERWRITE	false; skip existing cache; true forces refresh
DATA_DIR	/app/src/api/data in prod Helm
AWS_PROFILE / AWS_ROLE_ARN / AWS_REGION	Cross-account access to model S 3
DD_TRACE_ENABLED / DD_TRACE_SAMPLE_RATE	true /0.001 prod =0.1% sampling
filepath_for_*	Local seven artifact paths
API_FALL_RISK_URL	Caller routing switch; cutover/rollback
API_ENABLE_FALL_RISK_LOGGING / API_S 3_FALL_RISK_BUCKET	Orchestrator logging controls, external to this service

Settings use Pydantic BaseSettings with API_ prefix and an optional .env file. AWS/Datadog/DATA_DIR wiring may live outside those settings; verify branch configuration rather than assuming all variables are API-prefixed.

Documented

Observability

Request trace_id → ContextVar via set_trace_id() → tagged loguru output to stdout/stderr.
Uvicorn/Gunicorn access logs routed through loguru. Datadog ddtrace-run auto-patches supported libraries (boto 3, requests, etc.).
Only application/APM logs in the service: no request/response data-capture or audit pipeline. The orchestrator can log to S 3 independently.
Proposal: monitor batch p 50/p 95, error rates, model startup failures, RSS/PSS, CPU, invalid residents, missingness and outcome performance. Keep operational and PHI logging scopes explicit.

Documented

Security and clinical governance

Resident PHI is processed in memory. Private networking/ingress and IAM provide access control; app-level auth is absent.
Cross-account IAM role scoped to artifact reads; non-root appuser UID/GID 1000; no-new-privileges.
Container capability descriptions differ: introduction says all dropped; detailed table permits NET_BIND_SERVICE. Verify the actual manifest.
Confirm encryption, retention and access for orchestrator PHI logs and contractual rights for training data.
Regulatory classification, intended use and model change-control obligations are TBD. Do not claim a confirmed SaMD/CDS classification.
Proposal: artifact provenance/checksums and trusted sources for pickle/joblib; those formats should not be loaded from untrusted inputs.

Documented

Cutover and rollback

V 2 runs side by side as ehr-fallrisk-v2 with a private route, separate image and CI pipeline. Tags deploy-fallrisk-v2-{dev,qa,stg}-* are gated to feature/fallrisk-v2.

V 1 pipeline: continuous-deployment-fallrisk-devx-eks.yml. V 2 pipeline: continuous-deployment-fallrisk-v2-devx-eks.yml. Cutover changes the caller’s API_FALL_RISK_URL; rollback points back to V 1.

Proposal: define rollback triggers before cutover, verify config propagation, audit requests by version and retain the previous artifact/service bundle.

Documented

Probe mismatch to resolve

The documented health route is /api/health, but Helm probe paths are /api/monitoring/health. Both readiness and liveness are disabled in the reviewed production values.

Proposed fix: verify the real router, align paths, distinguish ready-to-infer from live-process state, then enable probes. Consider a startup probe for cold artifact downloads; measure timings before choosing thresholds.

Incident playbook · click to inspect

Documented

Validation coverage and existing tests

Phase 1 regression: baseline capture → replay → CSV diff + deep JSON equality → zero output deviation as reported.
Phase 2 model change: holdout +14-day real-world backtest; not expected to produce identical scores.
Recommended coverage: endpoint plain/zlib parsing, eligibility boundaries, clinical weights/recency, ML feature/preprocess/inference, all hybrid branches, download/cache and output schema.
Existing tests listed: test_api.py, test_process.py, test_ml_data_creators.py, test_ml_falll_utils.py. The README’s tests section and tree disagree on the test directory (api/tests vs service/tests); verify checkout.
Proposal: performance tests across realistic batch sizes, golden feature snapshots, missing-data cases, startup failure handling and model/explainer bundle consistency.