Scaling Laws for Electronic Health Records

Overview

Question: Do scaling laws found in other fields of Deep Learning generalize to medical data?

Why it matters:

• Scaling laws tell us what happens in the extremes
• Can we reach clinical utility with available data? Or do we hit fundamental limits?
• Determines whether more compute/data helps, or if we need different approaches

Approach:

• Train GPT models at multiple scales on EHR data
• Measure loss curves and fit power laws
• Analyze what medical data properties affect scaling behavior

Scaling Laws Background

Model performance follows predictable power laws with compute budget

Fix compute budget, vary parameters vs. data to find optimal allocation of data vs compute

Motivation

Medical AI development lacks systematic understanding of scaling

• Transformer models have been applied to EHR data for a while (Li et al. 2019; Meng et al. 2021)
• Resource allocation based on intuition, NLP analogies, Hyperparameter tuning
• No verification that NLP scaling laws hold for medical data

Medical data differs fundamentally

• Non-random sampling (patient selection bias)
• Sparse, irregular temporal structure
• Institutional heterogeneity
• Severe class imbalance

Research Question

Do scaling laws exist for EHR data, or do these (or other) characteristics prevent them?

This research: Systematic verification using GPT architecture across multiple compute budgets.

Methodology

Dataset & Experimental Design

Dataset: EHR data

• Treatment sequences, diagnoses, prescriptions
• Anonymized clinical records

Experimental design:

• GPT architecture (autoregressive transformer)
• 6-8 compute budgets (spanning 2-3 orders of magnitude)
• ISOFlops curves: fix compute, sweep parameter/data tradeoff
• Ensure proper convergence (critical for valid scaling laws)

Current progress

Analysis

• Test power law fits: \( L(C) = aC^{-\alpha} + L_{\infty} \)
• Distribution shift effects (train on one institution, test on others ..? maybe..?)
• Ablation: what data properties enable/prevent scaling?

Implications

Three Possible Outcomes

1. Clean scaling laws exist (favorable exponents)
   • Can forecast compute needed for target performance
   • Shifts question from “how to do it” to “should we do it”
   • Focus moves to: what does “test loss” translate to clinically?
2. Scaling laws exist but with unfavorable exponents
   • Even using all available data → insufficient performance
   • Implies: need better algorithms, not just more compute/data
   • Precedent: speech language models (Cuervo and Marxer 2024)
3. No clean scaling laws
   • Most interesting: what properties of medical data prevent scaling?
   • May require fundamentally different approaches beyond transformers

Broader Questions

• What data properties enable scaling laws?
• Does statistical loss translate to clinical utility?
• Can models handle medical distribution shift at scale?

Discussion

Thank you for your attention.

Contact: [email protected]

References