Case Study 01 · 2023 — Present · McGill University & Mila
Teaching language models to say "I'm not sure."
Ask a large language model about a suspicious oral lesion and it will answer — fluently, confidently, and with exactly the same tone whether it is right or wrong. In a clinic, that isn't a quirk. It's a safety problem.
§1The problem
LLMs are remarkably capable diagnostic reasoners, but the standard ways of reading their confidence are broken. Model logits are systematically overconfident, and simply asking the model "how sure are you?" — elicited probabilities — produces numbers that sound calibrated but aren't. For safety-critical domains like oral cancer diagnosis, a wrong answer delivered with high confidence is worse than no answer at all.
What a clinician actually needs is not a single probability, but an honest picture of how much the model's belief would wobble if you probed it again and again — a distribution over its confidence.
A model that cannot say "I'm not sure" has no place at the chairside.
§2The idea
We treat the LLM as a stochastic simulator — a black box you can sample from, but whose internal likelihood is intractable. That framing unlocks a classic tool from likelihood-free inference: Approximate Bayesian Computation (ABC).
Instead of trusting one forward pass, we repeatedly sample the model's answers to a clinical case, compress those answers into summary statistics, and keep only the parameter values whose simulated behaviour lands close to what we observed. What comes out the other side is a posterior distribution over the model's predictive probability — uncertainty about the uncertainty, quantified without ever computing a likelihood.
Try it — watch the model hedge
Enough theory. Pick a case below: the model is sampled many times as a stochastic simulator, its answers disagree, and ABC turns that disagreement into an honest posterior over its confidence — set against the single overconfident number you'd read off the logits.
⚠ Synthetic, illustrative cases to demonstrate the method — not real patient data, not medical advice.
The model, sampled 40×
Posterior over its confidence
§3What it changes
Evaluated on two clinically relevant benchmarks — a synthetic oral lesion diagnosis dataset and the GretelAI symptom-to-diagnosis dataset — the ABC posteriors don't just describe uncertainty better; they make the model measurably more useful.
Against standard baselines — raw model logits and verbally elicited probabilities — the approach improves accuracy by up to 46.9% and cuts the Brier score by 74.4%, with better Expected Calibration Error and more informative predictive entropy. In plain terms: when this model says it's 90% sure, it's right about 90% of the time — and when it isn't sure, it tells you.
§4Where it's going
The work is under review at AAAI 2026 and has been presented at the Harvard Global Symposium on AI in Dentistry, IADR, the Canadian Cancer Research Conference, and CORTO 2025. It anchors my doctoral research at McGill, funded by CIHR and FRQ-S — and it is the scientific backbone of AlveoAI, where calibrated diagnostic intelligence meets dental education.