Oncology Safety Bench

I tested whether small AI models that can run on a hospital’s own computers can plan cancer treatment safely.

Role: Lead researcher
Mentors: Uri Ilan, MD (clinical oncology) and Liron Yatziv, PhD (computer science)
When: 2025–present
Links: Preprint
Code

Diagram of the study: a patient record, the ways evidence was given to the model, the draft plan, and the final check

150patient cases
13AI models tested
5databases combined

Background

Hospitals usually can’t send patient records to big cloud AI services. A small model running on the hospital’s own computers gets around that problem. I wanted to know if those small models are actually safe to use for cancer treatment plans.

The risky part usually isn’t picking the cancer drug. It’s everything else in the patient’s chart. A patient might already take a medication that reacts badly with the new drug, or have a gene that changes how their body handles it.

The cases

To test this, I wrote 150 patient cases covering 21 types of cancer. Each patient has two visits, so there are 300 treatment decisions in total.

Most cases have one problem hidden in an otherwise normal chart. It might be a drug interaction, a gene result, a tumor mutation that makes the usual drug stop working, or a lab result like poor kidney function. Some cases have no problem at all, so a model can’t score well just by refusing to recommend anything.

Each problem comes from a public medical database. I combined five of these databases into one that the models can search offline.

Testing the models

I ran 13 models on every case. Eleven were open models small enough to run on one computer. The other two were Google’s largest Gemini models, for comparison.

Each model tried every case three ways. First with no extra help. Then with the relevant database entries given to it up front. Then with search tools, where it had to look things up itself.

Two other AI models graded every plan against an answer key. If a plan missed the problem, or the model refused to answer, it counted as unsafe.

Dot plot of unsafe and safe-and-useful rates for each model — Unsafe rate (left) and safe-and-useful rate (right) for each model. Figure 2 of the preprint.

Results

With no help, about a quarter of the small models’ plans were unsafe. Giving them the right database entries up front brought that down to about one in ten.

The search tools didn’t help as much as I expected. Most models looked up the cancer drug but never checked the patient’s other medications, and that’s where most of the problems were. When a model did find the right record, its plan was usually safe.

Scatter plot of how often each model looked up the affected drug against its unsafe rate — How often each model looked up the affected drug, against how often its plans were unsafe.

Final check

Since the models weren’t looking up the right things, I added an automatic check after the plan is written. It looks up every drug in the plan and gives the results back to the model so it can fix the plan. That fixed about 60% of the unsafe plans.

With this check, the biggest open model, Gemma 4 31B, did almost as well as Google’s top model while running locally. The smaller models still fell short.

The preprint is on Zenodo, and I’m getting the paper ready to submit to the Journal of Emerging Investigators.