FreeScript
PhysDox
by phish-tech · version 2026.11.12 · updated 3 days ago
No reviews yet
- Downloads
- 0
- Reviews
- 0
- Size
- 2.1 MB
- Code check
- Clean
- Licence
- MIT
- Released
- Nov 2026
PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols
PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols
He Liu · Boyuan Gu · Shuaiqi Cheng · Haiyang Sun · Siyu You · Xuming Hu
Why · Try it · Example · Results · Data · Use your model · Cite
Why PhysDox
An LLM can write an experimental protocol that reads perfectly and still cannot work:
the sampling rate is below the Nyquist limit, a metal plate sits between the radar and the
subject, or a healthy volunteer is given an arterial catheter. We call this a Physical Paradox.
PhysDox measures whether a model can catch it.
Try it in two minutes
No API key needed. This runs the real pipeline (data download, prompting, parsing, metrics) against a
fake model so you can check your setup first.
On Windows PowerShell, set the variables with
You get a result table like this (the mock answers at random, so the scores mean nothing):
Evaluate your own model
Point the same command at any OpenAI-compatible endpoint (OpenAI, DeepSeek, vLLM, Ollama, a gateway, ...):
Option, Meaning
Runs write
What an item looks like
Every item is a full protocol (equipment, placement, acquisition, processing). This one is from the
public Gold dev subset:
paradox. Stage 1 asks for
(3 coarse categories, 12 fine ones) the fatal protocol violates.
At a glance
Gold set, 683 expert-curated protocols (232 valid, 74 minor issue, 377 fatal violation)
Silver set, 5,000 protocols with balanced constraint coverage (1,500 / 1,250 / 2,250)
Domains, EEG/BCI, PPG blood pressure, IMU motion, radar vital signs, sEMG gesture, throat-mic speech
Evaluated, 6 LLM variants × 4 inference strategies × 2 splits = 48 runs
Best result, Stage-1 macro-F1 of only 53.0 on Gold
Results and what they mean
All numbers below come from the 48 released runs. Charts are rebuilt by
1. Severity detection is hard, and reasoning tricks do not help
The best Gold macro-F1 is 53.0, against about 31 for a uniform random guess. Five of six models score
lower on Gold than on Silver at their best strategy (DeepSeek V4 Flash is about equal), so the
expert-written protocols are harder than the expanded ones. Averaged over models, the four strategies
land within 2.2 points of each other on Gold (44.4 for CoT to 46.6 for self-consistency). Thinking longer
or sampling more does not fix a missing sense of physical feasibility.
2. Models are wrong in different directions
Similar macro-F1 hides opposite behavior. GPT-5.5 recognizes only 2% of the valid Gold protocols and
flags nearly everything as flawed, which also gives it the highest fatal recall (80%). Both MiMO models
approve about 70% of the valid protocols but catch only 37–41% of the fatal ones. Claude Opus 4.7 is the
most balanced. A single accuracy number would hide this, so report per-class recall, not just macro-F1.
3. Errors cascade into diagnosis
If a model is told which protocols are fatal, it names the right 3-class category 40–59% of the time.
Run end-to-end, so that fatal protocols missed in Stage 1 count as wrong, accuracy falls
…
He Liu · Boyuan Gu · Shuaiqi Cheng · Haiyang Sun · Siyu You · Xuming Hu
Why · Try it · Example · Results · Data · Use your model · Cite
Why PhysDox
An LLM can write an experimental protocol that reads perfectly and still cannot work:
the sampling rate is below the Nyquist limit, a metal plate sits between the radar and the
subject, or a healthy volunteer is given an arterial catheter. We call this a Physical Paradox.
PhysDox measures whether a model can catch it.
Try it in two minutes
No API key needed. This runs the real pipeline (data download, prompting, parsing, metrics) against a
fake model so you can check your setup first.
git clone https://github.com/phish-tech/PhysDox.git && cd PhysDox
uv sync # Python 3.11+, installs the package and the physdox-eval command
uv run python examples/mock_llm_server.py & # fake OpenAI-compatible endpoint on :8765 (or run it in a second terminal)
export LLM_API_KEY=dummy LLM_BASE_URL=http://127.0.0.1:8765/v1 LLM_MODEL_NAME=mock
uv run physdox-eval --data gold_dev --model custom --stage1-onlyOn Windows PowerShell, set the variables with
$env:LLM_API_KEY="dummy" and so on.You get a result table like this (the mock answers at random, so the scores mean nothing):
=== PhysDox result ===
model: mock strategy: zero-shot items: 100
Stage 1 accuracy 34.0% balanced acc 28.4% macro-F1 28.3%
label precision recall f1 support
valid 28.6% 23.5% 25.8% 34
minor_issue 6.7% 18.2% 9.8% 11
fatal_violation 57.1% 43.6% 49.5% 55
confusion (rows = truth, columns = prediction):
valid minor_iss fatal_vio __missing
valid 8 10 16 0
minor_issue 7 2 2 0
fatal_violation 13 18 24 0Evaluate your own model
Point the same command at any OpenAI-compatible endpoint (OpenAI, DeepSeek, vLLM, Ollama, a gateway, ...):
export LLM_API_KEY=sk-... LLM_BASE_URL=https://api.openai.com/v1 LLM_MODEL_NAME=gpt-4o
uv run physdox-eval --data gold_dev --model custom --stage1-only # 100 items, a few minutes
uv run physdox-eval --data silver --model custom --max-items 500 --sample-seed 0 # random 500 of Silver, both stages
uv run physdox-eval --data silver --model custom --strategy cot # full Silver, chain-of-thoughtOption, Meaning
--data, silver (5,000), gold_dev (100), gold (683, needs access), or a JSON path/URL. Downloaded and cached automatically--strategy, zero-shot (default), cot, self-cons, tool-aug--stage1-only, Skip diagnosis and only classify severity--max-items N --sample-seed S, Evaluate a random subset of N items--concurrency, --rpm, Parallel requests and rate limit. Lower them if your provider returns 429/502Runs write
summary.json, per_item_results.jsonl, and a README.md under experiments/. Pass--output-dir to make an interrupted run resumable. The result table above is printed at the end.What an item looks like
Every item is a full protocol (equipment, placement, acquisition, processing). This one is from the
public Gold dev subset:
Radar vital-sign monitoring with three FMCW radars (24, 60, and 77 GHz).A model that reads the protocol as a well-organized radar study and answers
"Chirp repetition frequency set to 1 kHz … Sampling rate: 0.1 MSPS" …
Label:fatal_violation· Category: Signal & Sampling · Constraint:sampling_theorem
Why: the annotation computes a beat frequency of about 80 kHz for a target at 3 m, which needs
at least 160 kHz sampling. 100 kSPS aliases, so range and vital-sign extraction cannot work.
valid has fallen for theparadox. Stage 1 asks for
valid, minor_issue, or fatal_violation. Stage 2 asks which constraint(3 coarse categories, 12 fine ones) the fatal protocol violates.
At a glance
Gold set, 683 expert-curated protocols (232 valid, 74 minor issue, 377 fatal violation)
Silver set, 5,000 protocols with balanced constraint coverage (1,500 / 1,250 / 2,250)
Domains, EEG/BCI, PPG blood pressure, IMU motion, radar vital signs, sEMG gesture, throat-mic speech
Evaluated, 6 LLM variants × 4 inference strategies × 2 splits = 48 runs
Best result, Stage-1 macro-F1 of only 53.0 on Gold
Results and what they mean
All numbers below come from the 48 released runs. Charts are rebuilt by
scripts/make_result_figures.py from the public result tables.1. Severity detection is hard, and reasoning tricks do not help
The best Gold macro-F1 is 53.0, against about 31 for a uniform random guess. Five of six models score
lower on Gold than on Silver at their best strategy (DeepSeek V4 Flash is about equal), so the
expert-written protocols are harder than the expanded ones. Averaged over models, the four strategies
land within 2.2 points of each other on Gold (44.4 for CoT to 46.6 for self-consistency). Thinking longer
or sampling more does not fix a missing sense of physical feasibility.
2. Models are wrong in different directions
Similar macro-F1 hides opposite behavior. GPT-5.5 recognizes only 2% of the valid Gold protocols and
flags nearly everything as flawed, which also gives it the highest fatal recall (80%). Both MiMO models
approve about 70% of the valid protocols but catch only 37–41% of the fatal ones. Claude Opus 4.7 is the
most balanced. A single accuracy number would hide this, so report per-class recall, not just macro-F1.
3. Errors cascade into diagnosis
If a model is told which protocols are fatal, it names the right 3-class category 40–59% of the time.
Run end-to-end, so that fatal protocols missed in Stage 1 count as wrong, accuracy falls
…