94 online 3 members  + Join RP Base  +
FreeScript

PhysDox

by phish-tech  ·  version 2026.11.12  ·  updated 3 days ago

No reviews yet

Downloads
0
Reviews
0
Size
2.1 MB
Code check
Clean
Licence
MIT
Released
Nov 2026

PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols

PhysDox
PhysDox: Benchmarking LLMs on Physical Feasibility Auditing of Physiological Sensing Protocols
He Liu · Boyuan Gu · Shuaiqi Cheng · Haiyang Sun · Siyu You · Xuming Hu

Why · Try it · Example · Results · Data · Use your model · Cite

Why PhysDox
An LLM can write an experimental protocol that reads perfectly and still cannot work:
the sampling rate is below the Nyquist limit, a metal plate sits between the radar and the
subject, or a healthy volunteer is given an arterial catheter. We call this a Physical Paradox.
PhysDox measures whether a model can catch it.

Try it in two minutes
No API key needed. This runs the real pipeline (data download, prompting, parsing, metrics) against a
fake model so you can check your setup first.

git clone https://github.com/phish-tech/PhysDox.git && cd PhysDox
uv sync                                          # Python 3.11+, installs the package and the physdox-eval command

uv run python examples/mock_llm_server.py &      # fake OpenAI-compatible endpoint on :8765 (or run it in a second terminal)
export LLM_API_KEY=dummy LLM_BASE_URL=http://127.0.0.1:8765/v1 LLM_MODEL_NAME=mock
uv run physdox-eval --data gold_dev --model custom --stage1-only

On Windows PowerShell, set the variables with $env:LLM_API_KEY="dummy" and so on.

You get a result table like this (the mock answers at random, so the scores mean nothing):

=== PhysDox result ===
model: mock   strategy: zero-shot   items: 100
Stage 1  accuracy 34.0%   balanced acc 28.4%   macro-F1 28.3%
label              precision   recall      f1  support
valid                  28.6%    23.5%   25.8%       34
minor_issue             6.7%    18.2%    9.8%       11
fatal_violation        57.1%    43.6%   49.5%       55
confusion (rows = truth, columns = prediction):
                        valid  minor_iss  fatal_vio  __missing
valid                       8         10         16          0
minor_issue                 7          2          2          0
fatal_violation            13         18         24          0

Evaluate your own model
Point the same command at any OpenAI-compatible endpoint (OpenAI, DeepSeek, vLLM, Ollama, a gateway, ...):

export LLM_API_KEY=sk-...  LLM_BASE_URL=https://api.openai.com/v1  LLM_MODEL_NAME=gpt-4o

uv run physdox-eval --data gold_dev --model custom --stage1-only        # 100 items, a few minutes
uv run physdox-eval --data silver   --model custom --max-items 500 --sample-seed 0   # random 500 of Silver, both stages
uv run physdox-eval --data silver   --model custom --strategy cot       # full Silver, chain-of-thought

Option, Meaning

--data, silver (5,000), gold_dev (100), gold (683, needs access), or a JSON path/URL. Downloaded and cached automatically
--strategy, zero-shot (default), cot, self-cons, tool-aug
--stage1-only, Skip diagnosis and only classify severity
--max-items N --sample-seed S, Evaluate a random subset of N items
--concurrency, --rpm, Parallel requests and rate limit. Lower them if your provider returns 429/502

Runs write summary.json, per_item_results.jsonl, and a README.md under experiments/. Pass
--output-dir to make an interrupted run resumable. The result table above is printed at the end.
What an item looks like
Every item is a full protocol (equipment, placement, acquisition, processing). This one is from the
public Gold dev subset:

Radar vital-sign monitoring with three FMCW radars (24, 60, and 77 GHz).
"Chirp repetition frequency set to 1 kHz … Sampling rate: 0.1 MSPS" …

Label: fatal_violation · Category: Signal & Sampling · Constraint: sampling_theorem

Why: the annotation computes a beat frequency of about 80 kHz for a target at 3 m, which needs
at least 160 kHz sampling. 100 kSPS aliases, so range and vital-sign extraction cannot work.

A model that reads the protocol as a well-organized radar study and answers valid has fallen for the
paradox. Stage 1 asks for valid, minor_issue, or fatal_violation. Stage 2 asks which constraint
(3 coarse categories, 12 fine ones) the fatal protocol violates.
At a glance

Gold set, 683 expert-curated protocols (232 valid, 74 minor issue, 377 fatal violation)
Silver set, 5,000 protocols with balanced constraint coverage (1,500 / 1,250 / 2,250)
Domains, EEG/BCI, PPG blood pressure, IMU motion, radar vital signs, sEMG gesture, throat-mic speech
Evaluated, 6 LLM variants × 4 inference strategies × 2 splits = 48 runs
Best result, Stage-1 macro-F1 of only 53.0 on Gold

Results and what they mean
All numbers below come from the 48 released runs. Charts are rebuilt by
scripts/make_result_figures.py from the public result tables.
1. Severity detection is hard, and reasoning tricks do not help
The best Gold macro-F1 is 53.0, against about 31 for a uniform random guess. Five of six models score
lower on Gold than on Silver at their best strategy (DeepSeek V4 Flash is about equal), so the
expert-written protocols are harder than the expanded ones. Averaged over models, the four strategies
land within 2.2 points of each other on Gold (44.4 for CoT to 46.6 for self-consistency). Thinking longer
or sampling more does not fix a missing sense of physical feasibility.
2. Models are wrong in different directions
Similar macro-F1 hides opposite behavior. GPT-5.5 recognizes only 2% of the valid Gold protocols and
flags nearly everything as flawed, which also gives it the highest fatal recall (80%). Both MiMO models
approve about 70% of the valid protocols but catch only 37–41% of the fatal ones. Claude Opus 4.7 is the
most balanced. A single accuracy number would hide this, so report per-class recall, not just macro-F1.
3. Errors cascade into diagnosis
If a model is told which protocols are fatal, it names the right 3-class category 40–59% of the time.
Run end-to-end, so that fatal protocols missed in Stage 1 count as wrong, accuracy falls

…