Research collaboration

The tool works. The evaluation is too small. That is the part I need help with.

Last updated 19 August 2026

The ask, in one sentence: I am looking for a clinical collaborator to co-design a validation study of an eligibility screening tool, and specifically to help produce a clinician labelled gold standard set of eligibility criteria.

Everything else is built and measured. A labelled set is the one input I cannot generate myself, because the labels have to come from people who screen patients for a living.

What the tool is

It reads every study the ClinicalTrials.gov registry lists as recruiting for a condition, decides the criteria the registry publishes as structured fields, compiles what it can from the eligibility prose into predicates with negation and numeric thresholds, and quotes every remaining line verbatim rather than guessing at it. A solver then ranks the next question by how many still open studies it would settle.

Architecturally it is a hybrid neuro symbolic system with three inference layers, ordered by authority. The symbolic layer is deterministic and is the only one permitted to produce a verdict; the learned layers exist to improve recall and are constrained to surfacing rather than deciding. All three execute client side.

Rule compiler: Clause segmentation, a 35 concept lexicon, negation scoped to the clause, numeric extraction across four unit families. Returns byte offsets for every span it used. The only layer allowed to decide.
Statistical layer: One vs rest multi label classification by distant supervision: the symbolic layer labels 32,628 criteria and an L1 regularised logistic model over a 21,000 dimensional word and character n gram space learns to generalise past the lexicon. Isotonic calibration on a second disjoint fold, split conformal thresholds on a third, giving distribution free coverage. Says a line is about a concept, never that a patient passes it.
Semantic layer: Nearest centroid classification over sentence embeddings. all-MiniLM-L6-v2, int8 quantised, run through ONNX Runtime on WebAssembly in the tab. Prototypes are built from the symbolic layer's own positives, so it is few shot without any additional annotation. Off by default. Decides nothing.
Where it runs: Entirely client side. No prompt, no API call to a model, no server of ours. A criterion never leaves the browser, which is why there is no PHI to protect in the first place.

What is already measured

Every figure below is printed by a command in the repository and is reproducible. None of it is estimated.

Rule compiler: 100.0% precision, 95.7% recall
Calibration error: 0.0065 raw → 0.0008 isotonic, over 10 reliability bins
Conformal coverage: 92.9% against hand labels at a 90% target, mean prediction set 0.89 concepts of 32
Semantic layer: 95.6% precision, recovers the criteria the first two layers both missed
Compiler unit tests: 24 assertions on negation scoping, numeric extraction, polarity and abstention

Why those numbers are not enough

They are scored against 46 criteria labelled by hand, by one rater, who is also the author. That is a regression harness. It catches a change that breaks the compiler. It is not a validation study, and I have said so on the landing page rather than leaving it for a reader to discover.

Three problems with it, in order of severity:

One rater, who wrote the tool. There is no inter-rater agreement statistic because there is only one rater, and that rater is not blind to what the tool outputs.
46 items. Far too few to estimate performance per concept, and far too few for a confidence interval anyone should act on.
Not clinically sampled. The items were chosen to exercise the compiler, not drawn to represent what a coordinator actually reads.

The study I am proposing

Written out so you can tell me where it is wrong. I would expect this to change once someone who has run an accrual study looks at it.

SampleDraw roughly 400 to 600 criteria at random from currently recruiting studies, stratified across therapeutic areas and across inclusion and exclusion, so the set represents what gets read rather than what the compiler handles well.
LabelTwo or more raters with screening experience label each criterion for the concepts present and, where the criterion is decidable, the polarity and threshold. Raters are blind to the tool's output. A written codebook is fixed before labelling starts.
AgreementReport Cohen's or Fleiss' kappa on the raters before touching the tool. If humans do not agree with each other, no tool number means anything, and that finding is worth publishing on its own.
AdjudicateDisagreements resolved by a third rater or by consensus, with the rate of disagreement reported rather than smoothed away.
ScorePrecision, recall and F1 for the compiler; empirical conformal coverage for the statistical layer against these labels rather than against its own training signal; and the share of criteria no layer reads, which is the number I most expect to be uncomfortable.
Pre-registerThe analysis plan and the primary endpoint fixed before scoring, so the thresholds cannot be chosen after seeing the result.

The labelled set is the deliverable, not just an input. No public, clinician labelled corpus of eligibility criteria at this granularity exists that I have been able to find. If we build one and release it, it outlives this tool and anyone else working on the problem can use it. That is the part I think is genuinely publishable.

What I am asking for, concretely

Time: An hour to tell me the protocol above is wrong, then whatever labelling time you or a coordinator can spare. The labelling tool is mine to build, and I will build it around however you want to work.
Expertise: Whether the 35 concept lexicon reflects what actually rules patients out, and what it is missing.
Access, if it exists: A coordinator who screens patients and would be willing to say where the tool wastes their time.
Not asking for: Funding, patient data, PHI, or an IRB submission for the tool itself. The tool reads a public registry and never transmits a patient profile to Inclusion Health.

What you would get

Co-authorship on anything that comes out of it, with the order agreed before we start.
The labelled corpus, jointly released under whatever licence you prefer.
The tool free for your site, permanently, with or without a study. It is already free to use and always will be for patients and caregivers.
Every method above documented, reproducible from the repository, and open to your review before anything is published.

What this is not

Not a medical device, and not submitted as one. It reports no diagnosis, no advice and no probability of enrolment.
Not a recruitment claim. The literature on eligibility criteria length and accrual failure describes the problem. It says nothing about whether this tool helps, because that has not been studied. That is what I am asking you to help find out.
Not an LLM wrapper. There is no prompt in the product and no model API call. This matters for a study because the system is deterministic and versioned: the same input gives the same output, and a result stays reproducible.
Not asking you to endorse it. No name or institution goes anywhere near the site without written agreement, and a negative result gets published the same as a positive one.

Who is asking

I am Akshobya Rao, a high school student in Naperville, Illinois. I built Inclusion Health on my own: the rule compiler, the training pipeline, the calibration and the interface. Before this I spent a year on independent work in non-invasive diagnostics, which is where I got used to the idea that a measurement is worth nothing until you know how wrong it is allowed to be.

That is the whole reason I am writing to you. I can measure this system against my own labels, and I have. What I cannot do alone is find out whether it is any use to a person trying to find a trial, or to the coordinator who screens them afterwards. That question needs someone who has done the screening. I am looking for a few hours of your judgement on the study design, not an endorsement of the software.

Where to start

The fastest way to judge this is to use it for two minutes on a condition you know, then open a study and look at what it refused to decide. The abstentions say more about the tool than the verdicts do.

Open the tool Read the method Email me

Everything is reproducible from source, including the training pipeline and both evaluation harnesses. Happy to send the repository to anyone considering this.

The reference behind the problem statement

Peterson JS, Plana D, Bitterman DS, Johnson SB, Aerts HJWL, Kann BH. Growth in eligibility criteria content and failure to accrue among National Cancer Institute (NCI) affiliated clinical trials. Cancer Med. 2023;12(4):4715–4724. doi:10.1002/cam4.5276. PMID 36398619.

Median unique content words in eligibility criteria rose 95% between 2008 and 2018, from 214 to 416. Of 1,197 trials, 231 (19.3%) failed due to low accrual, and the failure rate rose from 11.8% in the shortest decile of criteria to 29.4% in the longest. Cited to describe the problem. It is not evidence about this tool.