Research collaboration

The tool works. The evaluation is too small. That is the part we need help with.

Last updated 28 August 2026

The ask, in one sentence: We are looking for a clinical collaborator to co-design a validation study of an eligibility screening tool, and specifically to help produce a clinician labelled gold standard set of eligibility criteria.

Everything else is built and measured. A labelled set is the one input we cannot generate ourselves, because the labels have to come from people who screen patients for a living.

If a study is more than you want to take on

The protocol below is the full version. Most of what we need from a first reader is far smaller than that, and any one of these three is worth more to us than nothing. They are listed shortest first.

One sentence: Whether this is worth continuing, in your words. We would quote you, or not, however you prefer.
Ten minutes: A call telling us where the method is wrong, or where it could be dangerous for a patient. No preparation, no slides.
Twenty minutes: Open the tool, search a condition you know well, and press Labelling sheet. It exports every criterion the tool declined to answer, deduplicated across studies, with blank columns for a verdict and a confidence. Thirty rows is about twenty minutes. Send it back and we will publish what it says about us, including the parts that are bad. The tool now reads a filled sheet as well as writes one: it reports what share of the lines it refused you decided anyway, split by the reason it gave, and Cohen's kappa between raters if more than one sheet comes back.

The reason the twenty minute version matters most: abstention is the central claim of the design, and we cannot tell from the inside whether the tool declines where a clinician would also hesitate, or whether it is simply avoiding the difficult lines. Only someone who reads criteria professionally can tell us which.

What the tool is

It reads every study the ClinicalTrials.gov registry lists as recruiting for a condition, decides the criteria the registry publishes as structured fields, compiles what it can from the eligibility prose into predicates with negation and numeric thresholds, and quotes every remaining line verbatim rather than guessing at it. A solver then ranks the next question by how many still open studies it would settle.

Architecturally it is a hybrid neuro symbolic system with three inference layers, ordered by authority. The symbolic layer is deterministic and is the only one permitted to produce a verdict; the learned layers exist to improve recall and are constrained to surfacing rather than deciding. All three execute client side.

Rule compiler: Clause segmentation, a 35 concept lexicon, negation scoped to the clause, numeric extraction across four unit families. Returns byte offsets for every span it used. The only layer allowed to decide.
Statistical layer: One vs rest multi label classification by distant supervision: the symbolic layer labels 32,628 criteria and an L1 regularised logistic model over a 21,000 dimensional word and character n gram space learns to generalise past the lexicon. Isotonic calibration on a second disjoint fold, split conformal thresholds on a third, giving distribution free coverage. Says a line is about a concept, never that a patient passes it.
Semantic layer: Nearest centroid classification over sentence embeddings. all-MiniLM-L6-v2, int8 quantised, run through ONNX Runtime on WebAssembly in the tab. Prototypes are built from the symbolic layer's own positives, so it is few shot without any additional annotation. Off by default. Decides nothing.
Where it runs: Entirely client side. No prompt, no API call to a model, no server of ours. A criterion never leaves the browser, which is why there is no PHI to protect in the first place.

What is already measured

Every figure below is printed by a command in the repository and is reproducible. None of it is estimated.

Rule compiler: 100.0% precision, 95.7% recall
Calibration error: 0.0065 raw → 0.0008 isotonic, over 10 reliability bins
Conformal coverage: 92.9% against hand labels at a 90% target, mean prediction set 0.89 concepts of 32
Semantic layer: 95.6% precision, recovers the criteria the first two layers both missed
Compiler unit tests: 24 assertions on negation scoping, numeric extraction, polarity and abstention

Why those numbers are not enough

They are scored against 46 criteria labelled by hand, by one of us, who is also the author. That is a regression harness. It catches a change that breaks the compiler. It is not a validation study, and we have said so on the landing page rather than leaving it for a reader to discover.

Three problems with it, in order of severity:

One rater, who wrote the tool. There is no inter-rater agreement statistic because there is only one rater, and that rater is not blind to what the tool outputs.
46 items. Far too few to estimate performance per concept, and far too few for a confidence interval anyone should act on.
Not clinically sampled. The items were chosen to exercise the compiler, not drawn to represent what a coordinator actually reads.

The study we are proposing

Written out so you can tell us where it is wrong. We would expect this to change once someone who has run an accrual study looks at it.

SampleDraw roughly 400 to 600 criteria at random from currently recruiting studies, stratified across therapeutic areas and across inclusion and exclusion, so the set represents what gets read rather than what the compiler handles well.
LabelTwo or more raters with screening experience label each criterion for the concepts present and, where the criterion is decidable, the polarity and threshold. Raters are blind to the tool's output. A written codebook is fixed before labelling starts.
AgreementReport Cohen's or Fleiss' kappa on the raters before touching the tool. If humans do not agree with each other, no tool number means anything, and that finding is worth publishing on its own.
AdjudicateDisagreements resolved by a third rater or by consensus, with the rate of disagreement reported rather than smoothed away.
ScorePrecision, recall and F1 for the compiler; empirical conformal coverage for the statistical layer against these labels rather than against its own training signal; and the share of criteria no layer reads, which is the number we most expect to be uncomfortable.
Pre-registerThe analysis plan and the primary endpoint fixed before scoring, so the thresholds cannot be chosen after seeing the result.

The labelled set is the deliverable, not just an input. No public, clinician labelled corpus of eligibility criteria at this granularity exists that we have been able to find. If we build one and release it, it outlives this tool and anyone else working on the problem can use it. That is the part we think is genuinely publishable.

What we are asking for, concretely

Time: An hour to tell us the protocol above is wrong, then whatever labelling time you or a coordinator can spare. The labelling tool is ours to build, and we will build it around however you want to work.
Expertise: Whether the 35 concept lexicon reflects what actually rules patients out, and what it is missing.
Access, if it exists: A coordinator who screens patients and would be willing to say where the tool wastes their time.
Not asking for: Funding, patient data, PHI, or an IRB submission for the tool itself. The tool reads a public registry and never transmits a patient profile to Inclusion Health.

What you would get

Co-authorship on anything that comes out of it, with the order agreed before we start.
The labelled corpus, jointly released under whatever licence you prefer.
The tool free for your site, permanently, with or without a study. It is already free to use and always will be for patients and caregivers.
Every method above documented, reproducible from the repository, and open to your review before anything is published.

What this is not

Not a medical device, and not submitted as one. It reports no diagnosis, no advice and no probability of enrolment.
Not a recruitment claim. The literature on eligibility criteria length and accrual failure describes the problem. It says nothing about whether this tool helps, because that has not been studied. That is what we are asking you to help find out.
Not an LLM wrapper. There is no prompt in the product and no model API call. This matters for a study because the system is deterministic and versioned: the same input gives the same output, and a result stays reproducible.
Not asking you to endorse it. No name or institution goes anywhere near the site without written agreement, and a negative result gets published the same as a positive one.

Who is asking

We are Akshobya Rao and Abhinav Byju, juniors at Neuqua Valley High School in Naperville, Illinois, and the two co-founders of Inclusion Health. We built the whole thing ourselves: the rule compiler, the training pipeline, the calibration and the interface. Before this we spent a year on independent work in non-invasive diagnostics, which is where we got used to the idea that a measurement is worth nothing until you know how wrong it is allowed to be.

That is the whole reason we are writing to you. We can measure this system against our own labels, and we have. What we cannot do alone is find out whether it is any use to a person trying to find a trial, or to the coordinator who screens them afterwards. That question needs someone who has done the screening. We are looking for a few hours of your judgement on the study design, not an endorsement of the software.

Where to start

The fastest way to judge this is to use it for two minutes on a condition you know, then open a study and look at what it refused to decide. The abstentions say more about the tool than the verdicts do.

Two longer documents sit behind this page. The method note describes the architecture, the training procedure, the evaluation and the failure modes, and is written for a methodologist. The partnership brief sets out the study we propose, the governance position, and exactly what a collaboration would involve for your institution.

Open the tool Read the method note Email us

Everything is reproducible from source, including the training pipeline and both evaluation harnesses. Happy to send the repository to anyone considering this.

The reference behind the problem statement

Peterson JS, Plana D, Bitterman DS, Johnson SB, Aerts HJWL, Kann BH. Growth in eligibility criteria content and failure to accrue among National Cancer Institute (NCI) affiliated clinical trials. Cancer Med. 2023;12(4):4715–4724. doi:10.1002/cam4.5276. PMID 36398619.

Median unique content words in eligibility criteria rose 95% between 2008 and 2018, from 214 to 416. Of 1,197 trials, 231 (19.3%) failed due to low accrual, and the failure rate rose from 11.8% in the shortest decile of criteria to 29.4% in the longest. Cited to describe the problem. It is not evidence about this tool.