HotpotBio is the data lab and research group of Hotpot.ai dedicated to biomedicine.

Our lab philosophy is to advance AI models without Internet-scale data. While supermodels have dominated since GPT-3, the smart model paradigm is finally shifting from fringe to credible. Labs like ours have long believed the future of AI will mirror the computing industry, where supercomputers tackle the most complex cases, but smartphones serve billions of people.

Unable to publish commercial research, we established HotpotBio to advance science in other ways. We draw inspiration from open source, where ephemeral teams innovate by attracting talent across organizational boundaries. Since biomedicine is characterized by sparse data and evolving facts, the field presents a high-impact opportunity for validating hypotheses aligned with our lab vision. Furthermore, ML research resembles biomedical research more than most realize.

Just as findings in one patient may not generalize to others due to genetic and lifestyle differences, ML findings -- even on core parameters like learning rate -- may not generalize due to architecture and training differences. In both, generalizability is far weaker than in physics or mathematics.

Both cancer and ML research are constrained by data quality. Central to our effort is rethinking biomedical datasets and training approaches in clinical reasoning, oncology, neuroimmunology, drug development, and other specialty areas.

While some data errors are tolerable for general ML models, uncommon variants in biomedicine may drive pathology. Training on imprecise medical information may cause misdiagnosis, clinical errors, flawed protein designs, or drug candidates with elevated risk of adverse events.

Complicating matters, evolving medical facts may invalidate training data and model knowledge. What was true last year may be false today. For instance, in April 2024 the U.S. Preventive Services Task Force updated its longstanding guidance and now urges biennial mammograms starting at age 40 -- down from the previous benchmark of 50 -- for average-risk women, citing rising breast-cancer incidence in younger patients.

Accurate annotation of medical data is challenging and demands expert verification based on the latest information. Even Google DeepMind's 2024 relabeling effort for MedQA contained errors, which could produce subtle failures and model hallucinations if not addressed.

Ultimately, HotpotBio seeks to advance biomedical research by:

1. Developing open-source frameworks and tools to help patients, researchers, and clinicians leverage AI
2. Publishing papers to highlight where the current understanding in cancer and other diseases may be incomplete due to methodological gaps
3. Building benchmarks to measure AGI progress in healthcare and general reasoning
4. Developing healthcare and drug discovery models

These tools are available for free in GPT, Claude, and Gemini.

• Patient.md: simplifies case sharing for second opinions, promotes patient autonomy, and provides a foundation for clinical trial matching by providing a way to organize medical data for AI assistants.

• Conclusion Checker: helps readers understand a study and identify which conclusions may not generalize to real patients, real-world settings, or broader populations. This highlights concrete limitations in study design, sampling, measurement, and external validity.