Benchmarks and Evaluation: How Do We Know If Omics AI Actually Works?
When a new foundation model in computational biology is released, the accompanying paper inevitably features tables of bolded numbers demonstrating state-of-the-art performance. Whether it is predicting protein structures or annotating single-cell data, the claims are often spectacular. But how do we truly know if these AI systems work in ways that matter to biology, rather than just optimizing arbitrary computational metrics? For the vision of Agentic Omics to become reality—where autonomous agents orchestrate models like AlphaFold and DNABERT-2 to drive drug discovery—we need a rigorous understanding of when these models succeed, when they hallucinate, and when their benchmarks deceive us. Claims of AI breakthroughs are only as strong as their evaluation methodologies. ...