Hugging Face Attempts to Reproduce 2,200 ICML Papers: What It Reveals About ML Research Practices
Reproducibility has long been a concern in machine learning research, where papers often lack the code, data, or computational details needed to verify published results. Hugging Face has undertaken a systematic effort to address this gap by attempting to reproduce 2,200 papers from ICML, one of the field's most prestigious venues.
The initiative represents a significant investment in research transparency. Rather than simply accepting published claims at face value, the Hugging Face team worked through papers methodically, attempting to replicate experiments and verify reported outcomes using the methods and resources described in each paper.
Large-scale reproducibility studies are rare in machine learning due to the substantial time, compute, and expertise required. By taking on this effort, Hugging Face has generated empirical data about how often published ML findings hold up under scrutiny—a topic that researchers and practitioners frequently discuss but rarely measure directly.
The findings from this project could inform how the research community evaluates papers, how venues approach reproducibility requirements, and how practitioners assess which results are robust enough to build upon.