While on a call today for the ACM Emerging Interest Group on Reproducibility and Replicability, I realized that for computational reproducibility to become pervasive, we need to solve a scalability problem. Today, a lot of computational reproducibility is checked by hand, whether this is in terms of artifact evaluation in a conference or a paper, or with post-publication checks.