Best Practices

Best Practice Guidelines

Best practices in reproducibility are a set of actions and principles that you can take in order to ensure that your work is reproducible. Best practices should not be considered a single, static recipe, but rather they are a flexible knowledge of processes and strategies that evolve over time.

To get started understanding best practices, we suggest that you take a look at the ACM Digital Library’s webpage entitled Software and Data Artifacts in the ACM Digital Library, which provides information on motivations for reproducibility and a look into how reproducibility is supported and continues to evolve in the ACM Digital library. From that page you can get more information on badges.

Researchers in the area of databases started explicitly emphasizing reproducibility early on. Much of what we propose to implement for ACM Multimedia Retrieval is inspired from their DOs and DON’Ts. We suggest that you take a look at the material on reproducibility that has been published in the database community and consider how their best practices transfer to your work.

A good source of information can be found in the ICDE 2008 tutorial by Ioana Manolescu and Stefan Manegold. The tutorial includes a road map of tips and tricks on how to organize and present code that performs experiments so that an outsider can repeat them.

A discussion about reproducibility in research including guidelines and a review of existing tools can be found in the SIGMOD 2012 tutorial by Juliana Freire, Philippe Bonnet, and Dennis Shasha.

Closer to multimedia retrieval, ACM MMSys and ACM MM have led the multimedia research community in explicitly emphasizing reproducibility and Gwendal Simon chaired the reproducibility track at MMSys 2019. Emailing Gwendal or reading what he wrote about reproducibility in his blog can be helpful. Take also a look at ACM MM’s reproducibility efforts that are documented here. Our approach follows very much the principles brought forward for ACM MM.

Examples of Badged Papers

Here are a few links to papers with badges inside the ACM DL. They have been picked because they nicely illustrate what is described in this page, not due to their scientific value. They come from multiple domains, giving you a broad view of what colleagues have done.

The first two papers from ACM MM 2019 comply with the guidelines specified on this page. The latter two do not necessarily comply with our rules. For example, sometimes there is no companion paper, sometimes the companion paper is not always a 2-4 page paper, ACM style. Furthermore, papers may target other types of badges that are about the availability of the artifacts, not solely about replicability or reproducibility of results.

Reproducibility Companion Paper: Knowledge Enhanced Neural Fashion Trend Forecasting. Yunshan Ma, Yujuan Ding, Xun Yang, Lizi Liao, Wai Keung Wong, Tat-Seng Chua, Jinyoung Moon, Hong-Han Shuai. ACM ICMR 2021. Paper’s DOI. The artifacts are on Zenodo.
Reproducibility Companion Paper: Human Object Interaction Detection via Multi-level Conditioned Network, Yunqing He, Xu Sun, Hui Jiang, Tongwei Ren, Gangshan Wu, Maria Sinziana Astefanoaei, Andreas Leibetseder. ACM ICMR 2022. Paper’s DOI. The artifacts are on GitHub.
Reproducible Experiments on Adaptive Discriminative Region Discovery for Scene Recognition, Z. Zhao, Z. Liu, M. Larson, A. Iscen, N. Nitta. ACM MM 2019. . Paper’s DOI. The artifacts are on GitHub.
Generating Preview Tables for Entity Graphs, N. Yan, S. Hasani, A. Asudeh and C. Li. Winner of the 2017 most reproducible papers from SIGMOD 2016. . Paper’s DOI.