Overall Reproducibility Objectives

Why consider reproducibility? Reproducibility of scientific results is at the core of the Scientific Method. The reason why we do research is to create new knowledge. This knowledge is built up over time by research findings that are confirmed because they are reproducible by the research community.

An experimental result is not fully established unless it can be independently reproduced.
» ACM DL. Read it here.

We aim to reach that goal through the use of a series of instruments deployed by the ACM that specifically address the reproducibility of scientific results. The long-term objective of ICMR Reproducibility is to promote a common research culture where sharing the full set of products of multimedia research (e.g., not just the algorithm description, but also the data, the code, the plots) is the norm rather than an exception. The main challenge we face is promoting research reproducibility both efficiently and effectively. To tackle this challenge, we must build technical expertise that can support repeatable and shareable research.

The ICMR Reproducibility Committee is here to help you with this.

Overall, our objectives are to:

  • Promote reproducibility as a strong foundation for scientific productivity and progress in the multimedia community,
  • Increase the impact of multimedia research by accelerating the dissemination and uptake of multimedia research results,
  • Establish a common understanding among researchers about the reproducibility of scientific results in order to communicate clearly and visibly the effort that authors have invested in ensuring that their findings can be recreated,
  • Support easier dissemination of code, data sets, experimental frameworks and other products of multimedia research.

ACM ICMR: Call for Reproducibility Papers

The authors who already have a paper accepted at ACM ICMR (as a regular long/short paper or a special session paper) can participate in the reproducibility review process. The authors are invited to prepare and submit a reproducibility paper. The reproducibility paper is a companion paper that is distinct from the main paper. The companion paper will be formally reviewed by the ICMR Reproducibility Committee or reviewers. If this paper passes the review process with success, then both the companion paper and the corresponding main paper are awarded reproducibility badges that are embedded in their .pdf. These two badged papers appear in the ACM Digital Library, and the companion paper is linked to the original paper. The authors of an accepted reproducibility paper present their work as part of a specific reproducibility poster session at ACM ICMR.

Please visit the page for information about the ACM ICMR 2024 call for reproducibility papers. You will find there the submission guidelines this year, the timeline, the paper badge, the description of the companion paper, and the artifacts associated with this companion paper.

ACM Reproducibility Badge

Reproducibility at ACM takes the concrete form of a reproducibility badge that is given to papers whose authors have invested effort to describe their algorithms and experiments in detail. Reproducibility badges are awarded to scientific contributions that have successfully passed a formal reproducibility reviewing phase. Contributions that have passed get embedded in their .pdf reproducibility badge, which is also prominently displayed in the ACM Digital Library. Having a reproducibility badge awarded to your contribution promotes the dissemination and uptake of your work. This helps the work receive wider recognition and is positioned to make a higher impact. When they see the badge, other researchers will understand whether it is possible to build on your contribution and whether there are resources available that will allow them to adopt or extend your approach.

ACM defines several ACM badges here, spanning several levels of reproducibility engagements, from light to strong. The reproducibility badges involve developing and releasing research artifacts. Quoting ACM:

By “artifact” we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.
» ACM DL. Read it here

Reproducibility Review Process

Submitted reproducibility companion papers are reviewed by a collaborative reproducibility review process, which is distinct from the main conference paper reviewing process. Please visit the page for detailed information about the Review Guidelines. It is the ICMR Reproducibility Committee that is carrying out the review process, determining whether or not a paper will receive a badge. The committee assigns reviewers to each reproducibility companion paper submission.

During the process of assessing the submission and possibly the artifacts, the reviewers interact with the submitting authors in order to work through any technical issues or gaps in the documentation that they find. The review process is fully open: the reviewers know the authors; the authors know the reviewers.

If it is possible to address all the issues, then the reviewers document the review process in a description which is added as a section to the reproducibility companion paper that is under review. Because the reviewers interact with the authors and devote a large amount of time and effort to assess your submission, and because they write a section in the final reproducibility companion paper, the reviewers become co-authors of the final reproducibility companion paper. If the reviewers find that the technical issues or problems with the documentation cannot be resolved, then the reproducibility companion paper is rejected. If, during the evaluation, a serious flaw invalidating the scientific results published in the original contribution is discovered, then the companion paper is rejected, and the reproducibility reviewers will encourage the original authors to publish an errata.

Note that the reproducibility companion paper is not published in the same ACM proceedings as the original paper but rather appears in the next year’s proceedings. This provides authors with the opportunity to present their work twice at the conference: The original paper is presented first, and the next year the reproducibility companion paper is presented as a poster. In this way, the work of authors who devote attention to reproducibility receives additional visibility at the conference.

ICMR Reproducibility Committee

The committee is chaired by two chairpersons who each serve for two years. At the end of each year, one chair is replaced and the other continues. This system ensures a one-year overlap, which helps the committee to maintain freshness while also facilitating the transmission of the gradually improved expertise in managing reproducibility.

The two chairpersons nominate the members of the committee who are enrolled for at least one year. Chairs can also take part in the evaluation. There is a strict conflict of interest policy: the chairs cannot submit a reproducibility paper while they hold their positions.

The committee in charge of reproducibility at the 2024 edition of ACM ICMR is provided in the page of ACM ICMR 2024 — Call for Reproducibility papers.

References

The process described in these pages was originally designed and implemented by Laurent Amsaleg, General Chair and Reproducibility Chair of ACM Multimedia 2019, with inputs from co-chairs Martha Larson, Benoit Huet, and Björn Þór Jónsson. The text here originated from the Web page for ACM MM Reproducibilityand was used with the authorization of the original authors. Naturally, however, that text was adapted when necessary. Also, see the credits on https://project.inria.fr/acmmmreproducibility.