See the overall guidelines for ICMR Reproducibility here.

ICMR Reproducibility Committee 2025

Chairs:

• Bo Wu, MIT-IBM Watson AI Lab, USA
• Qiushi Huang, University of Surrey, United Kingdom

Committee members:

• TBA

Dates

The important dates along the timeline for submitting the companion paper and the associated artifacts are:

• March 30, 2025: submission due for the paper and reproducibility archive
• April 5, 2025: notification of acceptance/rejection (same deadline as for regular papers),
• April 14, 2025: deadline for preparing the final version of the accepted companion paper (same deadline as for regular papers),

ICMR 2025 - Call for Papers

This page gives the instructions for submitting reproducibility companion papers for the ACM ICMR 2025. Such a submission has essentially two parts:

• Companion Paper: The companion paper is 2–3.5 pages in length (with an optional page for references). It contains a high-level description of the experiments carried out in the original paper and that are implemented in the archive.

• Archive: Contains the artifacts (e.g., code, scripts, data sets, protocols), which are finalized and packaged, ready to download and reproduce the results in the paper. It contains detailed readme file(s), examples, and all information needed to successfully carry out the experiments.

These instructions are split into sections that detail:

• ICMR Reproducibility Committee 2025
• Dates
• Submission Guidelines
• Paper Badge
• Contents of the Companion Paper
• Contents of the Archive of Artifacts
  • About Experiments
  • About Graphs and Plots
  • Ideal archive of artifacts
• Packaging Guidelines
  • Environment
  • System
  • Tools

Submission System

1. Registration & Submission: please register at ICMR 2025 CMT, select Reproducibility on the left top option bar to enter the ICMR Repro 2025 Submission Webpage.

1. Reproducibility Paper (2-3.5 pages in standard ICMR format): A high-level description of the original experiments and their implementation in the submitted archive.
2. Archive: A cleanly organized and packaged git repository containing code, scripts, datasets, and protocols, along with detailed README files and instructions for reproducing the results.

For best practices in preparing your submission, please refer to: https://icmr-reproducibility.github.io/website/best_practices

Please select Reproducibility Track for submission.

---

Submission Guidelines

Authors that have a regular long/short paper published at academic conferences during 2023-2025 are invited to submit a short reproducibility companion paper to the ICMR Reproducibility track at ACM ICMR 2025. That companion paper typically focuses on the technical details of what you published at previous publications.

The companion paper should be submitted as a short paper that is 2–3.5 pages long, excluding references. It must follow the standard ACM style format, double column. It has to involve a majority of the authors of the original paper and provide in the clear their names and affiliations.

The reproducibility companion paper and the associated artifacts will undergo a reproducibility review, which will result in an accept or a reject decision. Rejected papers are not disclosed.

If accepted, a Results Reproduced ACM badge is added to the original paper and to the companion paper, which are both stored in the ACM Digital Library, together with the artifacts. A badged companion paper will appear in the ACM ICMR 2025 proceedings and will be presented as a poster in the ACM ICMR 2025 Reproducibility poster session. The reviewers of the badged companion paper add a section documenting their efforts and become co-authors of the paper. The final version may be up to 4 pages (with an optional page for references).

If, during the evaluation, a serious flaw invalidating the scientific results published in the original contribution is discovered, then the companion paper is rejected, and the authors are encouraged to publish an errata.

Note: If your original paper turns out to be especially challenging to reproduce, the committee may also recommend that the companion paper be published in the next year of ACM ICMR, instead of ACM ICMR 2025. If replication cannot be completed in time for the next year of ACM ICMR too, then the paper cannot be accepted.

---

Paper Badge

We are committed to setting the standard high for reproducibility at ACM ICMR, and the Reproducibility Committee asks authors to target a top-quality badge for the companion papers they submit. The target is the Results Reproduced badge:

ACM gives the following definition of this Results Reproduced badge:

The main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.
» ACM DL. Read it here.

---

Contents of the Companion Paper

The companion paper must provide means for the committee to download the artifacts that will enable replicating the findings that are in the original paper contribution. The author-created artifacts that are relevant to this paper must have been placed on a publicly accessible archival repository (e.g. GitHub).

The companion paper must describe the procedure allowing reviewers to easily find their way in the artifacts. That procedure might for example explain the reasons for having organized the artifacts in hierarchies of folders. It might specify what items inside the artifacts must be read, and in which order, what are the main elements to fully review. It might clearly establish relationships between items in the artifacts and the corresponding elements that can be found in the original scientific contribution. It might explain and justify why this or that part of the original paper is not reproducible. The paper should indicate the expected duration of running the whole experimental pipeline.

Ideally, the companion paper should include text/schemas/illustrations that describe what the artifacts contain and how they should be deployed and then used. The companion paper should also contain notes about parameters that can be set or adjusted and about how to recreate the plots. It has to have examples, with comments. You can of course create documents in the archive that provide additional text/schemas/illustrations. In this case, the companion paper should clearly indicate where that can be found in the artifacts.

---

Contents of the Archive of Artifacts

Replicability is grounded in code, scripts, and datasets that you provide and that form so called “artifacts”. More formally, ACM defines artifacts as follows:

By “artifact” we mean a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.
» ACM DL. Read it here.

Artifacts contain digital objects that supplement the companion paper. Artifacts are typically a series of files, possibly organized according to a clear and easy-to-grasp hierarchy. Artifacts include for example:

1. The configuration files and scripts to set up and deploy the environment needed for the reviewers to subsequently run your code,
2. The source code if you expose your system as a white box,
3. Input Data: Either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided,
4. The set of experiments (system configuration and initialization, scripts, workload, measurement protocol, …) used to run the experiments that produce the raw experimental data,
5. The scripts needed to transform the raw experimental data into the graphs, tables, plots, …, that can be found in the original submission already published in the proceedings of ACM ICMR. All this material should be extensively described, documented, commented, and easy to understand, for example in specific files coming together with what they describe.

Note: We strongly recommend exposing your system as a white box so that anyone can reproduce and reuse your results. If the system is only made accessible as a black box, however, reviewers may still require confidential access to source code, such that they can properly assess the validity of the reproducibility results.

About Experiments

The central results and claims of the corresponding published paper should be supported by the submitted experiments, meaning we can recreate result data and graphs that demonstrate similar behavior to that shown in that paper. Typically when the results are about response times, we do not expect to get identical results. Instead, we expect to see that the overall behavior matches the conclusions from the paper, e.g., that a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system.

Given a system, the authors should provide the complete set of experiments to reproduce the results that are in the original paper. Typically, each experiment will consist of the following parts.

• A setup phase where parameters are configured and data is loaded,
• A running phase where a workload is applied and measurements are taken,
• A clean-up phase where the system is prepared to avoid interference with the next round of experiments.

The authors should document (i) how to perform the setup, running and clean-up phases, and (ii) how to check that these phases are complete as they should. The authors should document the expected effect of the setup phase (e.g., a cold file cache is enforced) and the different steps of the running phase, e.g., by documenting the combination of command line options used to run a given experiment script.

Each experiment should be automatic, e.g., via a script that takes a range of values for each experiment parameter as arguments, rather than manual, e.g., via a script that must be edited so that a constant takes the value of a given experiment parameter.

We do not expect the authors to perform any additional experiments on top of the ones in their original paper. Any additional experiments submitted will be considered and tested but they are not required.

About Graphs and Plots

For each graph/plot in the original paper, the authors should describe how the graph/plot is obtained from the experimental measurements. The submission should contain the scripts (or spreadsheets) that are used to generate the graphs/plots. We strongly encourage authors to provide scripts for all their graphs using a tool such as Gnuplot or Matplotlib. Here are two useful tutorials for Gnuplot: a brief manual and tutorial, and a tutorial with details about creating eps figures and embed them using LaTeX and another two for Matplotlib: examples from SciPy, and a step-by-step tutorial discussing many features.

Similar procedures must be provided by the authors in order to create the tables that are in the original paper.

Packaging Guidelines

These packaging guidelines are meant to cover general cases. Please keep in mind that every individual case is slightly different.

Environment

Authors should explicitly specify the operating system and tools that should be installed as the environment. Such specification should include dependencies with specific hardware features (e.g., 25 GB of RAM are needed) or dependencies within the environment (e.g., the compiler that should be used must be run with a specific version of the operating system).

Note: The submitted artifacts should not require specific hardware and software that are hard to get access to. The committee will try to do the best to find reviewers who can setup the required environments; however, if no reviewer can be found, the companion paper cannot be accepted.

System

System setup is one of the most challenging aspects when replicating experiments. System setup will be easier to conduct if it is automatic rather than manual. Authors should test that the system they distribute can actually be installed in a new environment. The documentation should detail every step in the system setup:

• How to obtain the system?
• How to configure the environment if need be (e.g., environment variables, paths)?
• How to compile the system? (existing compilation options should be mentioned)
• How to use the system? (What are the configuration options and parameters to the system?)
• How to make sure that the system is installed correctly?

The above tasks should be achieved by executing a set of scripts provided by the authors that will download needed components (systems, libraries), initialize the environment, check that software and hardware is compatible, and deploy the system.

Tools

The committee suggests that authors use ReproZip in order to streamline this process. ReproZip can be used to capture the environment, the input files, the expected output files, and the required libraries. A detailed how-to guide (installing, packing experiments, unpacking experiments) can be found there. ReproZip will help both the authors and the evaluators to seamlessly rerun experiments.

If using ReproZip to capture the experiments proves to be difficult for a particular paper, the committee will work with the authors to find the proper solution based on the specifics of the paper and the environment needed. More tools are available here: https://reproduciblescience.org/reproducibility-directory/