FM 2024 will include artifact evaluation (AE) for accepted papers.
An artifact is any additional material, such as software, data sets, log files, machine-checkable proofs, etc., that substantiates the claims made in the paper. Ideally, the artifact allows for the full reproduction of all results in the corresponding paper by providing details on all relevant steps, inputs, configurations and parameters used. For tools, an artifact typically includes the source code or a binary of the tool and corresponding documentation on how to use, reuse, and possibly extend it.
For all accepted FM papers and preliminarily accepted tutorials, authors can submit an artifact substantiating the paper’s or tutorial’s claims. Participation in the AE is optional, but we strongly encourage participation, particularly for tool papers and tutorials.
Important Dates (Artifacts Only)
| Artifact Abstract Submission | June 17th, 2024 (Mon) | 23:59 AoE |
| Artifact Submission | June 24th, 2024 (Mon) | 23:59 AoE |
| Artifact Notification | July 15th, 2024 (Mon) |
Evaluation Criteria
The FM 2024 Artifact Evaluation Committee (AEC) will read the corresponding paper and evaluate the artifact according to the following criteria:
- consistency with and reproducibility of results presented in the paper;
- completeness;
- documentation and ease of (re-)use;
- availability in an online repository with a DOI.
The evaluation will be based on the EAPLS guidelines, and the AEC will decide which badge types — among “functional”, “reusable”, and “available” — will be assigned to a given artifact and added to the title page of the paper in the proceedings. Availability in an online repository with a DOI is a requirement for the “reusable” badge.
Submission Guidelines
The artifact submission is handled via EasyChair. Select the FM 2024 Artifact Evaluation track and provide the following information:
- Artifacts should have the same title and authors as the accepted paper.
- Upload a PDF of the accepted paper.
- The (short) abstract should summarize the content of the artifact and explain its relation to the paper. If there are any special requirements for running the artifact (e.g., specific hardware or software, number of cores, etc.), state them clearly in the abstract.
- A URL (preferably a DOI) to a publicly available zip file containing the artifact and all relevant files. We recommend Zenodo for hosting the artifact.
- The SHA256 checksum of the zip file (to ensure consistency), which can be generated with:
- Linux: sha256sum <file>
- Windows: CertUtil -hashfile <file> SHA256
- MacOS: shasum -a 256 <file>
Artifact Guidelines
The artifact on the permanent storage (e.g. Zenodo) should be based on a virtual machine (VM) image or a docker image.
The artifact should contain:
- A file License.txt containing the license for the artifact. The license must at least allow the AEC to evaluate the artifact w.r.t. the criteria mentioned above.
- A README file containing step-by-step instructions on how to use the artifact. In particular, please document, in step-by-step instructions, how to reproduce the results of the paper using the artifact. If part of the results cannot be reproduced, shortly explain why this is the case.
- All code, binaries, example files, documentation, scripts, etc. required to reproduce the results in the paper.
To obtain the “available” badge, make the artifact publicly and permanently available with a DOI, e.g. on Zenodo.
To obtain the “functional” badge, make sure the artifact is documented, consistent, complete, and exercisable as per the EAPLS guidelines.
The “reusable” badge is awarded instead of the “functional” badge for functional and available artifacts of particularly high quality, that are suitable for reuse and repurposing beyond the associated paper as per the EAPLS guidelines.
Suggestions for preparing the artifact
- In case of a VM image, please use VirtualBox and save the VM image as an Open Virtual Appliance (OVA) file.
- Make it simple for AEC members to exercise the artifact and reproduce the results of the paper via easy-to-use scripts and detailed instructions.
- When writing step-by-step instructions, assume minimum expertise of users.
- The artifact should run out-of-the-box and not require the user to install any additional software. All required packages should already be provided in the VM or Docker image.
- For experiments that require a large amount of resources (time, memory, number of cores, etc.), we recommend to indicate a subset of the results of the paper which can be reproduced with reasonably modest resources (w.r.t. RAM, number of cores, etc.) and in a reasonable amount of time. Please also include the full set of experiments (for reviewers with sufficient hardware or time), just make it optional.
- In case the artifact cannot comply with some of the guidelines, please contact the AE chairs before the AE submission deadline. A common example are artifacts requiring restrictively-licensed software such as Matlab.
Artifact Evaluation Committee (to be announced)
| Member(s) | Affiliation | Role |
| Carlos E. Budde | University of Trento, IT | AE Committee Chair |
| Arnd Hartmanns | University of Twente, NL | AE Committee Chair |