Artifact Evaluation
Artifact Evaluation (or AE for short) is a process adopted by a number of academic conferences to reproduce experimental results from published articles.[1][2][3][4] Authors are usually invited to submit the supporting materials (code, data, models, experimental workflows, results) for their papers to the AE process. A special committee then assess how these artifacts support the work described in the papers. Camera-ready papers that successfully go through the AE process receive a set of approval badges[2] and include an Artifact Appendix.[5][6]
Conferences and journals[edit]
Many leading computer systems and machine learning conferences and journals including PLDI, OOPSLA, PPoPP, SysML, TOMS and Supercomputing use Artifact Evaluation to promote reproducibility of experimental results and encourage code and data sharing.[3][4]
Formalization[edit]
The ACM Task Force on Data, Software, and Reproducibility in Publication was set up in 2015 to formalize Artifact Evaluation[7]. It produced the ACM Artifact Review and Badging Policy now adopted by major computer science conferences and journals sponsored by ACM.[2] [4]
References[edit]
- ↑ Childers, Bruce R.; Fursin, Grigori; Krishnamurthi, Shriram; Zeller, Andreas (2015). "Artifact Evaluation for Publications (Dagstuhl Perspectives Workshop 15452)". Dagstuhl Perspectives Workshop 15452. 5 (11): 29–35. doi:10.4230/DagRep.5.11.29.
- ↑ 2.0 2.1 2.2 ACM Artifact Review and Badging Policy
- ↑ 3.0 3.1 Artifact evaluation for software conferences
- ↑ 4.0 4.1 4.2 Artifact Evaluation for computer systems and machine learning conferences
- ↑ Artifact Appendix Template
- ↑ Example of a paper PDF with a set of ACM artifact evaluation badges and an Artifact Appendix
- ↑ The ACM Task Force on Data, Software, and Reproducibility in Publication
This article "Artifact Evaluation" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Artifact Evaluation. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
