Welcome to the Strengthening the Reporting of Empirical Simulation Studies (STRESS) Guidelines pages
Why STRESS?
We need to improve how we report and publish simulation studies
Incomplete or ambiguous reporting means that many simulation studies are not reproducible, leaving other modellers with an incomplete picture of what has been done and unable to judge the reliability of the results. Crucially, unclear reporting makes it difficult to reproduce or reuse findings.
In response to this we developed STRESS to help to improve the reporting of discrete-event simulation, system dynamics and agent-based simulation models.
We hope that STRESS can help to improve reproducibility of simulation results and to help modellers compose and structure Data Management Plans reflecting the broader aspects of simulation.
Please see the STRESS wiki for advice on how to use the Guidelines and a growing set of examples.
Please feel free to suggest modifications or even new STRESS Guidelines (e.g. for hybrid simulation). We also want to develop a listing of published STRESS Guidelines. Please contact us with suggestions and Guideline examples.
Latest versions of the STRESS Guidelines (v1_1)
The latest versions of the STRESS Guidelines are as follows:
-
STRESS for Discrete Event Simulation (STRESS_DES_V1_1.docx)
-
STRESS for Agent Based Simulation (STRESS_ABS_V1_1.docx)
-
STRESS for Systems Dynamics (STRESS_SD_V1_1.docx)
You can download all of the above from our Github Repository (Click on the filename in the repository and then click on the download on the right)
Citation
If you are using the STRESS Guidelines, we would be grateful if you could cite the following paper:
Thomas Monks, Christine S. M. Currie, Bhakti Stephan Onggo, Stewart Robinson, Martin Kunc & Simon J. E. Taylor (2018) Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines, Journal of Simulation, DOI: 10.1080/17477778.2018.1442155.
Revision history
Guidelines v1_0
First version of the Guidelines.
Guidelines v1_1
Added section 6 to enable specification of how to access code, data, etc.
Interesting papers
- Taylor, S.J.E., Anagnostou, A., Monks, T., Currie, C., Onggo, B.S., Kunc, M., Robinson, S. (2018). Applying the STRESS Guidelines for Reproducibility in Modeling & Simulation: Application to a Disease Modeling Case Study. In Proceedings of the 2018 Winter Simulation Conference, to appear.
An example of how STRESS-ABS can be used to document and agent based simulation. The STRESS record for the Disease Modelling Case Study is at http://dx.doi.org/10.15169/sci-gaia:1494421530.94.
- Taylor, S.J.E., Eldabi, T., Monks, T., Rabe, M., Uhrmacher, A.M. (2018). Crisis, What Crisis: Does Reproducibility in Modeling & Simulation Really Matter? In Proceedings of the 2018 Winter Simulation Conference, to appear.
An interesting panel at WSC2018 that presents different views on the reproducibility crisis.
- Taylor, S.J.E., Anagnostou, A., Fabiyi, A., Currie, C. and Monks, T. (2017). Open Science: Approaches and Benefits for Modeling & Simulation. In Proceedings of the 2017 Winter Simulation Conference, 535-549. doi: 10.1109/WSC.2017.8247813.
Reproducibility is one aspect of Open Science. This introductory tutorial paper shows how one could take an Open Science approach to documenting data, models, results, people and even Science Gatewats in an acadmic paper. It’s a good template to use if you are interested.
- Taylor, S. J. E., Fabiyi, A., Anagnostou, A., Barbera, R., Torrisi, M., Ricceri, R., & Becker, B. (2016). Demonstrating Open Science for Modeling & Simulation Research. In Proceedings of the 2016 ACM/IEEE International Symposium on Distributed Simulation and Real Time Applications (DS-RT 2016), 191-192. doi:10.1109/DS-RT.2016.35.
The original paper just focusing on a demonstration of the above.
- Uhrmacher, A. M., Brailsford, S. C., Liu, J., Rabe, M. and Tolk, A. (2016). Panel – Reproducible Research in Discrete Event Simulation - A Must or Rather a Maybe? In Proceedings of the 2016 Winter Simulation Conference, 1301–131. doi: 10.1109/WSC.2016.7822185.
An earlier panel on reproducibility with interesting views!