Abstract
Context
User activity logs should capture evidence to help answer who, what, when, where, why, and how a security or privacy breach occurred. However, software engineers often implement logging mechanisms that inadequately record mandatory log events (MLEs), user activities that must be logged to enable forensics.
Goal
The objective of this study is to support security analysts in performing forensic analysis by evaluating the use of a heuristics-driven method for identifying mandatory log events.
Method
We conducted a controlled experiment with 103 computer science students enrolled in a graduate-level software security course. All subjects were first asked to identify MLEs described in a set of requirements statements during the pre-period task. In the post-period task, subjects were randomly assigned statements from one type of software artifact (traditional requirements, use-case-based requirements, or user manual), one readability score (simple or complex), and one method (standards-, resource-, or heuristics-driven). We evaluated subject performance using three metrics: statement classification correctness (values from 0 to 1), MLE identification correctness (values from 0 to 1), and response time (seconds). We test the effect of the three factors on the three metrics using generalized linear models.
Results
Classification correctness for statements that did not contain MLEs increased 0.31 from pre- to post-period task. MLE identification correctness was inconsistent across treatment groups. For simple user manual statements, MLE identification correctness decreased 0.17 and 0.12 for the standards- and heuristics-driven methods, respectively. For simple traditional requirements statements, MLE identification correctness increased 0.16 and 0.17 for the standards- and heuristics-driven methods, respectively. Average response time decreased 41.7 s from the pre- to post-period task.
Conclusion
We expected the performance of subjects using the heuristics-driven method to improve from pre- to post-task and to consistently demonstrate higher MLE identification correctness than the standards-driven and resource-driven methods across domains and readability levels. However, neither method consistently helped subjects more correctly identify MLEs at a statistically significant level. Our results indicate additional training and enforcement may be necessary to ensure subjects understand and consistently apply the assigned methods for identifying MLEs.
Similar content being viewed by others
Notes
This study was approved by the North Carolina State University Institutional Review Board (#5354)
To calculate readability metric values, we use the calculators provided by https://readability-score.com/
References
Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Computat Linguist 22(2):249–254, http://dl.acm.org/citation.cfm?id=230386.230390
Chuvakin A, Peterson G (2010) How to do application logging right. IEEE Secur Priv 8(4):82–85. doi:10.1109/MSP.2010.127
Falcao L, Ferreira W, Borges A, Nepomuceno V, Soares S, Baldassare M (2015) An analysis of software engineering experiments using human subjects. In ACM/IEEE International Symposium on Empiriral Software Engineering and Measurement. Beijing, China
Fu Q, Zhu J, Hu W, Lou J-G, Ding R, Lin Q, Zhang D, Xie T (2014) Where do developers log? An empirical study on logging practices in industry. In Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, 24–33. New York, New York, USA: ACM Press. doi:10.1145/2591062.2591175. http://dl.acm.org/citation.cfm?id=2591062.2591175
Health Insurance Reform: Security Standards (2013) United States Department of Health & Human Services. http://www.hhs.gov/ocr/privacy/hipaa/administrative/securityrule/securityrulepdf.pdf
Host M, Regnell B, Wohlin C (2000) Using students as subjects—a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214
Kincaid JP et al (1975) Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel. (January 31). http://eric.ed.gov/?id=ED108134
King J, Williams L (2013) Cataloging and comparing logging mechanism specifications for electronic health record systems. (August 12): 4. http://dl.acm.org/citation.cfm?id=2696523.2696527
King J, Smith B, Williams L (2012) Modifying without a trace: general audit guidelines are inadequate for open-source electronic health record audit mechanisms. In 305–314. IHI ’12. ACM. doi:10.1145/2110363.2110399. http://doi.acm.org.prox.lib.ncsu.edu/10.1145/2110363.2110399
King J, Smith B, Williams L (2012b) Audit mechanisms in electronic health record systems: protected health information may remain vulnerable to undetected misuse. Int J Comput Models Algorithm Med 3(2):23–42. doi:10.4018/jcmam.2012040102
King J, Pandita R, Williams L (2015) Enabling forensics by proposing heuristics to identify mandatory log events. In Proceedings of the 2015 Symposium and Bootcamp on the Science of Security - HotSoS ’15, 1–11. New York, New York, USA: ACM Press. doi:10.1145/2746194.2746200. http://dl.acm.org/citation.cfm?id=2746194.2746200
McKenney M (2014) Additional consideration of prior conduct and performance issues is needed when hiring former employees. United States Treasury Inspector General for Tax Administration. http://www.treasury.gov/tigta/auditreports/2015reports/201510006fr.pdf
McLaughlin GH (1969) SMOG grading: a new readability formula. J Read 12(8):639–646
Meneely A, Smith B, Williams L (2012) iTrust electronic health care system: a case study. In: Software and systems traceability. Cleland-Huang J, Gotel O, Zisman A (eds.) Springer
Payment Card Industry Data Security Standards (2010) Paymenet card industry security standards council. https://www.pcisecuritystandards.org/documents/pci_dss_v2.pdf
Family Educational Rights and Privacy (1974) United States: United States Government Publishing Office. http://www.ecfr.gov/cgi-bin/text-idx?rgn=div5&node=34:1.1.1.1.33
Read R (2015) University of oregon unlawfully releases 22,000 pages with confidential faculty, staff and student records. The Oregonian/OregonLive. http://www.oregonlive.com/education/index.ssf/2015/01/university_of_oregon_illegally.html
Riaz M, King J, Slankas J, Williams L (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts.In: 2014 I.E. 22nd International Requirements Engineering Conference (RE), 183–192. IEEE. doi:10.1109/RE.2014.6912260. http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6912260
Riaz M, Slankas J, King J, Williams L (2014) Using templates to elicit implied security requirements from functional requirements - a controlled experiment. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement - ESEM ’14, 1–10. New York, New York, USA: ACM Press. doi:10.1145/2652524.2652532. http://dl.acm.org/citation.cfm?id=2652524.2652532
Roper E (2013) Driver’s license snooping gets costly for taxpayers. StarTribune. http://www.startribune.com/local/west/220066801.html
Salman I, Misirli A, Juristo N (2015) Are students representatives of professionals in software engineering experiments?” In Proceedings of the 37th International Conference on Software Engineering 666–676
Shull F, Singer J, Dag IK, Sjøberg (2008) Guide to advanced empirical software engineering. Springer, London. doi:10.1007/978-1-84800-044-5
Smith EA, Senter RJ (1967) Automated Readability Index. AMRL-TR. Aerospace Medical Research Laboratories (6570th) (May): 1–14. http://www.ncbi.nlm.nih.gov/pubmed/5302480
Solingen V, Basili V, Caldiera G, Romback HD (2002) Goal Question Metric (GQM) approach. Encyclopedia of Software Engineering
Standard Specification for Audit and Disclosure Logs for Use in Health Information Systems (2013) ASTM International. http://www.astm.org/Standards/E2147.htm
Survey of Patient Privacy Breaches (2011) Veriphyr Inc. http://www.veriphyr.com/landing/HIPAA_violation_survey/
Vance A, Lowry PB, Eggett D (2013) Using accountability to reduce access policy violations in information systems. J Manag Inf Syst 29(4):263–290. doi:10.2753/MIS0742-1222290410
Yskout K, De Win B, Joosen W (2008) Transforming security audit requirements into a software architecture. In: CEUR Workshop Proceedings. Sun SITE Central Europe CEUR-WS. Vol. 413
Yuan D, Park S, Zhou Y (2012) Characterizing logging practices in open-source software. In 2012 34th International Conference on Software Engineering (ICSE), 102–112. IEEE. doi:10.1109/ICSE.2012.6227202. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6227202
Yuan D, Zheng J, Park S, Zhou Y, Savage S (2012b) Improving software diagnosability via log enhancement. ACM Trans Comput Syst 30(1):1–28. doi:10.1145/2110356.2110360
Acknowledgments
This work is funded by the United States National Security Agency (NSA) Science of Security Lablet. Any opinions expressed in this report are those of the author(s) and do not necessarily reflect the views of the NSA. We thank the Realsearch research group for providing helpful feedback on this work. This study was approved by the North Carolina State University Institutional Review Board (#5354).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Richard Paige, Jordi Cabot and Neil Ernst
Rights and permissions
About this article
Cite this article
King, J., Stallings, J., Riaz, M. et al. To log, or not to log: using heuristics to identify mandatory log events – a controlled experiment. Empir Software Eng 22, 2684–2717 (2017). https://doi.org/10.1007/s10664-016-9449-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-016-9449-1