Effectiveness of Safety Investigations and Recommendations Generation
Project“Effectiveness of Safety Investigations and Recommendations Generation” was a project executed by the Aviation Academy of the Amsterdam University of Applied Sciences following a request from a European Nuclear Power Plant (NPP).

The scope of the project was to evaluate NPP’s safety documentation and accident investigation reports, in terms of strengths and areas for improvement, especially regarding the effectiveness of safety recommendations. The ultimate goal was to propose measures by which the quality of safety investigations can be increased and the effectiveness of recommendations stated in accident reports will be enhanced, taking into account NPP’s local conditions. This was a 5-months project that was launched in October 2014 and accomplished in February 2015.
Taking into consideration that the quality of accident investigations affect determinatively the effectiveness of safety recommendations derived from the corresponding reports, the project emphasised on the research of both accident investigations’ and recommendations’ quality. In addition, the case that safety recommendations might have been effective, but not seen as such was considered. Hence, the main research question was:
“Why are NPP’s safety recommendations not effective, or are not seen as such?”
Taking into account the typical sequence from accident investigation to the design, formulation and implementation of safety recommendations, the hypotheses addressing the main research question were:
Hypothesis 1:
The safety recommendations stated in the accident investigation reports are not effective because:
- The official accident investigation procedure has not been developed effectively.
- The accident investigation procedure is not fully followed.
- The accident investigation procedure is followed but the recommendations development is not effective.
Hypothesis 2:
The safety recommendations are effective, but not seen as such.
Hypothesis 3:
Although the safety recommendations are effective in theory, they are not appropriately implemented to be effective in practice.
Based on the hypotheses, the following questions were identified as relevant to be incorporated in the analysis framework:
Question 1:
To what extent are NPP’s accident investigation procedures well-structured and complete?
Question 2:
To what extent are NPP’s accident investigation procedures comprehended and implemented? NPP accident investigations must comply with the relevant company internal procedures.
Question 3:
To what extent does NPP encompass the state-of-the-art views on safety? The NPP specifically requested that the state-of-the-art views on safety, as disseminated by safety theorists and pioneers, should be researched.
Question 4:
To what extent do NPP's accident investigation methods embed state-of-the-art accident models?
Question 5:
How effective are NPP's safety recommendations?
In order to address the questions above, contemporary literature was researched for the topics shown below
No |
Theory topic |
References (see relevant section below) |
Remarks |
T1 |
Accident investigation procedure’s design evaluation |
Katsakiori, Sakellaropoulos, and Manatakis, 2009 |
|
T2 |
Man – Technology – Organization (MTO) accident model |
Hollnagel and Speziali, 2008 |
Used by the NPP in accident investigation |
T3 |
Functional Resonance Analysis Method (FRAM) |
Hollnagel, 2004 |
Used by the NPP in accident investigation |
T4 |
Safety recommendations’ development |
Doran, 1981 Karanikas, 2014 |
Including SMART and various design criteria |
T5 |
State-of-the-art views on safety |
Catino, 2008 Dekker, 2006 Dekker and Hollnagel, 2004 Hollnagel, 2014 Leveson, 2011 |
Used to develop a framework for evaluating safety thinking and approaches in accident reports |
T6 |
Accident model types |
Heinrich, 1931 Ohno, 1988 Watson, 1961 |
Single (root) cause models (e.g., Domino model, 5 Why’s model, Fault Tree Analysis) |
Qureshi, 2007 Reason, 1990 Reason, 1997 Reason et al., 2006 Shappell and Wiegmann, 2003 Underwood and Waterson, 2013 |
Epidemiological models (e.g., Swiss Cheese model, HFACS, Tripod) |
||
Hollnagel, 2004 Hollnagel, 2012 Leveson, 2004 Leveson, 2011 Rasmussen, 1997 |
Systemic models (STAMP, FRAM, ACCIMAP) |
||
T7 |
Recommendation - barrier quality |
Hollnagel, 2008 |
An analysis framework was developed based on the theoretical background, specifically for the accident investigation reports/safety documentation analysis. In order to validate the results of the accident investigation reports/safety documentation analysis, and to obtain insights on the NPP local conditions, interviews were held with NPP’s staff. The table below presents the correspondence among research questions (Qx) and theory topics (Tx) along with the criteria used to answer the questions.
RQ |
Theory Topics |
Criteria |
Remarks |
Q1 |
T1 |
Requirements the NPP accident investigation procedures fulfil |
|
T2 |
Extent to which NPP’s accident investigation procedures regarding MTO model are in compliance with the stages stated in literature |
||
T3 |
Extent to which NPP’s accident investigation procedures regarding FRAM model are in compliance with the steps stated in literature |
||
T4 |
Requirements that the NPP’s design of recommendations fulfil |
||
Q2 |
(NPP procedures) |
% of accident reports fully complying with the investigation stages Diagnosis of non-compliance (frequency analysis per investigation stage) |
For investigations that employed the MTO model |
T3 |
% of accident reports fully complying with FRAM procedures Diagnosis of non-compliance (frequency analysis per FRAM step) |
For investigations that employed the FRAM model |
|
Q3 |
T5 |
% each aspect of new views on safety was present in the causation approach of the accident report Aspects that were indicated in the safety documents |
New safety views were integrated in a tool, which concluded to 8 aspects |
Q4 |
T6 |
% of accident reports using the single cause model % of accident reports using the MTO (epidemiological) model % of accident reports using the FRAM (systemic) model |
|
T2, T3 |
% of accident reports following and addressing all four levels/steps of each analysis (MTO and FRAM) Diagnosis of which levels/steps are excluded (frequency analysis) |
||
Q5 |
T4 |
% of recommendations that fully comply with the Specific, Measurable, Time-bounded criteria Diagnosis of non-compliance (frequency analysis) |
“Agreed” and “Realistic” criteria were assessed during interviews for results validation |
(Fundamental management principle) |
% of causes linked to at least one recommendation; % of recommendations not linked to a cause. |
||
T7 |
% of physical barriers % of functional barriers % of symbolic barriers % of incorporeal barriers |
Barrier quality in ascending order: incorporeal, symbolic, functional, physical |
The research framework was assessed for reliability through the analysis of NPP’s three (3) accident reports by two independent researchers. Initial inter-reliability results revealed few differences in the comprehension of new safety views and recommendation – barrier types. After proper rewording, the inter-reliability score was significantly increased and was calculated between 81% and 100% for various research framework elements.
The results derived from the application of the research framework were validated during 13 interviews, which were held with senior staff at NPP’s premises. The interviews also collected information about NPP’s local conditions, which were taken into account by the Aviation Academy in order to recommend to the NPP realistic and tailored measures.
- The NPP’s accident investigation procedure needs improvement in its design and it is not accompanied with an investigation handbook;
- NPP lacks specific policy and procedures for recommendations’ design, prioritisation, follow-up, and effectiveness assessment;
- The NPP’s accident investigation reports include a blend of traditional and state-of-the-art views on safety. More specifically, although human error is rightly perceived as a symptom (not a cause), management level problems are not adequately addressed, and there is a tendency towards judgmental and counterfactual approaches;
- The quality of safety recommendations is poor. Concisely, they are too abstract (not SMART enough) and numerous, recommendations which are filtered by department heads with no provision of structural feedback; and
- There is an unclear definition of accountabilities among the Human Factors department and the managers (operational, maintenance, support etc.) regarding accident investigation review meetings and recommendations’ generation, implementation, and monitoring.
- Enhance NPP leaders’ accountability for safety, and specifically the monitoring of the implementation of recommendations and timely feedback in cases of conflicts or difficulties encountered during preparation of the recommendations ;
- Clarify roles and accountabilities of team members, particularly during accident investigation review meetings, recommendations’ generation and preventive measures’ implementation;
- Commit sufficient resources during accident investigations;
- Develop a common framework addressing design, types, prioritization and assessment of recommendations and measures published by NPP departments;
- Develop a confidential reporting system, a safety rewarding system, just culture policy and proactive safety management and performance indicators;
- Amend the existing accident investigation procedure in order to comply with all design requirements of this report and complement this with a descriptive investigation manual;
- Promote the MTO and FRAM model advantages and focus more on successful operations; and
- Adequately train investigators in all accident levels.
The Aviation Academy has already launched two separate research projects conducted by four graduate students, who have been assigned with the task to review and apply the tool developed in this project for evaluating contemporary safety thinking through accident report analysis.
In 2015, 52 aviation accident reports published by the Dutch Safety Board and 40 aviation accident reports published by the Australian Safety Board will be analysed. Interviews with the aforementioned authorities will be scheduled in order to validate the results and discuss underlying factors, which may have affected the extent of new safety approaches presence in accident reports.
The plan for next years it to retrofit the framework with potential new theories and apply it to accident reports published in a large time span by authorities of various industry sectors. It is expected that this combination of longitudinal and ethnography research will depict the extent and the reasons of new safety thinking adoption by the industry.
Bishop, J., & LaRhette, R. (1988, June). Managing human performance-INPO's human performance evaluation system. In Human Factors and Power Plants, 1988., Conference Record for 1988 IEEE Fourth Conference on (pp. 471-474). IEEE.
Catino,M. (2008). A review of literature: individual blame vs. organizational function logics in accident analysis. Journal of Contingencies and Crisis Management, 16(1), 53-62.
Dekker, S. (2006). The field guide to understanding human error. Bedford, UK.
Dekker, S. (2012). Just culture: Balancing safety and accountability. Ashgate Publishing, Ltd..
Dekker, S., & Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology & Work, 6(2), 79-86.
Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives. Management Review, 70(11), 35-36.
Gilbert, R. (2008). A Quick Guide to Health & Safety. UK: Woodhead Publishing Ltd.
Heinrich, W.H. (Ed.), 1941. Industrial Accident Prevention. McGraw-Hill, New York
Hollnagel, E. (2004). Barriers and accident prevention. Aldershot: Ashgate.
Hollnagel, E. (2008). Risk+ barriers= safety?. Safety Science, 46(2), 221-229.
Hollnagel, E. (2014) Safety-I and Safety-II: The Past and Future of Safety Management. Ashgate Publishing. Ltd.
Hollnagel, E., & Goteman, O. (2004). The functional resonance accident model.Cognitive System Engineering in Process Plant 2004.
Hollnagel, E., & Speziali, J. (2008). Study on Developments in Accident Investigation Methods: A Survey of the" State-of-the-Art.
International Civil Aviation Organization (ICAO). (2013). Safety Management Manual Doc. 9859. CA.
Jeynes, J. (2007). Managing Health & Safety – Learning Made Simple. UK: Butterworth – Heinemann.
Karanikas, N. (2014). Developing Safety Performance Indicators and Employing Quality Management Principles for Transforming a Military Aviation Organization’s Safety Program to an Operational Safety Management System. Doctoral dissertation, Middlesex University.
Katsakiori, P., Sakellaropoulos, G., & Manatakis, E. (2009). Towards an evaluation of accident investigation methods in terms of their alignment with accident causation models. Safety Science, 47(7), 1007-1015.
Leveson, N. (2004). A new accident model for engineering safer systems.Safety science, 42(4), 237-270.
Leveson, N. (2011). Engineering a safer world: Systems thinking applied to safety. Mit Press.
Manuele, F.A. (2003). On the practice of Safety. NJ: John Wiley & Sons.
Reason, J., Hollnagel, E., & Paries, J. (2006). Revisiting the «Swiss cheese» model of accidents. Journal of Clinical Engineering, 27, 110-115.
Rollenhagen, C., Westerlund, J., Lundberg, J., & Hollnagel, E. (2010). The context and habits of accident investigation practices: A study of 108 Swedish investigators. Safety science, 48(7), 859-867.
Wong, W. (2010). The Risk Management of Safety & Dependability. UK: Woodhead Publishing Ltd.
Woods, DD., Dekker, S., Cook, R., & Sarter, N. (2010). Behind human error. Farnham: Ashgate.
Underwood, P., & Waterson, P. (2013). Accident analysis models and methods: guidance for safety professionals.
- Dr. Nektarios Karanikas, Aviation Academy
- Pedram Soltani, Aviation Academy
- Prof. Robert Jan de Boer, Aviation Academy
- Dr. Alfred Roelen, Aviation Academy
- Prof. Sidney Dekker, Griffith University
- Prof. John Stoop, Aviation Academy
Dr. Nektarios Karanikas, n.karanikas@hva.nl.