Effectiveness of Safety Investigations and Recommendations Generation

Taking into consideration that the quality of accident investigations affect determinatively the effectiveness of safety recommendations derived from the corresponding reports, the project emphasised on the research of both accident investigations’ and recommendations’ quality. In addition, the case that safety recommendations might have been effective, but not seen as such was considered. Hence, the main research question was:

“Why are NPP’s safety recommendations not effective, or are not seen as such?”

Taking into account the typical sequence from accident investigation to the design, formulation and implementation of safety recommendations, the hypotheses addressing the main research question were:

Hypothesis 1:

The safety recommendations stated in the accident investigation reports are not effective because:

• The official accident investigation procedure has not been developed effectively.
• The accident investigation procedure is not fully followed.
• The accident investigation procedure is followed but the recommendations development is not effective.

Hypothesis 2:

The safety recommendations are effective, but not seen as such.

Hypothesis 3:

Although the safety recommendations are effective in theory, they are not appropriately implemented to be effective in practice.

Based on the hypotheses, the following questions were identified as relevant to be incorporated in the analysis framework:

Question 1:

To what extent are NPP’s accident investigation procedures well-structured and complete?

Question 2:

To what extent are NPP’s accident investigation procedures comprehended and implemented? NPP accident investigations must comply with the relevant company internal procedures.

Question 3:

To what extent does NPP encompass the state-of-the-art views on safety? The NPP specifically requested that the state-of-the-art views on safety, as disseminated by safety theorists and pioneers, should be researched.

Question 4:

To what extent do NPP's accident investigation methods embed state-of-the-art accident models?

Question 5:

How effective are NPP's safety recommendations?

In order to address the questions above, contemporary literature was researched for the topics shown below

An analysis framework was developed based on the theoretical background, specifically for the accident investigation reports/safety documentation analysis. In order to validate the results of the accident investigation reports/safety documentation analysis, and to obtain insights on the NPP local conditions, interviews were held with NPP’s staff. The table below presents the correspondence among research questions (Qx) and theory topics (Tx) along with the criteria used to answer the questions.

The research framework was assessed for reliability through the analysis of NPP’s three (3) accident reports by two independent researchers. Initial inter-reliability results revealed few differences in the comprehension of new safety views and recommendation – barrier types. After proper rewording, the inter-reliability score was significantly increased and was calculated between 81% and 100% for various research framework elements.

The results derived from the application of the research framework were validated during 13 interviews, which were held with senior staff at NPP’s premises. The interviews also collected information about NPP’s local conditions, which were taken into account by the Aviation Academy in order to recommend to the NPP realistic and tailored measures.

• The NPP’s accident investigation procedure needs improvement in its design and it is not accompanied with an investigation handbook;
• NPP lacks specific policy and procedures for recommendations’ design, prioritisation, follow-up, and effectiveness assessment;
• The NPP’s accident investigation reports include a blend of traditional and state-of-the-art views on safety. More specifically, although human error is rightly perceived as a symptom (not a cause), management level problems are not adequately addressed, and there is a tendency towards judgmental and counterfactual approaches;
• The quality of safety recommendations is poor. Concisely, they are too abstract (not SMART enough) and numerous, recommendations which are filtered by department heads with no provision of structural feedback; and
• There is an unclear definition of accountabilities among the Human Factors department and the managers (operational, maintenance, support etc.) regarding accident investigation review meetings and recommendations’ generation, implementation, and monitoring.

• Enhance NPP leaders’ accountability for safety, and specifically the monitoring of the implementation of recommendations and timely feedback in cases of conflicts or difficulties encountered during preparation of the recommendations ;
• Clarify roles and accountabilities of team members, particularly during accident investigation review meetings, recommendations’ generation and preventive measures’ implementation;
• Commit sufficient resources during accident investigations;
• Develop a common framework addressing design, types, prioritization and assessment of recommendations and measures published by NPP departments;
• Develop a confidential reporting system, a safety rewarding system, just culture policy and proactive safety management and performance indicators;
• Amend the existing accident investigation procedure in order to comply with all design requirements of this report and complement this with a descriptive investigation manual;
• Promote the MTO and FRAM model advantages and focus more on successful operations; and
• Adequately train investigators in all accident levels.

The Aviation Academy has already launched two separate research projects conducted by four graduate students, who have been assigned with the task to review and apply the tool developed in this project for evaluating contemporary safety thinking through accident report analysis.

In 2015, 52 aviation accident reports published by the Dutch Safety Board and 40 aviation accident reports published by the Australian Safety Board will be analysed. Interviews with the aforementioned authorities will be scheduled in order to validate the results and discuss underlying factors, which may have affected the extent of new safety approaches presence in accident reports.

The plan for next years it to retrofit the framework with potential new theories and apply it to accident reports published in a large time span by authorities of various industry sectors. It is expected that this combination of longitudinal and ethnography research will depict the extent and the reasons of new safety thinking adoption by the industry.

Bishop, J., & LaRhette, R. (1988, June). Managing human performance-INPO's human performance evaluation system. In Human Factors and Power Plants, 1988., Conference Record for 1988 IEEE Fourth Conference on (pp. 471-474). IEEE.

Catino,M. (2008). A review of literature: individual blame vs. organizational function logics in accident analysis. Journal of Contingencies and Crisis Management, 16(1), 53-62.

Dekker, S. (2006). The field guide to understanding human error. Bedford, UK.

Dekker, S. (2012). Just culture: Balancing safety and accountability. Ashgate Publishing, Ltd..

Dekker, S., & Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology & Work, 6(2), 79-86.

Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives. Management Review, 70(11), 35-36.

Gilbert, R. (2008). A Quick Guide to Health & Safety. UK: Woodhead Publishing Ltd.

Heinrich, W.H. (Ed.), 1941. Industrial Accident Prevention. McGraw-Hill, New York

Hollnagel, E. (2004). Barriers and accident prevention. Aldershot: Ashgate.

Hollnagel, E. (2008). Risk+ barriers= safety?. Safety Science, 46(2), 221-229.

Hollnagel, E. (2014) Safety-I and Safety-II: The Past and Future of Safety Management. Ashgate Publishing. Ltd.

Hollnagel, E., & Goteman, O. (2004). The functional resonance accident model.Cognitive System Engineering in Process Plant 2004.

Hollnagel, E., & Speziali, J. (2008). Study on Developments in Accident Investigation Methods: A Survey of the" State-of-the-Art.

International Civil Aviation Organization (ICAO). (2013). Safety Management Manual Doc. 9859. CA.

Jeynes, J. (2007). Managing Health & Safety – Learning Made Simple. UK: Butterworth – Heinemann.

Karanikas, N. (2014). Developing Safety Performance Indicators and Employing Quality Management Principles for Transforming a Military Aviation Organization’s Safety Program to an Operational Safety Management System. Doctoral dissertation, Middlesex University.

Katsakiori, P., Sakellaropoulos, G., & Manatakis, E. (2009). Towards an evaluation of accident investigation methods in terms of their alignment with accident causation models. Safety Science, 47(7), 1007-1015.

Leveson, N. (2004). A new accident model for engineering safer systems.Safety science, 42(4), 237-270.

Leveson, N. (2011). Engineering a safer world: Systems thinking applied to safety. Mit Press.

Manuele, F.A. (2003). On the practice of Safety. NJ: John Wiley & Sons.

Reason, J., Hollnagel, E., & Paries, J. (2006). Revisiting the «Swiss cheese» model of accidents. Journal of Clinical Engineering, 27, 110-115.

Rollenhagen, C., Westerlund, J., Lundberg, J., & Hollnagel, E. (2010). The context and habits of accident investigation practices: A study of 108 Swedish investigators. Safety science, 48(7), 859-867.

Wong, W. (2010). The Risk Management of Safety & Dependability. UK: Woodhead Publishing Ltd.

Woods, DD., Dekker, S., Cook, R., & Sarter, N. (2010). Behind human error. Farnham: Ashgate.

Underwood, P., & Waterson, P. (2013). Accident analysis models and methods: guidance for safety professionals.

• Dr. Nektarios Karanikas, Aviation Academy
• Pedram Soltani, Aviation Academy
• Prof. Robert Jan de Boer, Aviation Academy
• Dr. Alfred Roelen, Aviation Academy
• Prof. Sidney Dekker, Griffith University
• Prof. John Stoop, Aviation Academy

Dr. Nektarios Karanikas, n.karanikas@hva.nl.