Measuring Safety In Aviation – Developing Metrics For Safety Management Systems

Project

The project “Measuring safety in aviation – developing metrics for Safety Management Systems” will be executed by the Aviation Academy of the Amsterdam University of Applied Sciences with a wide range of partners, including authorities, universities and companies. The project will last from September 2015 to August 2019 and be financed through a RAAK Pro grant of the Dutch Ministry of Education.

The scope of the project is to:

Assess the suitability of existing safety performance indicators;
Develop alternative process and safety outcome metrics
Validate the alternative indicators in practice with help of the industry
Translate this knowledge into a tool for the industry to evaluate safety in a more objective way than is currently possible.

About the project

Currently, the relation between safety management activities carried out by aviation organizations and their achievements in safety performance is anecdotal; companies do not actually know what effect their safety management has on their safety outcomes. This makes it impossible for them to define and assess safety risks that exist under their current safety management approach. They assume certain measures warrant safety, and they adjust their assumptions when these proven wrong. In order to improve safety in aviation new international regulations and guidelines for the management of safety have been introduced to the industry: the so-called Safety Management Systems (SMS). The aim of SMS is to shift safety management to a concept that, next to compliance to safety regulations, also provides evidence of a link between their safety management activities and performance. If companies know the effect their safety management has on safety outcomes, they have insight in what the risks are and to what extent these exist. By being able to “predict” their safety performance, organizations can adjust their safety management to these predictions before incidents occur, resulting thus, in improved safety performance. Moreover, companies will be able to claim to the authorities that their SMS processes are effective.

Research framework

It is commonly assumed that the “effective” management of safety is correlated with safety performance; however this has not yet been scientifically validated. This study aims to validate the relation between the effective management of safety according to the systemic models, and safety performance defined as “the ability to succeed under varying conditions”. Such validation will allow companies that do not have the benefit of enough direct safety data, representing their safety performance through their safety management performance.

In the research framework, five alternative projects have been identified. All of these projects will be tested separately for their suitability as safety performance indicators. These projects are:

The gap between Work-as-Done and Work-as-Imagined as an indicator for safety

In practice rules are not always followed to the letter, and even sometimes various rules contradict each other. Therefore, the manner work is done in reality differs from how it was originally designed, prescribed or imagined. Under this context, Work-as-Done (WaD) and Work-as-Imagined (WaI)will be represented with the help of systemic modelling. After a paper case has been build, the project will be extended to agent-based modelling in order to simulate the processes and explore the cases in which each model is safer. The hypothesis is that if the gap between WaD and WaI is large, more negative events will happen.

Safety culture prerequisites

Safety culture is widely discussed topic in aviation industry, but due to the diverse corresponding approaches and definition its measurement remains debatable. Instead of measuring safety culture, the research team has been developing a tool which will be used to evaluate whether companies plan for the prerequisites to build a positive safety culture and the extent to which such prerequisites are visible to the workforce. The hypothesis is that the fewer the prerequisites organizations plan and realise the lower their safety performance.

Effectiveness of risk control measures

Barriers are products of risk assessment and are introduced to prevent incidents and accidents. Although in the risk management cycle the effectiveness of barriers is mentioned, relevant metrics have not been introduced and the effectiveness is retrospectively evaluated. The team will research how effectiveness of barriers can be measured before the occurrence of unwanted safety events. The hypothesis is that low effectiveness of risk controls will be associated with high number of safety events.

Planned resources versus Available resources

The objective of this research is to create metrics that can depict the difference between documented, necessary and available resources (people, time, equipment and budgets) for the performance of daily activities. This is another translation of the distance between WaI and WaD and is connected to the literature suggesting that scarcity of resources contribute into degradation of organizational performance. The hypothesis is that the higher the gaps between planned, necessary and available resources the more the safety events.

The measurement of complexity in a system

Increased complexity of modern socio-technical systems and tight coupling of system components and processes has been linked to lower safety performance due to the limited ability to control such systems and react to unforeseeable situations. The research team based on networks theory will derive metrics that quantify the complexity and/or coupling of socio-technical networks. The hypothesis will be that the higher the complexity and/or the tighter the couplings, the lower the safety performance.

Evaluation of Safety Management System (SMS) with the System Theoretic Process Analysis (STPA) technique

Aviation SMS is currently assessed for compliance with standards, but initiatives have been launched toward a performance-based evaluation. Under this reality, we will analyse SMS with the STPA in order to derive hazardous system states and causal factors that can render SMS ineffective and we will introduce a respective scoring method based on both compliance (i.e. existence and implementation of SMS processes) and system performance (i.e. interdependencies of SMS elements), in addition to the outcomes of individual SMS processes. The hypothesis is that the lower the SMS scores, the lower the safety performance.

Partners

The partners who will contribute to the project are: Netherlands Aerospace Group (NAG), JetSupport BV, Nationaal Lucht- en Ruimtevaartlaboratorium (NLR), Koninklijke Luchtvaart Maatschappij NV (KLM), Ministerie voor Infrastructuur en Milieu, Sky Service Netherlands BV, SAMCO Aircraft Maintenance BV, Koninklijke Nederlandse Vereniging voor Luchtvaart (KNVvL), HeliCentre BV, Transavia, Luchtverkeersleiding Nederland (LVNL), KLM Cityhopper, Team HF PartG, Griffith University – Safety Science Innovation Lab, Delft University of Technology – Safety & Security Science Group, Militaire Luchtvaart Autoriteit (MLA), STg. Human Factors in Aviation Group (HUFAG).