This book is intended for practitioners and students in asset management, primarily those with technical backgrounds. It starts with why Root Cause Analysis (RCA) is essential in overall asset management, followed by a step-by-step guide to conducting an effective RCA that yields productive outcomes. Three consecutive chapters present genuine examples of RCA investigations conducted by the author on prominent incidents.
Gas supply disruption at a receiving facility in Jurong, Singapore: This unexpected shutdown incident received attention from the government of Singapore via its Energy Market Authority (EMA).
Turboexpander failure at a Liquified Natural Gas (LNG) plant offshore Darwin, Australia: The mechanical failure of a bearing during a turnover caused a production loss worth $14 million and an asset replacement cost of $1 million.
Electrical fire at an offshore Floating Production, Storage, Off-take (FPSO), South China Sea, Indonesia: Not only from a safety point of view, the incident had caused a significant gas sales loss worth $25 million as the FPSO acted as an export hub for other gas fields.
The last chapter concludes the book with an insight into improving asset management practices through RCA.
Contents
Chapter 1. Why RCA is Fundamental
1.1 A Simple Failure that Cost $9.8 Million
1.2 What and Why a Root Cause?
1.3 RCA in Reliability and Safety Management
1.4 RCA in the Overall Asset Management Cycle
1.5 Where and When to Do RCA
1.6 A Good RCA is a Wise Investment
1.7 Time Frame and Behavioral Aspects
1.8 How to Use This Book
Chapter 2. Step-by-Step Guide for an Effective RCA
2.1 Calling for an Investigation
2.2 Problem Statement
2.3 RCA Team
2.4 Sequence of Events
2.5 Kick-Off Meeting
2.6 Simple Schematics
2.7 Evidence Gathering
2.8 Cause-Effect or Why-Tree
2.9 Corrective Actions or Solutions
2.10 RCA Stakeholders’ Meeting
2.11 Tracking the RCA Closure
Chapter 3. Actual Case Study 1: Disruption of Gas Supply to Singapore
3.1 Background
3.2 Problem Statement
3.3 Sequence of Events (SoE)
3.4 RCA Team
3.5 Simple Schematics
3.6 Evidence Highlights
3.7 Cause-Effect Diagram
3.8 Key Findings
3.9 Corrective Actions or Solutions
3.10 Tracking RCA Corrective Actions
Chapter 4. Actual Case Study 2: Turboexpander Failure in Australia
4.1 Background
4.2 Problem Statement
4.3 Sequence of Events (SoE)
4.4 RCA Team
4.5 Simple Schematics
4.6 Evidence Highlights
4.7 Why-Tree Diagram
4.8 Key Findings
4.9 Corrective Actions or Solutions
4.10 Tracking of RCA Corrective Actions
Chapter 5. Actual Case Study 3: Offshore Electrical Fire
5.1 Background
5.2 Problem Statement
5.3 Sequence of Events (SoE)
5.4 RCA Team
5.5 Simple Schematics
5.6 Evidence Highlights
5.7 Cause-Effect Diagram
5.8 Key Findings
5.9 Corrective Actions or Solutions
5.10 Tracking of RCA Corrective Actions
Chapter 6. Becoming a Learning Organization
6.1 Monitoring Assets and Asset Management System
6.2 A Means of Learning for the Organization
6.3 A Means of Learning for the Individuals
6.4 RCA Software Tool
Bibliography
List of Figures
List of Tables
Book's Features
This book is intended for us practitioners and students in asset management, primarily those with technical backgrounds. It starts with why RCA is essential in overall asset management, followed by a step-by-step guide to conducting an effective RCA that yields productive outcomes. Three consecutive chapters present genuine examples of RCA investigations conducted by the author on prominent incidents:
- Gas supply disruption at a receiving facility in Jurong, Singapore: This unexpected shutdown incident received attention from the government of Singapore via its EMA (Energy Market Authority).
- Turboexpander failure at an LNG (Liquified Natural Gas) plant offshore Darwin, Australia: The mechanical failure of a bearing during a turnover caused a production loss worth $14 million and an asset replacement cost of $1 million.
- Electrical fire at an offshore FPSO (Floating Production, Storage, Off-take), South China Sea, Indonesia: Not only from a safety point of view, the incident had caused a significant gas sales loss worth $25 million as the FPSO acted as an export hub for other gas fields.
The last chapter concludes the book with an insight into improving asset management practices through RCA.
About the Author
Andy P. Marjoko, M.Sc., M.B.A., Ph.D., is an asset management consultant who had previously spent 20+ years of his career at multinational energy companies, primarily with ConocoPhillips and Chevron. With a first background in mechanical engineering, in 2002 he received the British Chevening Award scholarship to pursue an M.Sc. in Offshore and Ocean Technology (with Risk Management specialization) at Cranfield University, UK. As a Principal Investigator (PI), Andy has led more than 15 RCA investigations on significant reliability and integrity-related incidents. He’s a highly respected PI due to his thorough approach to combining field experience, management judgment, and analytical capability to discover latent causes and recommend actionable corrective actions. Dr. Marjoko is certified in Asset Management from the Institute of Asset Management (IAM), Bristol, UK, and holds a doctorate in Management Science from Université de Haute-Alsace (UHA), France.
List of Abbreviations
A&OI: Asset & Operating Integrity
CAPA: Corrective and Preventive Action
CoF: Consequence of Failure
DP: Differential Pressure
DCS: Distributed Control System
D/S: Downstream
EDG: Emergency Diesel Engine
EMA: Energy Market Authority
EPB: Emergency Push Button
ESD: Emergency Shutdown
ESDV: Emergency Shutdown Valve
EMA: Energy Market Authority
F&G: Fire and Gas
FAT: Factory Acceptance Test
FME(C)A: Failure Mode and Effect (Criticality) Analysis
GTG: Gas Turbine Generator
HMI: Human-Machine Interface
HP: High Pressure
HQ: Headquarters
IAM: Institute of Asset Management
ISO: International Organization for Standardizations
IT: Information Technology
LNG: Liquified Petroleum Gas
LP: Low Pressure
LTS: Low Temperature Separator
JB: Junction Box
JT: Joule-Thompson
KoM: Kick-Off Meeting
LP: Low Pressure
LTS: Low Temperature Separator
MoC: Management of Change
OIM: Offshore Installation Manager
OREDA: Offshore/Onshore Reliability Database
ORF: Onshore Receiving Facility
P&ID: Piping and Instrumentation Diagram
PDCA: Plan-Do-Check-Act
PFD: Process Flow Diagram
PHA: Process Hazard Analysis
PI: Principal Investigator
PIC: Person in Charge
PM: Preventive Maintenance
PoF: Probability of Failure
PSD: Process Shutdown
RA(S)CI: Responsible-Accountable-(Supporting)-Informed
RAM: Reliability, Availability, Maintainability
RBI: Risk-Based Inspection
RCA: Root Cause Analysis
RCFA: Root Cause Failure Analysis, a synonym of RCA
RCM: Reliability-Centered Maintenance
S(E)CE: Safety (& Environmental) Critical Equipment/Element
SIS: Safety Instrumented System
TBD: To be Decided
TBX: Turboexpander
SoE: Sequence of Events
Chapter 1. Why RCA is Fundamental
1.1 A Simple Failure that Cost $9.8 Million
After a regular meeting with my engineers on 24 September, I received an email from Bruce, a Canadian offshore Field Manager. He was rather upset by a three-day gas export shut-off following a failure on an electric motor that drives a gas compressor. The downtime caused a loss of an opportunity to sell natural gas worth $9.8 million. He asked me to lead an RCA (Root Cause Analysis) investigation to reveal what had caused the motor failure and why it took so long to recover the system.
Some engineers and operation specialists assisted me in the RCA team. We found that the motor had not failed, but the insulation inside the junction box had deteriorated and finally broke down. Humidity and moisture in the junction box played an important role in the deterioration [2]. The junction box had been poorly positioned by design, exposing it to water vapors when the weather was bad. It was also challenging to maintain because of its elevated position. To visually check the junction box, technicians must erect scaffolds first.
Among other corrective actions (or “solutions”), the RCA investigation recommended installing an electric heater inside the junction box and other junction boxes exposed to the same environmental issue. It also proposed installing a semi-permanent ladder to easily maintain the junction box and other critical equipment that are difficult to reach. RCA is a systematic method to prevent incidents with similar root causes from happening on other relevant assets [2]. The key is that the root causes need to be revealed first [7].
1.2 What and Why a Root Cause?
The term "root cause" refers to the fundamental reason or underlying factor that directly or indirectly leads to a problem, issue, or undesired outcome. It's the deepest layer of causation that, when addressed, will prevent the recurrence of the problem [8]. Root causes are distinguished from apparent causes or symptoms, which are more superficial aspects of the problem. Let’s look at an example to see the difference:
- Undesired outcome: Motor breakdown;
- Apparent cause: Arching in the junction box;
- Root causes: Moisture, improper design.
That is why, when conducting an RCA investigation, it is essential to delve beyond the obvious or apparent causes and identify the root cause(s) driving the issue [7]. As a result, the investigation recommends a set of corrective actions or solutions (sometimes also referred to as “Corrective and Preventive Action” or CAPA) to address the root causes. By overcoming root causes, organizations can improve processes, enhance safety, and mitigate the risk of future incidents or failures [4][5].
1.3 RCA in Reliability and Safety Management
Today, risk and reliability engineering has become a mature enough discipline to be an integral part of the design process [6]. Years ago in 2002, when I was studying at Cranfield University, England, it wasn’t the case. I was doing a master’s in Offshore and Ocean Technology, focusing on Risk and Reliability. OREDA (Offshore Reliability Database) was at an early age, and very few people received access to the database. It still had a limited amount of failure data, and some equipment even had zero data populated. Now, almost every engineer knows OREDA, whose “O” also stands for “Onshore” in addition to (originally) “Offshore.” PHA (Process Hazard Analysis), RAM (Reliability, Availability, Maintainability), RBI (Risk-Based Inspection), and RCM (Reliability Centered Maintenance) analyses are now commonly done in the project and operation phases [6].
Even so, there have been no impeccably smooth asset operations with absolute zero failure. Once in a while, we encounter a type of failure that is sometimes unanticipated (Singapore’s gas supply disruption in Chapter 3 is an excellent example of it) [7]. Note that “failure” does not always mean “breakdown”. Any undesired outcome may mean a failure. A piece of equipment’s inability to achieve its expected output level is a failure. An energy consumption higher than normal can also be a failure, etc. [5] Here, RCA plays a fundamental role in continuously improving the reliability and safety of the systems we build and operate. Failures need to be seen as a learning opportunity, like the example of the junction box above [4].
It is best to adopt the asset management approach from the ISO-5500 series to see the constellation from a broader context. In a systematic asset management process, we look at the entire life cycle of the asset, i.e., from the acquisition to its disposal [3][8]. A schematic of an asset life cycle is simplified in Figure 1.1, consisting of four phases: Design and Build, Operate, Maintain, and Dispose. Of course, “Operate” and “Maintain” are not running in sequence but in parallel. However, they are drawn sequentially for simplicity, adopting a convention from the Institute of Asset Management (IAM) [8].
Risk and reliability analyses, such as PHA, RBI, RAM, and RCM, are closely engaged in the Design and Build phase, although they also interact with Operate and Maintain phases [8].
Illustrations are not included in the reading sample
Figure 1.1. Reliability and Safety Management in Asset Life Cycle
Picture: Author's elaboration from the IAM [8]
After the asset has been designed and built to be reliable and safe, “Operate” is the phase where these features are to be demonstrated. In case the asset experiences an undesired outcome, such an incident needs to be investigated through an RCA process. The RCA results will then become feedback to improve practices across the life cycle, including the risk and reliability analyses. And so forth.
1.4 RCA in the Overall Asset Management Cycle
Like other management systems, an asset management system is also a PDCA (Plan-Do-Check-Action) cycle (in the ISO-5500 series, the stages are called “Planning, Operation, Review, Improve”) [3]. We usually picture the PDCA cycle as a “wheel,” as illustrated in Figure 1.2. RCA is undoubtedly a fundamental part of the “Check” stage. Its primary duty is to understand the root causes when something goes wrong or deviates from what was planned. The result is a corrective action that is highly effective because it is based on the root cause instead of (for example) an assumption or belief [6].
Illustrations are not included in the reading sample
Figure 1.2. RCA in the PDCA of Asset Management
Picture: Author's elaboration from ISO-55001 [3]
In addition to the PDCA wheel as a core framework, ISO adds some essential management elements. The “Organizational Context” element represents an environment on which the wheel rolls, similar to a road for a vehicle’s wheels. This context depends on the organization's scope, from a work section comprising five personnel to a corporation with thousands of employees, or even an entire country. Upon any context, ISO-55001 requests organizations to map the expectations of their stakeholders. Chapter 3, about the EMA (Energy Market Authority) in Singapore, illustrates how an external stakeholder can have a significant influence on the RCA conducted by the organization. Another element is “Support,” whose metaphor is like lubrication to the wheel, such as competency, communication, and documentation [3]. The ultimate driving force comes from the “Leadership” element. The leadership must keep the wheel rolling by enforcing the RCA process in the asset management cycle and following up on its results [6].
1.5 Where and When to Do RCA
Although RCA was born to investigate incidents of technical failures, it is so powerful and generic that it should also be used in other undesired outcomes. RCA should be conducted whenever there is a need to understand the underlying causes of problems, incidents, or deviations from desired outcomes, namely:
- Post incidents or asset failures;
- Non-compliance of regulations or standards;
- Increasing risks;
- Customer complaints and feedback;
- Quality issues.
RCA should be integrated into an organization's processes for continuous improvement, risk management, and operational excellence [1][6][8].
Eventually, the RCA will form a culture of eliminating defects in the organization. The key is to apply it proactively and systematically to drive positive change and prevent the recurrence of issues [1][6]. An example of this will be given in the last chapter, Chapter 7.
1.6 A Good RCA is a Wise Investment
In the RCA of an electric motor above, installing heaters in junction boxes and semi-permanent ladders to easily reach them costs $19,800. However, such installations protect against another (or some other) $9.8 million in revenue loss in the future. Other root causes discovered by the RCA include improper design. Part of the RCA corrective actions was to meet the project team and Siemens, the electrical vendor, to improve the design for future projects. The RCA was a wise investment [7].
Another example related to design is in Chapter 4, where a trapped gas was released, damaging a piece of equipment and resulting in a $14 million LNG sales loss. This trapped gas in the design had already been questioned two years before the incident. An MoC (Management of Change) had identified this potential risk, but it was put on hold because of a lack of understanding of such risk. The RCA opened the door again for this MoC to be discussed to prevent future incidents. This will save many more millions of dollars in the future.
In brief, an effective RCA is a good investment when done effectively. This book aims to help you perform such an effective RCA that results in a productive outcome.
1.7 Time Frame and Behavioral Aspects
1.7.1 Cultural factor
RCA does pay in the long run, yet most organizations often need to overcome some cultural barriers. For example, establishing RCA needs a long-term orientation, and not all societies have a high score for this cultural dimension [6]. Also, the RCA process requires not only a “Check” but also an “Act,” and thus, the “forgive and forget” attitude can be very expensive in the long run for asset management. But again, that’s where the leadership is expected to act: It must keep the PDCA wheel rolling. Equally crucial to a quality investigation is the follow-up and closure of RCA’s corrective actions (or “solutions,” as some parties call them). Closing out an RCA’s solutions is integral to the RCA itself: Never close an RCA before every corrective action is closed 96].
A manager also needs to be a role model for making optimum decisions because that is the heart of asset management: Across an asset lifetime (acquire, operate, maintain, dispose), consistently make the optimum decisions on cost, risk, and performance. This is, hence, also a key to an effective RCA, i.e., to consider optimum options when identifying a corrective action [8].
1.7.2 On blaming and denial
Another major barrier to an effective RCA culture is the habit of blame. It does not help the learning and improvement process because blaming fixes nothing [7]. Instead, the organization needs to accept failures as an opportunity to learn, and the RCA is a tool for it. This is, again, where the role of leadership takes place. Managers need to consistently emphasize that the goal of an RCA is to improve the system and not to finger-point anybody.
That is why, even when the RCA process is assisted by an advanced IT (Information Technology) tool, this book strongly recommends a Kick-off Meeting (KoM) as an official start of the investigation. The asset owner needs to re-emphasize this “no blaming” attitude in the meeting. The same thing should also be mentioned before an interview when collecting people evidence—more on this in the following chapters.
1.7.3 How long does an investigation take?
“How long is an RCA?” is a tricky question to answer. A few say it is similar to the question, “How long is a piece of string?” The answer is that it depends on many factors. A complex incident with a large scale indeed needs more time than a smaller, simpler one (more on these RCA categories in Chapter 2) [7].
But there is also an aspect of how to do it. An RCA needs a rigorous approach. Doing RCA rashly and jumping to conclusions should undoubtedly be avoided altogether. In contrast, it can also be tempting to be a perfectionist, being obsessed with knowing a mysterious physical cause. My advice is to focus on solutions, i.e., corrective actions that give the maximum value to the stakeholders. If we work for an organization that manufactures a piece of equipment and are investigating a failure of our product, then, yes, we need to understand the physical details perfectly. However, if our organization uses the equipment, we may focus on how we operate and maintain it [7]. This way, we can finish an RCA within a realistic timeframe.
Some organizations even apply a time limit for an RCA investigation. For example, 30 calendar days to produce a set of corrective actions. If a physical cause needs further laboratory tests to confirm, it can be put in the action plan. The branch on the Cause-Effect or Why-Tree diagram can be marked with “OR,” with notes attached to it (more about this in chapters 4 and 5).
1.8 How to Use This Book
This book is intended for us, practitioners and students in asset management, primarily with technical backgrounds. The next chapter, Chapter 2, gives a step-by-step guide to conducting an effective RCA giving productive results. The guide came from years of experience in leading more than a dozen RCA investigations of large-scale incidents.
The following three chapters contain real-world examples from the industry because the steps in Chapter 2 are surely easier to understand through examples [4].
And finally, the last chapter gives an insight into how RCA strengthens the learning capability of an organization on its journey toward excellence [8].
Chapter 2. Step-by-Step Guide for an Effective RCA
Let’s briefly outline what needs to be done step by step to conduct an effective, high-quality RCA investigation.
2.1 Calling for an Investigation
Following an incident of undesired outcome on an asset, the asset owner (a Field Manager, in many cases) immediately calls for an investigation. Depending on the severity of the impact, the Field Manager can assign a person from within their organization (if the incident is “small”) to conduct the investigation or has to invite a specialist from outside the organization (if the incident is “big”). These “small” and “big” scale categories are set by the enterprise (or other type of organization) based on its corporate risk matrix, whose example is given in Figure 2.1. The vertical axis is the Probability of Failure (PoF), and the horizontal axis is the Consequence of Failure (CoF), which can be financial, safety, environmental, etc. [6]
Illustrations are not included in the reading sample
Figure 2.1. Example of a Corporate Risk Matrix
Picture: Marjoko, Royer-Duffait & Saradjian [6]
Below is a popular categorization of incidents as suggested by Nelms [7]:
- Mini or Low: If the impact of the incident lies in the low-risk zone (green) of the risk matrix, personnel within the asset owner’s organization can investigate with a simple approach;
- Midi or Medium: If the impact of the incident lies in the medium-risk zone (yellow) of the risk matrix, personnel within the asset owner’s organization can investigate but with a rigorous approach (like what’s explained in this book);
- Maxi or High: If the impact of the incident lies in the high-risk (orange) zone or very-high (red) of the risk matrix, the asset owner needs to invite a specialist from outside the organization to be the Principal Investigator (PI) of Facilitator to perform the RCA rigorously (like what’s explained in this book). The asset owner is responsible for providing the necessary resources.
In most cases, serious safety and environmental incidents are immediately categorized as “Maxi.” Some organizations prefer to call them “Low,” “Medium,” and “High” impacts. The terms Mini, Midi, and Maxi are used in this book, although each organization may use different terms.
Once an investigation is called for, the RCA investigation is assigned a unique number according to the organization’s document management system.
2.2 Problem Statement
A clear problem statement is mandatory. It needs to focus on what has failed. The clearer the problem to be solved, the more likely it is going to be solved. The asset owner needs to start defining the problem statement, subject to review by the principal investigator. The following items need to be explicitly written in a problem statement:
- RCA ID: A unique number based on the company’s document control system.
- The undesired outcome, i.e., “what failed” and “what happened.”
- Date/time and duration of the failure.
- Location of the incident: This can be a hierarchical location based on the taxonomy of the equipment, as defined by the organization.
- Impact: It can be financial (e.g., revenue loss, cost of repair), safety (e.g., injury, loss of work time, loss of life), environmental (e.g., release of hazardous materials), reputational, etc. It should refer to what the organization has defined on what consequences matter.
Below is an example of a problem statement based on an actual incident in the South China Sea:
- RCA ID: Offshore-FPSO-0006
- Undesired outcome: Compressor LP-A Trip
- Date/time:
- Event: 21 September, at 10:12
- Recovered: 24 September, at 08:54
- Location: Offshore Business Unit, Natuna FPSO
- Impact (and therefore the category of the investigation: mini, midi, or maxi):
- Revenue loss: $9,800,000
- Rectification costs: $11,000
- Total impact: $9,811,000
- RCA Category: “Maxi”.
Following the problem statement, the author recommends adding short descriptions as preliminary information. For example: “ LP-A gas sales compressor is driven by a 3-phase 15 kV 13,900 HP electric motor. Restoration had to wait for the replacement of damaged parts from the vendor and electrical specialists from the office.”
2.3 RCA Team
With a few exceptions in certain organizations, RCA team membership (including the Principal Investigator) is ad-hoc and thus not a permanent position. At a minimum, a PI or Facilitator has to be appointed. Most major investigations involve teamwork, so the PI should propose additional personnel to be on the team [4]. They typically have the expertise pertinent to the incident. For example, in the compressor incident above, a senior electrical engineer can help because the compressor was driven by an electric motor. A senior electrician should also help reveal the maintenance practices.
These resources may come from inside or outside the organization. If a resource comes from within the organization, they should not have been a direct witness to the incident [1]. This is to avoid biases in judgments during the investigation. In addition to the competence of the PI, knowledge and experience of these resources are keys to a high-quality RCA [4].
2.4 Sequence of Events
The PI needs to be crystal clear about what major events have happened, so they need to start figuring them out. A sequence of events (SoE) needs to be composed, an example of which is given in Table 2.1 below. The initial SoE does not have to be detailed and precise. An outline with temporary information is sufficient. It will be discussed, verified, and refined further during the Kick-off Meeting (section 2.5) and evidence gathering (section 2.7).
Table 2.1. Sequence of Events Example
Illustrations are not included in the reading sample
2.5 Kick-Off Meeting
A kick-off meeting (KoM) is highly suggested as an official start of the RCA investigation. It is also an important means of engaging key stakeholders. The asset owner (e.g., Field Manager) should call the meeting, offline or online, and the PI must be present. Topics to be covered and agreed upon during the KoM are namely:
- Problem statement and SoE;
- Composition of the RCA team;
- Resources committed;
- The time frame of the investigation.
It is human nature to be defensive during an investigation, especially when the incident involves a significant loss [7]. Such an attitude potentially hinders the team from finding root causes. It is necessary for every KoM to clearly state that the objective of an RCA is to improve the system, not to finger-point anybody.
2.6 Simple Schematics
The PI has to be able to isolate the system related to the undesired outcome by drawing what elements were likely involved in the failure and excluding what were not [7]. A hand-drawn sketch is highly preferred, although the PI may use a simple drawing software tool to make it tidy. Figure 2.2 contains a simple sketch of the compressor incident.
Illustrations are not included in the reading sample
Figure 2.2. Example of an Incident’s Simple Schematics
Picture: Author’s documentation
Simply copying standard engineering drawings such as PFD (Process Flow Diagram), P&ID (Piping & Instrumentation Diagram), or circuit diagrams is not acceptable.
2.7 Evidence Gathering
The RCA investigation collects evidence from three mandatory sources abbreviated as the "3 P's": People, Physical, and Paper. The RCA team collects and analyzes them to build a Cause-Effect or “Why-Tree” diagram in the following section. In theory, they are done in sequence: Gathering all evidence first, then starting the analysis to build a cause-effect. In practice, the two tasks (gathering and analysis) often run in parallel, forming an interactive process. Instead of waiting for all evidence, the PI can start building the cause-effect diagram with the evidence gathered so far. In many cases, it helps understand potential causes and what further evidence is needed to confirm it [7].
2.7.1 Physical evidence
There are three common forms of physical evidence [7]:
- Photographs and/or videos;
- Sketch and notes on measurements, positions, etc.;
- Damaged parts (if applicable).
It is tempting to take too many photographs; try to limit it by pointing out anomalies. The best time to collect physical evidence is as early as possible to obtain the “fresh” post-incident conditions. In organizations where the RCA process has been established, field personnel are trained to preserve physical evidence and take proper photographs.
2.7.2 Paper evidence
Please note that in RCA, "paper" data also means data that are stored electronically. Depending on the nature of the incidents and potential causes to be confirmed, paper evidence may include [2]:
- Electronic data recorded in DCS or historian database
- Logbook
- Maintenance history
- Relevant procedures
- Personnel competency matrix
- Training records.
2.7.3 People evidence
Priority should be to interview the eyewitness(es) of the incident. The earlier the interview takes place, the better because human memory fades rapidly. When possible, interviews should be one-on-one, although they may have counsel or representation if appropriate. The interviewer should take notes of key items but should refrain from recording the interview.
The initial questions should ask what the witness saw/heard and what they did. Then, follow-up questions are asked to reveal what was happening. It’s human to become defensive following an incident. Therefore, avoid questions that make the witness feel “cornered.” For example, instead of saying “ Why did you …? ”, it’s better to ask “ What made you..? ” because the latter is much less intimidating. These interviews are particularly important if the investigation relies heavily on people evidence. It needs practice because it requires some “art” in addition to techniques. Sometimes, the best interview is mostly to listen and take notes [7].
Again, it's fundamentally crucial that the objective of the RCA is not to finger-point anybody or to find whose fault it was. Far from it. Everybody in the company must be made to understand that the objective is to improve the system. This needs to be re-stated before starting any interview [5].
2.7.4 Evidence leads
In a large-scale RCA, the PI may want to delegate responsibilities of evidence gathering to members of the RCA team [7]. They are called “Evidence leads”. Below is an actual example of such delegation from an incident at an LNG plant in Australia:
- Principal investigator: Andy
- Physical evidence lead: Gwen
- Paper evidence lead: Chantal
- People evidence lead: Les.
The PI holds final accountability for all evidence, and thus, they have to understand each piece of evidence gathered.
2.8 Cause-Effect or Why-Tree
The heart of an RCA meant to find root causes is the building of a Cause-Effect diagram or “Why-Tree” [7][5]. Don’t get confused about the two terms because they are the same thing. Cause-effect diagrams are more commonly drawn horizontally, while why-trees are drawn vertically. Another diagram that is also available for RCA is the fishbone (Ishikawa) diagram [4]. However, the fishbone diagram is not covered because it has a history linked to manufacturing processes, while cause-effect and why-tree are more generic since their very first start [7][6][4]. Therefore, all examples in this book are presented in either horizontal or vertical cause-effect/why-tree.
2.8.1 Horizontal style
Illustrations are not included in the reading sample
Figure 2.3. Cause-Effect, Horizontal Style
Picture: Author’s drawing
Figure 2.3 above is a schematic describing a cause-effect diagram that is drawn horizontally. The left-most node is the final event or “Undesired outcome” in the RCA’s problem statement. The next node on the right is the possible cause(s) of the final event, which by itself is an effect caused by the next node(s), and so forth [4].
2.8.2 Vertical style
Similarly, the cascade can be drawn vertically as the so-called Why-Tree, as outlined in Figure 2.4.
Illustrations are not included in the reading sample
Figure 2.4. Why-Tree, Vertical Style
Picture: Author’s drawing
A vertical Why-Tree is constructed by asking “why” several times to find the next possible cause [7]. Although it seemed like the opposite of the horizontal cause-effect, it is the same diagram because a particular node contains an effect of the cause contained by the node beneath it.
The Principal Investigator is the sole person with the authority to edit the diagram. Nevertheless, it is best to construct the cause-effect/why-tree diagram in collaboration among the RCA team members. In case additional evidence is needed to confirm a cause, the PI will assign it as a task to the evidence lead.
2.8.3 AND above OR
When there are more than one branches, such as Cause 2.a and Cause 2.b that result in Cause 1, a question may arise whether it is “ Cause 2.a AND Cause 2.b result in Cause 1 ” or “ Cause 2.a OR Cause 2.b result in Cause 1 ”? The answer is: As long as it is practicable and supported by the evidence available, strive for an “AND” connection [7]. An “OR” branch can result in a rather less than firm conclusion of what has happened leading to the incident. However, the situation is not always ideal. Sometimes, a task to confirm a cause needs considerable effort, whether it is too expensive, takes too much time, or both. For example, when complex laboratory analyses and simulations are needed. In such cases, remember that the final objective is to find solutions for the organization. It is fine to leave the branches open with an “OR” connection and then find practicable solutions for both causes.
2.8.4 Layers of causations
A proper cause-effect or why-tree analysis will reveal three layers of causations [7]:
1) Physical causes;
2) Human causes;
3) Management/Organization causes (also referred to as “latent” causes).
It is widely agreed that an investigator needs to ask "Why"five times (that’s the origin of the term "5 Why's") before the PI can reveal latent cause(s). These layers are illustrated in Figure 2.5, which shows a partial why-tree of an incident at an Australian LNG plant. For now, simply focus on the three zones: physical, human, and management because the RCA of this incident will be elaborated on in more detail in Chapter 4.
Illustrations are not included in the reading sample
Figure 2.5. Layers of Causations
Picture: Author’s elaboration from Marjoko, Royer-Duffait, & Saradjian [6]
The first layer, physical causes, lies in the zone shaded red. Numbers in the green circles show the level of "why" being asked, e.g., the question “Why did the bearing fail?” is answered by the causes “Rotating too fast” and “Lube oil offline.” These are physicals. Many investigations stop at this first layer. It would be okay if we affiliate with an equipment manufacturer, for instance, and want to know why a certain product has failed. However, if our organization operates assets, going further down in the layers will produce a lot more benefits [7].
While the first two "whys" are still talking about physical causes, we can see the third and fourth start to reveal human causes, such as “Work Instruction (WI) not followed” and “Lack of understanding of WI.” They sit in the zone shaded yellow. Some other investigations stop here, mistakenly fall into the “blame trap” to finger-point whose fault it was. This is absolutely against the spirit of improvement through finding root causes since blaming fixes nothing. Therefore, when revealing layers of causations, never stop at human error. Seek to know what made the personnel do what they did because most people want to do their jobs correctly.
In the example above, the fifth “Why” begins to discover the latent organizational or management causes in the blue-shaded zone. The importance of these latent causes was first introduced by Bob Nelms [7]. “Competence,” for instance, could be caused by issues in employee development and training, while “Multiple versions” of a single Work Instruction (WI) is a sign of inadequate document management. These two have the potential to be a latent cause of future incidents. Addressing these two issues will be a good investment for the enterprise. Another latent management cause that the author frequently encounters is that people are often not told that they are expected to perform something.
2.8.5 Linking a node with evidence
Evidence is vital to a cause. It is proof that becomes a basis for confirming (or rejecting) a hypothesis of a cause. Therefore, it is a good practice to attach evidence(s) to a node of cause associated with it. If the RCA is assisted by software that allows us to do so, do it. Manual notes will also do [4].
2.8.6 Focus on solutions
To some extent, the PI needs to rely on their judgment to decide how deep and how detailed the investigation should go. If we manufacture equipment and investigate frequent failures of our product, then yes, we need to get a bit more thorough on physical causes. But if we are users of failed equipment, our investigation will likely be a bit broad. Remember to focus on solutions. When doing causal analysis, it is tempting to be a perfectionist. In such a case, think of what the organization needs as solutions to the problem. For example, a PI may encounter that some “legs” (or branches) of the cause-effect chart are more “productive” than the others. In this case, again, think about solutions. If a cause node says “equipment design” (of manufactured equipment), we can notify the manufacturer as an action but expect a considerable time before the design is corrected. Rather, we may focus on operating conditions and/or installation design within our reach of control [7].
When a cause-effect chart is completed, highlight important nodes and list them as findings for identifying potential solutions. For example:
- Finding: Intruding animal (rat) à Solution: More stringent pest control.
- Finding: Condensation in junction box (JB) à Solution: Install heater.
More about this from the actual cases given in chapters 3, 4, and 5.
2.9 Corrective Actions or Solutions
The output of an RCA is a set of corrective actions, which are also often called “solutions,” “action plans,” “path forward,” etc. Each corrective action should also be linked to a cause node, either by software or written manually. Table 2.2 below is an example of RCA’s corrective actions.
Table 2.2. Examples of Corrective Actions or Solutions
Illustrations are not included in the reading sample
The PIC (Person in Charge) indicates the ownership of the corrective action. Cost estimates may or may not be present depending on the company’s nature of operation. It can also be left blank and be discussed in the stakeholder meeting. The status of the solution depends on the stage of the process flow. Below is a list of the status of an RCA solution commonly used in the industry:
- Recommended: The solution was recommended by the RCA team.
- Approved (or rejected): During the RCA stakeholder meeting, the asset owner can approve or reject the solution based on risk priority, budget constraints, etc.
- Completed: The solution has been implemented and signed off on by the PIC.
- Verified: Completion of the implementation has been verified by the PI or an assigned member of the RCA team, endorsed by the asset owner. At this stage, the corrective action or solution is marked as finally “closed.”
2.10 RCA Stakeholders’ Meeting
An offline or online meeting between the RCA team and the stakeholders is crucial. In this stakeholders meeting, the principal investigator has to present the investigation results, typically covering the following items:
- Problem statement and background (briefly);
- The Sequence of Events (briefly);
- Key findings (causal nodes that have been marked important);
- Recommended solutions/corrective actions on key findings;
- Other highlights: Frequently, the PI needs to show stakeholders why certain nodes have been more productive
Use objective and empathic language during the presentation; remember that we are showing weaknesses in the management of their assets. Expect a lively discussion after the presentation. The final objective is to agree upon the recommended solutions, including the PICs, resources, and the timeframe for completion.
2.11 Tracking the RCA Closure
An RCA is a closed-loop process, contributing to the “Check” and “Act” components of the PDCA cycle of management. An RCA investigation is useful when its corrective actions are followed up. No matter how thoroughly an investigation has been done, it will be useless if its corrective actions or solutions are not completed [6].
Therefore, corrective actions or solutions from an RCA investigation must be reviewed and tracked, forming a workflow shown in Figure 2.6. The flow can be effectively managed by the company’s asset management system software (such as SAP® or Oracle EAM®) or within an RCA software if it provides solution tracking features. A particular RCA can only be considered closed when each of its corrective actions is completed and verified [8].
Illustrations are not included in the reading sample
Figure 2.6. Workflow of an RCA Closure Tracking
Picture: Author’s drawing
This page is intentionally left blank.
Chapter 3. Actual Case Study 1: Disruption of Gas Supply to Singapore
A few years before this incident, a subsea pipeline in the South China Sea supplying gas for Singapore’s electricity had leaked. I was in the UK at that time, so I heard a rumor that a significant part of the city experienced a power outage. The government of Singapore was indeed upset. As a consequence, they started to sign gas purchase contracts with other parties (I believe it was a corrective action from the government’s RCA). Since then, they have also assigned the Singapore Energy Market Authority (EMA) to monitor the reliability of the gas supplies closely. Any incident has to be investigated, and the EMA needs to be updated regularly until the RCA is closed.
Then this ESD (Emergency Shutdown) incident happened on the morning of 14 September. This time, it was not on the seabed but on Singapore land.
3.1 Background
In April, at the end of the monsoon, there was a lightning strike at a gas ORF (Onshore Receiving Facility) in Jurong, Singapore. The strike was captured and redirected to the ground by the lightning arrest. However, the induced current affected an EPB (Emergency Push Button) no. 004, causing unnecessary plant shutdown. An RCA result recommended moving EPB-004 farther from the lightning arrest. On 13 September, the ORF maintenance team relocated the EPB and tested whether the switch and the circuit were working. On 14 September, the team attempted to remove the bypass to put the system back to normal operation. Before removing the bypass, the HMI (Human-Machine Interface) indicated that the status was green (OK) for removal. However, when the bypass was removed at 10:00 am local time, an Emergency Shutdown (ESD) was triggered, and the ORF experienced an interruption in the gas supply to Singapore. The system was recovered within one and a half hours, and “only” $340,000 in gas sales revenue was lost. However, it raised a concern from the Energy Market Authority, and thus, the reputation impact was significant.
The author was shortly called to Singapore to lead an RCA investigation, trying to answer this question: Why was an ESD activated while the HMI interface was showing green (OK) to remove the bypass?
3.2 Problem Statement
- RCA ID: Singapore-ORF-0004
- Undesired outcome: Emergency Shutdown
- Date/time:
- EPB-004 relocated: 13 September, at 18:00
- ESD occurred: 14 September, at 10:00
- Location: Singapore Business Unit, Jurong, ORF
- Impact:
- Revenue loss: $340,000
- Reputation: Medium-High
- RCA Category: “Midi”.
- Other descriptions: The stakeholders expect the RCA to answer the following questions:
- What caused the ESD while the HMI status was green (OK)?
3.3 Sequence of Events (SoE)
As usual, the PI initiated a temporary SoE as a start, which will be refined later along with the evidence-gathering process. Table 3.1 lists the final Sequence of Events used for the stakeholder’s meeting.
Table 3.1. Gas Supply Disruption – Sequence of Events
Illustrations are not included in the reading sample
3.4 RCA Team
With the author as a Principal Investigator, three resources were assigned to the RCA Team:
- Instrumentation and Control Engineer (as Physical Evidence lead)
- Pipeline systems specialist (as Paper Evidence lead)
- Pipeline operations coordinator (as People Evidence lead).
3.5 Simple Schematics
As shown in Figure 3.1, the ORF receives natural gas from the subsea pipeline and delivers it to buyers in Singapore. The process safety of the plant is supervised by an SIS (Safety Instrumented System) of ABB PlantGuard® type. If the SIS detects an unsafe condition, it will activate certain field devices to protect the plant, including ESDV 1 (upstream) and ESDV 2 (downstream). Operators and technicians interact with the SIS via a computer console as the HMI (Human-Machine Interface).
Illustrations are not included in the reading sample
Figure 3.1. Gas Supply Disruption - Schematic
Picture: Author’s drawing
To apply a more conservative safety approach, plant operators can manually enforce a plant shutdown if they observe an unsafe condition, even if the SIS does not detect it. For this purpose, there are five Emergency Push Buttons (EPB-001 to EPB-005) available at different positions in the plant. Before this incident, there had been a lightning strike affecting EPB-004 because its position was too close to the lightning arrest. It was recommended that EPB-004 be moved farther from the lightning arrest. This relocation was performed on 13 September, one day before this incident.
3.6 Evidence Highlights
The ESD logic latch was still active when the shutdown was initiated. If the EPB-001R reset button had been pushed, the SIS would not have shut down the system. However, the technician did not push it because the HMI showed a normal indication. This led to one intriguing question: why did the indication show normal (“Green”) while the latch was still active? During people evidence gathering, technicians firmly mentioned it was green, as shown in Figure 3.2.
Illustrations are not included in the reading sample
Figure 3.2. “Green” Status of EPB-004 on HMI
Picture: Author’s documentation
Figure 3.2 displays what was seen by the technician; the status of EPB-004 (circled in blue) clearly shows green. The RCA’s Cause-Effect development almost got stuck here because there was no further paper evidence to prove or disprove it. After all, the company didn’t have a device to simulate the logic. The nearest available simulator was located at the ABB’s office in Singapore. With approval from the Asset Owner, a copy of the software logic was downloaded from the HMI and taken to the ABB’s laboratory for a simulation. A series of simulations were run, and at a trial, the engineers found a shocking yet interesting fact. Figure 3.3 shows the order of colors stacked as programmed for the EPB’s status indicators, from top to bottom. The circles are moved slightly apart to show the orders.
Illustrations are not included in the reading sample
Figure 3.3. Color Orders as Programmed on EPBs
Picture: Author’s drawing
Each status indicator has three circles with different colors:
- Yellow = Fault Alarm
- Red = Trip Alarm
- Green = Normal
On EPB-001/002/003, the color order stacked from the top was Yellow, Red, and Green. The Green circle is static and supposed to be covered by another color if it is active: If there is a fault alarm, the yellow circle is active and blocking the green circle. If there is a trip alarm, the red circle is active and blocking the green circle as well. While in a normal condition, both yellow and red are OFF, showing a green status on the graph. However, somehow, EPB-004 was programmed strangely. Its color order from the top was Yellow, Green, and Red. With the red circle being placed behind the green circle, the red circle is thus never shown upon a shutdown alarm.
Illustrations are not included in the reading sample
Figure 3.4. EPB-004 HMI Status Indicator
Picture: Author’s documentation
Figure 3.4 illustrates what was seen by the technician and what was hidden, with the circles being moved slightly apart to show which color circle lies on top of another. This programming mistake reveals a loophole that remained undetected since the FAT (Factory Acceptance Test) nine years before this incident.
However, it was not the only cause. The bypass procedure itself was not complete because it should have covered all bypass removals. The bypass procedure had not been adequately reviewed by competent parties before being put into practice. With a complete procedure, even with such a normal indication (“Green”), the technician would suspect there was something wrong, provided that they were familiar with the function and logic of the master reset. Unfortunately, the interviews showed otherwise, so we identified this lack of competence as a latent management cause. Not only the competence in SIS in specific but also in SCE (Safety Critical Equipment) at the ORF in general.
3.7 Cause-Effect Diagram
Based on the above facts, the RCA team interactively built an RCA Cause-Effect chart, which is displayed in Figure 3.5. Through the RCA team’s discussions, important and productive causes are printed in bold as candidates for key findings in the next section.
3.8 Key Findings
The interactive sessions on collecting pieces of evidence and constructing the Cause-Effect chart resulted in the RCA’s key findings and their potential solutions/corrective actions:
1. Misleading status indicator à Solution: Change color arrangements on the SIS’ HMI;
2. Incomplete bypass removal procedure à Solution: Revise existing bypass procedure to include complete bypass removals;
3. Insufficient review of the bypass procedure à Involve vendor expert in reviewing SIS procedures;
4. ESD logic latch was still active for EPB-004 à Revise the bypass logic
5. Unfamiliar with logic and function of the master reset à Revisit competency requirements, include SCE in the competency matrix for ORF personnel;
6. Programming mistakes in the project phase → Solution: Improve relevant FAT procedures to detect programming mistakes.
Illustrations are not included in the reading sample
Figure 3.5. Gas Supply Disruption – Cause-Effect Diagram
Picture: Author’s drawing
3.9 Corrective Actions or Solutions
Therefore, based on the key findings above, the RCA team composed a corrective action plan, as exposed in Table 3.2
Table 3.2. Gas Supply Disruption – Corrective Actions
Illustrations are not included in the reading sample
3.10 Tracking RCA Corrective Actions
All corrective actions in Table 3.2, from 0-1 through 0-6, were tracked in the company’s SAP® system as PM (Preventive Maintenance) notifications until each item was completed and verified. Additionally, the company had to update Singapore's EMA regularly on the closure progress. Afterward, this RCA ID “Singapore-ORF-0004” on the gas supply disruption to Singapore was considered “Closed.”
Chapter 4. Actual Case Study 2: Turboexpander Failure in Australia
After hours of an extended layover in Denpasar, the author landed at Perth Airport, Australia, at 01:00 early morning on Monday. The RCA kick-off meeting started at 08:00 Western Australian time. The Operations Manager, Maintenance Manager, and Engineering Managers were present; they briefed the background of the incident. The incident occurred during a maintenance shutdown of Cold Train 1 of the LNG plant. A technician made a mistake by opening a valve out of sequence, which was supposed to be kept closed during the entire shutdown. When I arrived in Perth, the technician already resigned and left the company, probably because of guilt.
At least, it was the atmosphere that the author perceived during the meeting. However, as an experienced PI, I decided to avoid jumping to conclusions. As always, an RCA should start with a clean sheet of paper.
4.1 Background
The offshore LNG plant has two cryogenic process trains running in parallel, Cold Train 1 and Cold Train 2. They use a turboexpander to recover energy from the gas flow and rotate a re-compressor shaft. On 7 May, Cold Train 1 was shut down for routine maintenance. The General Electric-built turboexpander of Cold Train 1 (briefly called “TBX1”) was isolated, and the lubrication oil system was offline.
On 10 May, the maintenance shutdown was complete, and Cold Train 1 was restarted. TBX failed at the start-up, and the crew found that the TBX1 bearing had been damaged. The process engineer found out that on 7 May at 17:42, a section of trapped pressure was released into the TBX1, causing it to rotate without the lube oil system running. The damaged TBX1 was removed. There was no spare available in Australia; thus, the replacement had to be imported from the US. The system was put back online on 23 May. The total unexpected shutdown took place for 14 days with a $1 million revenue loss per day.
From this short description, we can detect three intriguing questions:
1. What caused the bearing failure?
2. Why was the failure unknown until May 10?
3. Why a spare bearing was not made available in Australia?
This is an example where an RCA’s scope could be too broad and overly complex; therefore, the PI decided to consult the stakeholders.
4.2 Problem Statement
After a discussion with the stakeholders, it was decided that the TBX1 bearing failure event should be the undesired outcome of the RCA. We deferred the investigations of the other two because investigating TBX1 bearing failure would yield the most “productive” results from the asset owner’s point of view. Therefore, the second problem (why it was not known until start-up) and the third (why no spares were available locally) were put out of scope. This case shows the importance of being realistic and focusing on solutions with the maximum value to the stakeholders.
Interestingly, the RCA process later on coincidently also answered the third problem (why no spares). In such a case, we can see it as a “bonus.”
Hence, below is the problem statement for this RCA:
- RCA ID: Australia-Offshore-0001
- Undesired outcome: TBX Bearing Failure
- Date/time:
- Failure detected: 10 May, at 00:30
- Recovered: 23 May, at 06:00
- Location: Australia Business Unit, Offshore Darwin LNG
- Impact:
- Revenue loss: $14,000,000
- Repair costs: $1,000,000
- Total impact: $15,000,000
- RCA Category: “Maxi”.
- Other descriptions: The stakeholders expect the RCA to answer the following questions:
- What caused the bearing failure?
4.3 Sequence of Events (SoE)
The PI wrote an initial Sequence of Events based on the background and the problem statement above. The SoE outline was then verified and developed along with the evidence gathering.
Table 4.1 contains the final SoE that was used in the Stakeholders Meeting. The recovery process is not covered in detail because the investigation focuses on the failure event. Also note that if a particular date is not written, it refers to the date in the previous column.
Table 4.1. Turboexpander Failure - Sequence of Events (Final)
Illustrations are not included in the reading sample
4.4 RCA Team
The Australian Business Unit provided three of their senior engineers to join the RCA investigation:
- Process engineer (also as paper evidence lead);
- Mechanical engineer (also as physical evidence lead);
- Maintenance and Reliability engineer (also as people evidence lead).
The author facilitated the team as a Principal Investigator.
4.5 Simple Schematics
The next crucial step is to draw a simple schematic diagram(s) showing only equipment/parts relevant to the incident. This is to make sure that the PI clearly understands the situation before moving ahead with analyzing the root causes. Figure 4.1 below depicts the first schematic of the system.
Illustrations are not included in the reading sample
Figure 4.1. Turboexpander Failure – Schematic #1
Picture: Author’s Drawing
In normal operations, all process ESDVs (Emergency Shutdown Valve) and the fast-stop valve are open, while the JT (Joule-Thompson) valve is closed. Therefore, gas flows from the Expander Inlet Separator (top-left corner) through TBX1 towards the Low-Temperature Separator (LTS) in the bottom right corner. The gas flow rotates the turboexpander (TBX1), whose rotation is used to rotate the re-compressor. The lube oil system is running to allow TBX1 to rotate at high speed (typically 15,000 RPM), preventing the bearing from getting damaged.
During a maintenance shutdown, Process ESDV #1 valve and Fast-stop valve are closed, blocking the gas flowing into TBX1. JT Valve and Process ESDV #2 are open, allowing the gas to bypass the TBX1 and flow directly to the LTS. This is where the system is called running in “JT Mode,” a system that is less than ideal from the hydrocarbon processing point of view. For this CT1 maintenance shutdown, the flow of CT1 was then decreased further to zero. However, notice that some high-pressure gas is still trapped inside a pipe segment marked red. This trapped gas makes any equipment installed downstream of it (i.e., TBX1) vulnerable. The pressure difference between HP (High Pressure) and LP (Low Pressure) segments is monitored by a DP (Differential Pressure) gauge.
When the maintenance shutdown is complete, the system is put back online step-by-step in sequence. The lube oil system, for example, is restarted before the TBX1 is allowed to rotate.
Illustrations are not included in the reading sample
Figure 4.2. Turboexpander Failure – Schematic #2
Picture: Author’s Drawing
A second schematic is necessary to simulate how the job was performed during the incident. During the work carried out on TBX1, an Operations Technician and an Instrument Technician attended the turboexpander. They communicated by radio with operators in the Control Room on another platform, who were coordinating the shutdown job.
4.6 Evidence Highlights
Paper evidence contributed significant knowledge to the RCA. During the paper evidence gathering, process parameters (pressures, flow rates, temperatures) from the DCS data historian in Perth were utilized to review and confirm the SoE. Figure 4.3 below displays a graph of some relevant parameters. Note that the time domain on the chart is lagging by one and a half hours because of the time zone difference between the database location and the location of the incident. For instance, the TBX1 stopped at 10:30 in the incident location but lies at 09:00 on the chart’s scale.
Illustrations are not included in the reading sample
Figure 4.3. Visuals of Selected Process Parameters
Picture: Author’s documentation
Do not get overly concerned by the chart's complexity, which is full of colorful fine lines. The purpose of Figure 4.3 is to illustrate how a piece of evidence is used to confirm the SoE and the causes within the Why-Tree or Cause-Effect diagram. Black arrows and notes mark important milestones. Refer to the Sequence of Events (Table 4.1) and simple schematic (Figure 4.1) for equipment names and the list of abbreviations at the beginning of this book.
The most important event is on 7 May at 17:42, when the DP of TBX1 drops from 70 to 0 barg. This was when the high-pressure trapped gas was released into TBX1 while the lube oil pressure was still zero (meaning no lubrication was working). The turboexpander’s bearing was damaged as a result. This flow of cold fluid was also confirmed by the increase of LTS pressure from 0.03 to 1.6 barg and the decrease of LTS temperature from -3 to -21 Celcius.
Paper and physical evidence mention that two conditions that occurred in parallel are the immediate causes of the turboexpander’s bearing failure:
- The trapped gas was released and rotated the TBX1; while:
- The lube oil system was shut down, no lubrication on the bearing.
Discussions with process engineers revealed that such a trapped gas scenario is uncommon. The design could have, for instance, added a bleed point to release the high-pressure gas [5]. Interestingly, we found that an MoC (Management of Change) numbered 11925958 addressing this trapped gas issue had been initiated two years before the incident. Despite this risk being recognized by the MoC, the MoC had been put on hold because the safety and environmental risks were low. The MoC didn’t evaluate the financial consequences, which also explains why no spares were made available in Australia (question no. 3 in Section 4.1). The absence of evaluating financial consequences further implies a competency issue in proposing an MoC.
Illustrations are not included in the reading sample
Figure 4.4. Turboexpander Failure - Why-Tree Diagram
Picture: Author’s drawing
Competency issues were also prominent when gathering people evidence. The trapped gas had a higher pressure (100 barg) than usual (40 barg), and the lubrication system was offline. Both were caused by the out-of-sequence shutdown following a lack of understanding of the Working Instructions (WI). The RCA team checked the WI used by technicians attending TBX1 (Schematic #2 in Figure 4.2). We found two gaps in the WI. Firstly, it contained some ambiguous and unclear steps, making it difficult for personnel who work under schedule pressure to follow. Secondly, we found the WI held by operators in the Control Room (again, Schematic #2 in Figure 4.2) differed from the one held by the technicians. There were, hence, multiple versions of the same procedure. This marks document control as a latent issue in the Australian LNG operation.
4.7 Why-Tree Diagram
Interactively with productive evidence-gathering sessions, the RCA team constructed an interesting Why-Tree, as drawn in Figure 4.4. Based on the discussion within the RCA team, important causes are printed in bold; they make good candidates for key findings in the next section.
For simplicity, a leg of causes named “WI not followed” is shared by several upper causes (marked by dotted lines). Also, notice a branch that is marked “OR” (as opposed to the default “AND”) and a node marked “no evidence.” These objects remain in the diagram for investigation documentation purposes.
4.8 Key Findings
Having analyzed the pieces of evidence and constructed the Why-Tree, the RCA team discussed the key findings and identified their potential solutions:
1. Trapped gas at a high pressure à Solution: Review the depressurizing and isolation process to identify any alternative blowdown methods of the trapped pressure.
2. MoC no. 11925958 was put on hold à Solution: Revisit MoC #11925958 to review the proposed change, e.g., including possible logic changes on valve sequencing.
3. Design issue on trapped gas à Solution: Revisit the LNG plant’s PHA (Process Hazard Analysis) to review all systems for trapped pressure scenarios.
4. Lack of understanding of the Working Instruction (WI) à Solution: Revise the Work Instruction for Cold Train isolation-desolation to improve understanding of the risks and treat it as a controlled document.
5. Multiple versions of WI exist à Solution: Validate the document management process to ensure all procedures are reviewed, validated, and approved before use.
6. Technician’s competency issue à Solution: Review the competency program of the offshore personnel and ensure all personnel are signed up for competency.
4.9 Corrective Actions or Solutions
Based on the key findings above and their potential solutions, Table 4.2 below summarizes the corrective actions or solutions resulting from the Australian LNG turboexpander failure RCA investigation. The RCA was assigned RCA ID “Australia-Offshore-0001.”
Table 4.2. Turboexpander Failure – Corrective Actions
Illustrations are not included in the reading sample
4.10 Tracking of RCA Corrective Actions
All corrective actions in Table 4.2, from G-1 through G-6, were tracked in the company’s SAP® system of the Australian Business Unit as PM (Preventive Maintenance) notifications until each item was completed and verified. Afterward, this RCA ID “Australia-Offshore-0001” on the LNG plant turboexpander failure was considered “Closed.”
Chapter 5. Actual Case Study 3: Offshore Electrical Fire
In late November, we heard shocking news from offshore: An electrical fire with an explosion had occurred on an FPSO (Floating Production, Storage, Offtake) vessel in the South China Sea. An FPSO is constantly receiving, processing, storing, and shipping highly flammable natural gas, and thus any unwanted fire is unacceptable. Furthermore, it’s offshore where immediate escape is harrowing, if not impossible.
It was a severe safety incident indeed. Have you read about Piper Alpha, a North Sea accident that changed the shape of the offshore industry? The incident that killed 167 workers was among the drivers of why RCA became mandatory. Not to mention the commercial loss: The FPSO was a sales hub for other producing gas platforms. Hence, the incident caused a total shutdown of the entire gas network for seven days. It was a “Maxi” category of incident [7]. Bruce, the field manager, urgently requested that my department send me to lead an RCA investigation.
5.1 Background
On 26 November at 01:41 AM, a PSD-0 (Process Shutdown Zero alias total shutdown) occurred on Natuna FPSO. The F&G (Fire and Gas) alarm was active, and personnel onboard were called to muster areas while the fire team inspected the source of the fire. Smoke came out from the Main Switchgear Room, and the fire was found at the Alpha-Platform feeder on the 13.8 kV main switchgear. The fire team put off the fire using a CO2 Extinguisher. After the fire, the electrical technicians cleaned up the site, finding the circuit breaker had been severely burnt.
Restoration efforts took seven days by mobilizing electrical experts from headquarters; the power was put online on 2 December by restoring Bus-A and starting Turbine Generator B at 16:20. The damage costs $30,000 to repair, while the extended total downtime has caused a gas production loss worth more than $25 million.
5.2 Problem Statement
The asset owner created an RCA problem statement and sent it to the principal investigator, containing the following information:
- RCA ID: Offshore-FPSO-0004
- Undesired outcome: Electrical Fire
- Date/time:
- Event: 26 November, at 01:41
- Recovered: 2 December, at 16:20
- Location: Offshore Business Unit, Natuna FPSO
- Impact:
- Revenue loss: $25,000,000
- Repair costs: $30,000
- Total impact: $25,030,000
- RCA Category: “Maxi”.
- Other descriptions: The stakeholders expect the RCA to answer the following questions:
- What caused the fire?
- Why did the protection system fail to shut down the impacted power system?
5.3 Sequence of Events (SoE)
Table 5.1 below is the SoE of the incident. For conciseness, the electrical recovery process details are not included in the table.
Table 5.1. Sequence of Events – Offshore Electrical Fire
Illustrations are not included in the reading sample
5.4 RCA Team
This incident raised a major concern from the top management; therefore, the leadership team dedicated several resources to the RCA team. The author (Andy) was appointed Principal Investigator, assisted by several specialists, namely:
- Electrical engineers (one of them also as people evidence lead);
- Electrical maintenance technicians (one as physical evidence lead);
- Instrumentation and control engineers (one as paper evidence lead);
- Offshore Installation Manager (OIM).
There is one interesting fact: The involvement of the OIM in the last line above was a special request from Bruce, the Field Manager. It raises a question: Won’t it be a conflict of interests, as an OIM is a part of the organization involved in the incident? There is no simple answer to this, but at least there are two considerations for accepting such requests:
- Skills and authority of the PI: If the PI is highly experienced and thus can firmly put themself in the authority position, it’s improbable that the OIM will intervene with them.
- Maturity of RCA culture within the organization: If the RCA culture is already mature, e.g., no blaming, being receptive, and not defensive, we can expect everybody in the team to stay objective.
5.5 Simple Schematics
The PI drew two simple schematics to isolate the equipment and parts of concerns from the others. The first is the general electrical configuration of the whole FPSO, shown in Figure 5.1.
Illustrations are not included in the reading sample
Figure 5.1. Electrical Fire – Simple Schematic #1
Picture: Author’s drawing
Figure 5.1 gives a bird’s view of what had failed. The FPSO apparently employs two electrical power backbones at 13.8 kV, Bus A and Bus B, connected by a bus tie. When an anomaly occurs within Bus A (like what had happened), the associated bus differential (87B) and relays (86-1 and 86-2) should protect the backbone by cutting off the electricity supplied by GTG (Gas Turbine Generators) A to C. Bus B stays online, supplied by GTG D to E. This didn’t happen: the anomaly developed until a real fire occurred, and both Bus A and Bus B were shut down due to PSD-0.
The second in Figure 5.2 gives a closer look at the failed equipment. It shows the electrical switchgear with three phases (A, B, C); each phase must always be separate.
Illustrations are not included in the reading sample
Figure 5.2. Electrical Fire – Simple Schematic #2
Picture: Author’s drawing
5.6 Evidence Highlights
During the evidence gathering, some evidence was found to be worth highlighting. From gathering physical evidence, Figure 5.3 shows a trace of an arc fault causing a short circuit between phase A and phase B on the circuit breaker.
Illustrations are not included in the reading sample
Figure 5.3. Electrical Fire – Evidence Highlights 1
Photograph: Author’s documentation
Meanwhile, Figure 5.4 shows the case of an intruding animal. The photograph on the left is of the rat found on the premises, which had acted as a bridge between Phase A and Phase B. The one on the right shows a cable clearance on the cubicle where the rat might have found its way to get in.
Illustrations are not included in the reading sample
Figure 5.4. Electrical Fire – Evidence Highlights 2
Photograph: Author’s documentation
Paper evidence reveals that the F&G system only activated the alarm and did not trigger the PSD-0. It was shown by the C&E (Cause and Effect) logic of the system (Note: C&E is an electrical control’s Cause and Effect, not the RCA Cause-Effect we will build below). So, what caused the PSD-0? Further paper evidence finds that the power outage triggered the PSD-0 due to a generator trip caused by a ground fault of relays. The arc fault and fire did not trigger the PSD-0.
Further, the F&G alarm was not activated by the 87B Bus Differential, as shown by the F&G event logs. The Principal Investigator then assigned a task to the electrical specialist to conduct a test on 87B’s performance, with the result that the delay time drifted from 0.1 seconds to 0.25 seconds. In other words, the arc fault was not detected and did not cause a power shutdown, letting it develop into a fire. No alternative arc-fault detection was installed in the system.
Maintenance records show none about the calibration of the relays. During the interviews, Electrical Technicians stated they were reluctant to decide on relay calibration because they feared it would cause a shutdown.
5.7 Cause-Effect Diagram
A cause-effect diagram was constructed collaboratively by the RCA team, along with the evidence-gathering process. The diagram is shown in Figure 5.5. Notice a branch that is marked “OR” (as opposed to the default “AND”) and a node marked “No evidence”. These objects remain in the diagram for investigation documentation purposes. After discussions, the RCA team agreed to pinpoint important causes as candidates for key findings, printed in bold.
Illustrations are not included in the reading sample
Figure 5.5. Electrical Fire – Cause-Effect Diagram
Picture: Author’s drawing
5.8 Key Findings
The key findings and potential corrective actions are listed below based on the evidence highlights and the Cause-Effect chart above.
1. Animal (rat) intrusion à Solution: Apply a more stringent pest control and seal the cable clearance holes;
2. No interval barrier on energized parts of CBs à Solution: Install interval barriers;
3. Defective 87B protection relay, inadequate settings of bus differential Protection, e.g., when extreme over-current occurs (> 1,000 Amps), no delay time à Solution: (1) Improve settings (2) perform a comprehensive review in the next shutdown;
4. No alternative arc-fault detection in switchgear rooms à Solution: Apply a second method of arc-fault detection;
5. No specific procedures on 87B à Solution: Establish procedures for high-risk systems;
6. Awareness and competency: Worrying of black-out as a consequence à Solution: Mentoring and training by an Electrical Specialist.
Please note in Figure 3.5 that these key findings are marked as bold on the cause-effect diagram.
5.9 Corrective Actions or Solutions
Based on the key findings above and their potential solutions, Table 5.2 below summarizes the corrective actions or solutions resulting from this Electrical Fire RCA investigation.
Table 5.2. Electrical Fire - Corrective Actions
Illustrations are not included in the reading sample
5.10 Tracking of RCA Corrective Actions
All corrective actions in Table 5.2, from F-1 through F-7, were tracked in the company’s SAP® system as PM (Preventive Maintenance) notifications until each item was completed and verified. Afterward, this RCA ID “Offshore-FPSO-0004” on the Offshore Electrical Fire was considered “Closed.”
Chapter 6. Becoming a Learning Organization
6.1 Monitoring Assets and Asset Management System
Numerous companies from different industries worldwide have proved that the outcome of practicing a systematic asset management system is enormous. If we refer to its applicable set of standards, i.e., the ISO-55000 series, closing the loop of the PDCA management process is the continuous improvement, or "Review" (group of clauses no. 9) and "Improve" (group of clauses no. 10). They can be best described as the flow diagram in Figure 6.1.
Illustrations are not included in the reading sample
Figure 6.1. Asset Management Review and Improvement
Picture: Author’s elaboration from Marjoko, Royer-Duffait & Saradjian [6]
The standard mandates that both the asset management system and the assets are monitored with reactive and proactive approaches. As a defect elimination tool, RCA can be seen as the backbone of reactive monitoring. The scope of RCA is marked with a purple dotted box. RCA results shall provide feedback for the proactive monitoring and even the planning and operating activities (Figure 1.2 in Chapter 1). In addition to completing the “PDCA” management cycle (Figure 1.2 in Chapter 1), RCA also forms a learning cycle for the organization.
6.2 A Means of Learning for the Organization
An organization should embrace a learning cycle toward excellence [6]. RCA not only eliminates defects but is also a powerful tool for learning. The spirit is to improve, not to blame, and to accept what has been learned instead of being defensive [7]. RCA is such a powerful tool that it should also be applied to any undesired outcome other than safety and reliability, to name a few: quality, environment, compliance, supply chain disruption, and customer satisfaction [1].
Illustrations are not included in the reading sample
Figure 6.2. A Why-Tree Diagram of Clause 8.2 Incompliance
Picture: Author’s drawing
Let us look at Figure 6.2, which illustrates the use of RCA in an ISO-55001 compliance audit [1]. Suppose there was a non-compliance finding to clause 8.2 (Management of Change) of ISO-55001. The finding was an increase in the pressure rating of a gas pipeline without a proper MoC. It was revealed that the MoC had been initiated but then put on hold in the approval stage because the accountability was unclear in the RACI (Responsible, Accountable, Consulted, Informed) chart. Competence appears to be a latent management cause, and training was proposed as one of the corrective actions.
6.3 A Means of Learning for the Individuals
RCA also provides an excellent learning opportunity for individuals within the organization [7]. It is a means to develop both hard and soft skills. Being appointed PI for more than 15 incidents was a privilege from which I learned more than other job assignments. I was assigned to be the PI for an RCA of a Sikorsky® S-76 helicopter’s abnormal landing on one of our offshore platforms, but I didn’t have an aeronautical background. The SoE was rather tricky to develop because the landing involved coincidences: The aircraft was facing the East (due to the wind direction), the sea was calm with a glass-like surface, and the sun was rising. The pilot had to turn the helicopter to avoid the sun glare, making the aircraft lose a little lift force. The author had to invite some helicopter pilots (from another company, not witnessing the incident) to understand what was happening. An excellent chance of learning, indeed.
This even applies to young, enthusiastic people as a career development tool. Imagine a young design engineer with 2-3 years of experience. They are exposed in the RCA teams of several investigations. When the engineer is back in the design role, the chance is that they will design better, more reliable, and more maintainable systems [6].
6.4 RCA Software Tool
Finally, we could conduct an RCA without any software tools, purely on a pen, paper, marker, and whiteboard. However, a simple word processor and a drawing application will help compose a more organized RCA report. Several software tools are available in the market to provide even further assistance, i.e., managing work processes, drawing diagrams, storing data, and tracking solutions.
When this book is written, a free yet versatile cloud-based RCA software tool named “RCA Soft” is being developed and tested by qualified RCA experts. Stay tuned and start using the free software when it is already available online at this web address: RCA-Soft.com.
For sharing questions, training, and assistance with the RCA process, the author can be reached at RCAsoft@iCloud.com.
Bibliography
1. DeZoort, F. T. & Pollard, T. J. (2023). An evaluation of root cause analysis used by internal auditors. Journal of Accounting and Public Policy. Vol. 42, Issue 3.
2. EASA (2002). Root Cause Failure Analysis: Pinpointing the true cause of electric motor failures. Electrical Apparatus Service Association: St. Louis.
3. ISO (2018). International Standard: ISO 55001:2018. Asset Management - Management systems - Requirements.
4. Ito, A., et al. (2022). Improved root cause analysis supporting resilient production systems. Journal of Manufacturing Systems. Vol. 64, pp.468-478.
5. Khodadadi-Mousiri, A., et al. (2023). Consequence modeling and root cause analysis (RCA) of the real explosion of a methane pressure vessel in a gas refinery. Helyon. Vol. 9, Issue 4, April 2023.
6. Marjoko, A., Royer-Duffait, A., Saradjian, C. (2023). Adaptive Corporate Culture in International Business Management. ISBN 9783346810038. GRIN Publishing: Munich. https://www.grin.com/document/1323193.
7. Nelms, C. Robert., n.d., Latent Causes of Industrial Failure. IEEE Explore. Retrieved on 2 April 2024 from https://ieeexplore.ieee.org/document/624866.
8. The Institute of Asset Management (2024). The Anatomy of Asset Management. IAM: Bristol.
List of Figures
Figure 1.1. Reliability and Safety Management in Asset Life Cycle
Figure 1.2. RCA in the PDCA of Asset Management
Figure 2.1. Example of a Corporate Risk Matrix
Figure 2.2. Example of an Incident’s Simple Schematics
Figure 2.3. Cause-Effect, Horizontal Style
Figure 2.4. Why-Tree, Vertical Style
Figure 2.5. Layers of Causations
Figure 2.6. Flow of an RCA Closure Tracking
Figure 3.1. Gas Supply Disruption - Schematic
Figure 3.2. “Green” Status of EPB-004 on HMI
Figure 3.3. Color Orders as Programmed on EPBs
Figure 3.4. EPB-004 HMI Status Indicator
Figure 3.5. Gas Supply Disruption – Cause-Effect Diagram
Figure 4.1. Turboexpander Failure – Schematic #1
Figure 4.2. Turboexpander Failure – Schematic #2
Figure 4.3. Visuals of Selected Process Parameters
Figure 4.4. Turboexpander Failure - Why-Tree Diagram
Figure 5.1. Electrical Fire – Simple Schematic #1
Figure 5.2. Electrical Fire – Simple Schematic #2
Figure 5.3. Electrical Fire – Evidence Highlights 1
Figure 5.4. Electrical Fire – Evidence Highlights 2
Figure 5.5. Electrical Fire – Cause-Effect Diagram
Figure 6.1. Asset Management Review and Improvement
Figure 6.2. A Why-Tree Diagram of Clause 8.2 Incompliance
List of Tables
Table 2.1. Sequence of Events Example
Table 2.2. Examples of Corrective Actions or Solutions
Table 3.1. Gas Supply Disruption – Sequence of Events
Table 3.2. Gas Supply Disruption – Corrective Actions
Table 4.1. Turboexpander Failure - Sequence of Events (Final)
Table 4.2. Turboexpander Failure – Corrective Actions
Table 5.1. Sequence of Events – Offshore Electrical Fire
Table 5.2. Electrical Fire - Corrective Actions
[...]
- Arbeit zitieren
- Andy Marjoko (Autor:in), 2024, Mastering Root Cause Analysis (RCA). Real-World Examples and Strategies, München, GRIN Verlag, https://www.grin.com/document/1472836
-
-
-
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen. -
-
Laden Sie Ihre eigenen Arbeiten hoch! Geld verdienen und iPhone X gewinnen.