The previous article on this topic can be found via this link.
The layered approach to upkeep data center infrastructure availability should not look like the Swiss cheese, i.e. the hazard or trigger should be eventually stopped and preferably as early as possible.
The layers should include the following:
- Design (in accordance with design intent of owner) with either concurrent maintainability objective or fault tolerance
- Implementation (in accordance with design brief) and fully tested via comprehensive testing and commissioning phase before handover with fully documented SOPs/MOPs/EOPs
- Maintenance and Operations Management, work by equipment service providers or any work on site through Method of Statement and Risk Assessment matrix by suitably qualified person/persons
- Incident and Problem management process, escalation management and mitigation process
and so forth
Possible problems arising from inadequacy in each of the layer can result in:
- Inherent Design / Setting flaw
- Outdated / swiss cheese situation
- Requires analysis and manual intervention
- Error Producing Conditions (EPC)
- Weakness in manual processes
- Inadequate automation
- Inadequate training / familiarity
- Inadequate operations procedures
- Insufficient Information / knowledge
- Capacity limit reached earlier than design intent
- Inadequate training / knowledge
- Inadequate documentations
- Insufficient Risk Assessment
- MOS / RA, risk matrix
- Vendor experience
Learn from other industry
Our data center industry is a relatively young industry and there are other industries with mission critical infrastructure that have undergone extensive research and iterative enhancements which we can learn from and adopt.
- Airline’s Crew Resource Management
- Checklist and double checking by pilot and co-pilot on the airplane air-worthiness
- Communications within the cockpit and the cabin staff with the cockpit to ensure timely and prioritized response
- US Nuclear Regulatory Commission
- Standardized Plant Analysis Risk – Human Reliability Analysis (SPAR-H) method to take account of the potential for human error
- OECD’s Nuclear Energy Agency
- Ways to avoid human error, e.g.,
- Systems should also be designed to limit the need for human intervention
- distinctive and consistent labelling of equipment, control panels & documents;
- displaying information concerning the state of the plant so that the operator don’t need to guess and make a faulty diagnosis; and
- designing systems to give unambiguous responses to operator actions so incorrect actions can be easily identified.
- operators to be better trained for plant emergencies, use of simulators
- Ways to avoid human error, e.g.,
Error Reduction Strategy and Error Predictor
In addition, error reducing strategies can be applied in all areas of data center maintenance and operations management to reduce the probability of occurrence of human error. Whether in design of the data center power and cooling infrastructure, or determining the risk of operations of particular maintenance operations (e.g. power switch-over exercise to perform UPS or back-up generator maintenance), all the strategy below should be applied.
Take for example the case of the AWS US-East-1 outage incident (http://mashable.com/2017/03/02/what-caused-amazon-aws-s3-outage/), the command set is powerful and a typo could bring down a lot of servers real quick. So AWS said in their post incident summary (https://aws.amazon.com/message/41926/) that they will limit the speed of the effect of the command and tool, i.e. put in safety check which is basically an application of the constraint strategy.
When service or repair task are assigned to operations staff, or by qualified technicians of equipment service provider, an evaluation of the existence of error precursors and eliminating these pre-cursors will reduce the likelihood of human error. For example, the combination of time pressure, inexperienced staff already at the end of the long work shift and ambiguous task objective are all contributor to a higher risk of the assigned task. Eliminate, reduce, and re-direct to an experienced staff at the start of the work shift with clear task objective will reduce the risk of the assigned task.
Risk Mitigation is a Continuous Process
A multi-prong multi-layer and attention to details approach are required to mitigate the risk of human error causing an outage in a data center facility.
Design and implementation of a data center to a set of clear and tested design intent (e.g. objective of data center being concurrently maintainable). Day-in and day-out the operations staff, vendors, client personnel interact with the system within the data center. So there need to be a well oil system in place, not only just documentations, that works 24×7 for as long as the data center is in existence.
An iterative risk mitigation system, relying upon consistent management support and attention, with knowledge learned from near misses and incidents are key attributes of an environment that is resilient in terms of the human aspect.
We Human can reduce Human Error, effort required
We should look at the data center organization especially the operations team, the resources and tools, the capability of the operations team, and so forth. A culture of no blame, and encouraging active participation by all staff to address potential weakness or error precursors, addressing near-miss which is a sign of error inducing conditions, are important to mitigate effects of human errors. We should get away from pointing fingers and to learn from past problems, like what AWS did with their incidents. And our data center industry can do more to share and learn from one another, to prevent future occurrence of issues that were faced and dealt with elsewhere.
This built-up knowledge of good practices should be documented and disseminated, with management support. The weakest link is an inexperience staff hesitating or worse making a wrong decision, so training everyone on the operations team is critical to maintaining availability of data center.
A periodic (for example annual basis) no-nonsense third party data center operations and management review coupled with improvement plans to strengthen those weakest links will boost insight and assurance to data center C-level executives, data center operations managers, and clients. Most operations manager will be too busy to review their own data center operations, coupled with the difficult position of finding your own fault and limited experience if the staff have not worked in more than one or two data center sites, therefore a third party operations and management review is the next best thing to enhance resilience against human error provided it has the full co-operation from top to bottom of the data center staff.
Furthermore, if a data center service provider has grown beyond 2 to 3 data centers, it will be difficult to consistently manage the data center operations across them especially if they are managed independently. A third party review that is applied to all of them will help to reign in inconsistent operations processes, subject to having a central data center operations programme function within the data center service provide, of course.
Therefore, a data center facility is ultimately dependent on well trained and knowledgeable staff, whom are clear about their data center facility information or knows where to quickly find the documentation that contains the detail information, do the risk assessment work of evaluating equipment service vendor or upgrade works properly.
- It is worthwhile to commit resources to reduce errors
- We can do improve our resiliency and thereby uptime through available methods and tools
- There are proven methods and tools we can borrow from other mission critical environments
- Third party data center operations and management review coupled with improvement plan should be considered for large data center operations especially those that have multiple sites