AmalfiCORE Blog

Disaster Exercise – Observing and Evaluating

Exercise activity is best observed and evaluated by individuals specifically assigned to the task of observation or evaluation.

A good rule of thumb is to have enough evaluators to observe each and every key activity, inject, and response by the participants. Often, it is either cost prohibitive or simply not feasible to have more than one or two evaluators. In such cases, the exercise should be designed to accommodate the number of evaluators.

The inject messages and interrupts that help change and move the scenario along should be spaced in such a way that the evaluator(s) has time to adequately observe and note each inject or key activity.

A good evaluator should above all be honest, fair, objective and a keen observer of detail while at the same time one who can interpret actions and activities as they unfold. In addition, these are some attributes of a good evaluator should be able to:

§  understand the organization and how it operates

§  be available to attend meetings

§  discern between the exercise activity and the people, and focus on the activity

§  determine if the exercise objectives are being met

§  know that if problems arise, how to inform the facilitator in a way that helps move the scenario along and help keep things ‘alive’

§  observe without getting drawn into the scenario or getting distracted

§  clearly utilize objectives, observe outcomes, and track key injects to successful conclusion.

In order for the participants to learn from the exercise in a supportive way, evaluators should refrain from making personal attacks, or singling out individuals behavior. Even singling out individuals who may have stood out doing a tasks very well can alienate others.

Comments that are helpful are those that connect the tasks performed to the objectives of the exercise. Also, the evaluators can acknowledge an issue that may have been cause by the nature of the event, the inject message design, or an inability to fully simulate a particular activity.

In the event that an evaluator primarily critiques individuals, the facilitator can do a few things to help mitigate any backlash or uneasiness. The facilitator can help diffuse the situation immediately by mentioning the issue in a broader context. Of course, a confident facilitator could counteract the individual critique by simply saying ‘we are not here to point out individual actions, we are a team and we all own every aspect of how the scenario unfolded. Therefore, let’s refrain from directing comments to individuals.”

Prior to the exercise, the evaluators should be instructed on how to provide proper feedback in a group. And if needed, subsequently, the evaluator(s) should be with by the facilitator privately to correct the approach taken by the evaluators if inappropriate.

Leading the Disaster Exercise Design Team

Exercise design team meetings are used to orient the design team members to goals and objectives, brainstorm a story narrative, and create injects into the scenario to validate the areas being assessed. Further, the design meetings are used to prepare all aspects of the exercise event including materials like participant guides, actor and simulator team instructions, and evaluation forms.

The length of each design meeting and the number of meetings needed depends on the size and complexity of the exercise. Full-scale operational exercises may take several months to prepare, whereas, a simple table-top scenario may only take a few days or meetings.

If there were only time for two design team meetings, I would approach the meeting agendas as follows:

Meeting 1:

• Orient participants to the exercise, what is important and painting a picture of success
• Review plan, goals objectives, describe the basic narrative
• Brainstorm to obtain team input regarding what might happen, what would be impacted.
• In real time, update the scenario based on modifications to the 5 buckets with new injects
• Assign homework for the team to create 5-7 new injects

Meeting 2:

• Review every new inject
• Have the team validate injects and eliminate any conflicts, make it real!
• Brainstorm what has not been covered
• Possible suggest more homework to evolve the narrative with new modifications
• Review final preparations for the exercise event

Exercise Injects

Table-top disaster exercises provide a safe, low-stress environment within which participants can validate policy and procedure, consider what-if scenarios, and evaluate and assess their capabilities to manage a major incident.

Key to bringing practical realism to a disaster exercise are “injects”. An inject is new data or information. The inject is provided to the participants by the facilitator, evaluator, simulation team or others. The inject continues the story began from the baseline narrative and helps move the storyline farther across in the continuum.

There are five inject categories:

§  People

§  Facilities

§  Technology

§  Mission-critical activities at risk

§  Communications

There may be other injects that are of a more broad type or nature, as well.

The design team should research the type of hazards and threats that could impact the mission critical functions, and/or may focus a particular exercise on one of the five inject categories, or both. Sources of information include:

§  Subject matter experts of important functions

§  HSEEP lessons learned (FEMA)

§  Priority injects necessary to assess specific functions can come from other departments that use that function and thus are internal customers

§  Process and standard operating procedures of the organization

§  Injects should reflect the basic goals and objectives, therefore resources associated with the objectives can be consulted

§  business impact analysis or service impact assessments

For example, if the scenario is a Emergency Sheltering Exercise, important injects would be People and Communications. In this case, injects might include citizens looking for missing loved ones, medical emergencies with evacuees at a shelter, press and media seeking information.

Keeping Momentum in an Exercise Design Team

Once an exercise design team has started its work, momentum is important. If good effort was made early on to include all the key players and commitment was confirmed, then the process should go smoothly.

Invariably, everyday schedules or interrupts get in the way or perhaps some design team members become disenchanted or become too busy with other work. When this happens the result may be lackluster creation of injects. If the missing or inadequate injects are mission-critical, then the design team has a problem that needs immediate attention. Remember the goals of injects is to challenge the exercise participants.

As team leader, here are a few suggestions to get the process back on track:

1.      Understand. Contact the members in question and simply find out what’s going on. Without knowing the reasons for missing work, no further actions will be helpful.

2.      Importance. Remind the members of the importance of their input to challenge the exercise participants.

3.      Help. Determine if there is any misunderstanding in now to go about creating injects or helping with the scenario framework. Offer help and bring clarity to the issue and restate the expectations.

4.      Recommit. Obtain a recommitment from the members including a due date.

5.      Escalate. If necessary, it’s now time to speak directly to the members manager or supervisor. Either get someone else from the same function to do the work or have the manager speak with the member and help the assignment get completed.

6.      Last resort. Go outside the function and seek help from others who have created similar injects for similar situations. There are some good online references that describe injects for similar scenarios.

This comes down to basic team leadership. Being fair and open about the desired results and getting commitment usually helps avoid the problem.

Disaster Exercises: ‘Hard’ Incident or ‘Soft’ Event?

Whether or not an exercise is ‘hard’ or ‘soft’ often depends on the type of business or organization wishing to prepare and conduct the practice event. It could also depend on recent events in their industry or geography, and current news.

Designing the narrative of a disaster exercise can be tricky. This is especially true when it comes to providing a practical and realistic scenario, both of which are necessary for success.

‘Hard’ incidents are those described by significant, physical tragedies or crisis. People can identify with a variety of expected or known hazards occurring, for example:

§  severe natural weather disaster (hurricanes, earthquakes, tornados)\

§  man-made accidents (train derailment spilling hazardous materials, plane crash, cruise liner sinking)

§  terrorist attacks (anthrax release, explosions, shootings, incendiary devices)

§  technological accidents (data center fires, equipment failures, etc).

Soft events can include non-physical crisis which can also have significant negative impact on a business, an organization, and people. For example:

§  name brand and image tarnished by ‘bad press’

§  security breach and information security leaks

§  poor customer service

§  bad product leading to massive product recalls

§  use of environmentally disruptive technology

§  employment or labor issues

§  geopolitics

§  insider trading, etc.

Both ‘hard’ and ‘soft’ incident scenarios have some similarities:

§  impetus that causes the incident to occur must be believable

§  psychological impacts may be present

§  need to follow a exercise plan and process template for the scenario to play out

§  decision-making, scenario interrupts, resource access are all necessary

There are some differences that ‘soft’ incidents don’t provide compared to ‘hard’ incidents:

• the number of people directly involved in the actual crisis and therefore involved in the exercise may be quite small for ‘soft’ incidents (e.g. a legal battle with bad press does not typically involved more than the executives, legal counsel, perhaps the public relations department)
• table-top scenarios are more common since the events occurring over time cannot be created without great difficulty (difficult to practice mass outcry, or actually have a breach in security)
• simulation of contact to outside resources may be inappropriate and cause undue alarm or confusion when no real crisis exists

The ‘hard’ and realistic scenarios are often easier to understand for most, although can be difficult to believe will actually ever occur.

Key Elements of a Disaster Exercise Narrative

A well crafted disaster exercise narrative is a critical element of any disaster training or exercise process. The narrative sets the stage for the exercise scenario. It provides background information and helps participants approach the exercise as a real and plausible event. The narrative also sets the stage and puts the players at the beginning of the exercise.

Several sources can be found to provide guidance on which elements to include in the narrative. Of the lists I’ve reviewed and based on my personal experience writing narratives, I believe the following questions need to be answered. In so doing, the answers to these questions provide the participants with all that’s necessary to understand, begin, and complete the exercise.

What is or are the….

[Set up]

a.      Assumptions?

b.      Resources available during the exercise versus what needs to be simulated?

c.      Procedures enacted prior to disaster?

[Event]

d.      Hypothetical moment in time and location of the disaster?

e.      Relevant weather conditions, threat or exposure level and continuation possibility?

f.       Notification given, and was there any advance warning?

g.      Description of what has triggered the event and situation or sequence of events leading up to the disaster?

h.      Damage, or impact to the organization, function or population?

i.       Damage internal to the organization versus external?

j.       Description of the speed, strength, depth, or level of continued danger?

k.      Likelihood of spread of the event to grow beyond its current location and go regional?

[Response]

l.       Response by the organization to the disaster currently being taken?

m.    Current response by local emergency services and what else can be expected?

n.      Status of all personnel?

o.     Status and availability of alternate and backup locations, vendors and suppliers, and utilities?

[Other ]

p.      Future predictions in the recovery scenario?

q.      Other factors that might influence emergency procedures?

Collectively, a narrative can answer these questions and provide a solid and clear foundation upon which to conduct a disaster exercise.

Reference

[1] Musson, M. (2007) Choosing and Developing the Test Scenario. In P. Rothstein (ed), Disaster Recovery Testing, Exercising Your Contingency Plan. (pp 47-56) Connecticut, Rothstein Associates

Why Are We Doing This Exercise?

It can be difficult to convince an organization to take the time and incur the cost of conducting and exercise, particularly a Functional or Full-Scale exercise. I’ve sometimes heard an answer to the basic question “Why are we doing this exercise” to be “because we’ve been told we have to”. While valid, hopefully there is a better reason. Why is this important to the exercise? Because without a compelling understanding of the ‘why’ a few things occur:

1.     Participation is average and perfunctory at best

2.     The level of possible learning may be diminished matched to the level of enthusiasm or purpose

3.     The task of creating a suitable narrative based on clear objectives may be clouded resulting in a mis-matched scenario.

4.     The entire event could be (perceived) or actually become a waste of time, money, and energy.

How do we resolve this ‘have to do it’ situation?. I’ve found it valuable to introduce the topic and build an understanding of the value of the exercise by including those most closely impacted directly in the design process. There will still be objections and sometimes a little less than full participation, however, this can usually be overcome by building in some fun, creating some energy around small tasks, being understanding about workload (the project lead – exercise designer) needs to be flexible, and of course, getting some management support in a positive way. We could also make the work part of routine expectations during the exercise cycle, that is, if employees, they can earn performance merits by excelling at this activity. Further , they need to be given time to do the task without feeling they are letting go of other work without impact.

It’s surprising how quickly people can come together, build a team with purpose, and rise to the challenge. Of course, don’t forget to offer appropriate praise and help along the way. Make sure each step of the process is a mini-celebration, and ensure the after action follow up encourages lessons learned and that it is rewarded.

The next time around, the answer to the question ‘why are we doing this’ will be ‘because we know how, it’s important, and we care about our organization and want to be ready!”.

When to Simulate and Not During an Exercise

One of the questions that often arises is the value of creating a full ‘simulation team’ versus using the ‘real’ people and functions during the exercise. There are pros and cons to each. In general, an exercise designer must consider how realistic a particular event needs to be, whether there is sufficient personnel, budget and other resources, and to what extent the business or organization can manage the impact of a practice scenario. Also, if there is an operational need for outside resources, will there be a cost and are they available.

In many cases, whether or not simulations are built into the scenario narrative depends on what portion of the organization or function are being exercised. To know that, we need to develop a suitable scope and set of objectives to be accomplished during a disaster exercise. Creating the scope and objectives is one of the essential steps to ensure a successful outcome.

Here are some general questions to consider that might help an exercise designer determine how much simulation to use, where to use it, and whether to simulate at all:

A comprehensive exercise program will already have evaluated its organization’s capabilities. Referring to and updating that assessment is an important step whenever a new exercise is considered for development.

The needs assessment will identify:

• Functions most requiring rehearsal.

• Potential exercise participants.

• Existing exercise requirements and capabilities.

• Plausible hazards and the priority levels of those hazards.

1.     Is the simulation part of our overall exercise program?

2.     Have we already established and evaluated our organization’s capabilities? If so, what is the purpose of the next event? We must incorporate plausible hazards and determine the feasibility of simulating outside participants versus not.

3.     Do we have the budget, time, and resources to impact operational elements of our organization? If so, then we do not need to simulate, and instead will use the ‘real’ resources.

4.     Do we want to test, evaluate, or drill on a particular resource? If so, then that resource should not be simulated and instead should be included in the scenario script as themselves.

5.     During Orientation and Table-top exercises, it may be too cumbersome to involve resources outside our organization or outside the primary function being evaluated. In such cases, it is often better to simulate. The simulation needs to be scripted with specific responses and exceptions so that some degree of realism can be evident.

6.     Do we have a simulation team that is available to act out and respond to communications by the participants? If so, the suggestions in item 5. above apply.

If these questions (there are many more) are not answered then we cannot be sure whether to include real or simulated events. In that case, the particular activity should either be eliminated from the exercise, or played down in importance. Playing down can be accomplished by writing specific directives and providing whatever data may be necessary to keep the exercise moving forward.

Obtaining Exercise Background Data

The background or preparatory data and information for continuity exercises may come from several sources. Of primary importance to the exercise designer should be a thorough understanding of the exercise objectives, scope, and participant functions. With these key elements defined, the designed can begin seeking out information necessary to begin crafting a suitable exercise plan.

The type of information to obtain includes

a.      communication channels, frequencies, equipment

b.      computer systems, data center capabilities

c.      confirmation from observers, controllers, evaluators

d.      confirmation of scope and purpose

e.      equipment and inventory list for event

f.       functions participating and their backup plans is exercise needs to cease in mid-event

g.      maps, GIS data, contact lists

h.     names of participants

i.       physical limitations, constraints, and test environment

j.       recovery point and recovery time objectives (if for information systems exercise)

k.      remote site and backup or devolution site and personnel

l.       traffic flow, rerouting, and parking

m.    vehicle inventory for event

n.     vital records

o.      and more

Typical information may sources include:

a.      Functional head or management of organizations within scope of exercise

b.      Facilities dept.

c.      Security dept.

d.      Public Works (if municipality)

e.      Utility Dept (if municipality)

f.       CIO or data center director

g.      Law enforcement, fire department, EMS liaison

h.     Local geographic, weather, crime, data

i.       and more

By bringing together the data and information early in the design process, the exercise designer can begin to identify gaps and take actions to resolve those gaps. Actions to resolve may include eliminating that portion of the exercise for which data will not be available, creating the data, or getting a commitment from the data source to provide the information when needed.

Bench Strength for Disaster Exercise Teams

A disaster exercise team is arguably the most value ingredient to preparing an organization to manage and recover from a disaster. There are other important ingredients, for example resources, technology, tools, vendors, executive support, communications, etc. However, a team of dedicated individuals who work well together and who understand how to implement actions in support of the disaster plan strategy can be the difference between an organization recovering with minimal life safety and business impact or not.
Some organizations may believe that the only way to be prepared is to create a team which acts like a ‘well-oiled machine’ able to respond to any event with competence and handle it successfully. The individuals on that team remain intact for long periods of time and become the ones relied on to ‘save’ the company in the event of a disaster. I do not support this approach. Rather, I believe we need to cultivate a sense of preparedness throughout an organization. There can and should still be a team leader, a team of trained and skills individuals with knowledge of various critical functions. But, in addition, the process of rotating other workers through the team can help build depth and strength which provides robustness. Robustness is needed especially during times of uncertain disaster or crisis for one primary reason: the team members themselves maybe impacted by the event and not available to help the organization respond and recover.

Conducting exercises with a more broad set of individuals affords us the opportunity to cross-train for redundancy of personnel, and get an additional set of eyes on how we perform. This additional input can help us learn from the exercise event and build a sense of the larger team, as in ‘we’re all in this together’ as opposed to one designated group of people are responsible to ‘save us’.

“Carrots and Sticks Don’t” Always Motivate

Emergency response exercises are a very valuable learning tool and can produce great energy and enthusiasm, despite the degree of successful completion or task accomplishment.

In a recent motivational talk by Dan Pink he shares the concept that businesses do not always follow established science findings related to how humans are motivated. Pink provides research that indicates people often perform to a higher degree of success without an organization imposing the twentieth century ‘carrots and sticks’ approach to management. This was found to be especially true where tasks and assignments required the application of cognitive skills beyond simple and narrowly defined tasks. In the latter case, simple defined tasks with narrow objectives can be achieved with higher performance using some amount of reward and discipline.

In my experience preparing and conducting disaster drills and similar exercises, a combination of guidelines and well known goals and objectives, a solidly built team, and well orchestrated event are critical to success. I have not found financial incentive nor punitive pressure to be of any value, particularly in the public sector setting. Here, employees are often motivated to do a good job and have a high level of pride in what they do for the community. Excelling during a disaster exercise and knowing that they can do the job well is incentive enough.

Of course, there are other very important steps in preparing for and conducting a successful exercise including clearly defined goals, selection of a skilled team of individuals, securing and use of resources, debriefing for lessons learned, and documentation of after action reports, to name a few.

I think establishing a sense of ownership and continuous improvement is the strategy that can produce the desired effect: the internal customer base (executives or leaders) and the external clients (big ‘C’, i.e. citizens or paying customers) feel confident in the organization and will continue to offer support and do business with the organization.

Those who have conducted exercises and drills know that such events, like the real thing, often don’t go as planned. The plan on paper often does not directly apply on first pass and needs updating and a different approach, mainly because every disaster presents a different set of inputs and circumstances, often unpredictable. Learning from our mistakes and learning from the variety of possible nuances of an event can help us be more ready for both the next drill and a real disaster.

Bottom line: cultivate a sense of ownership, awareness, pride, and motivation to be ready for a crisis or disaster.