Ultimate TTX

Which roles to put in the room

2026-09-15T00:00:00-06:00

The planning conversation reaches the participant list, and it is usually treated as an invitation problem. It is not. It is the decision that determines what the exercise is capable of finding, and it has a real cost on the other side.

What another role buys

An incident does not stay inside one team. It starts somewhere, is recognized somewhere else, and its consequences land in places that have nothing to do with computers.

A role that is missing from the room does not merely go unrepresented. Its part of the response gets assumed, and assumptions in an exercise are generous in a way that real people are not.

The first thing to get right is that "IT" is not a role. In most organizations of any size it is several groups with different tools, different on-call rotations and different managers, and the incident crosses all of them:

Network operations. Frequently the group that can actually stop lateral movement while it is happening, by taking segments apart. If containment is going to be tested at all, somebody who can do it has to be present.
Server administration, which owns the systems being restored and the real numbers about how long that takes.
Endpoint administration, which owns the machines where most incidents begin.
Security, and in larger organizations security operations as a separate function with its own tooling and its own queue.
The helpdesk, which is where most incidents are first visible, usually as something that does not look like an incident.
Identity administration, which holds the accounts and the authority to disable them.
Compliance, where it exists as its own group rather than as somebody's additional duty.

Then the parts of the organization that are not IT at all, and where the exercise usually finds its most durable material:

Management, which holds the decisions that cost money. A room without somebody who can commit the organization stalls at the first real choice.
Contracts. Worth separating two questions that sound like one. What data is on this system is usually answerable in the room: the project manager knows, and so does whoever uses the system daily, which is one of the better arguments for having an actual user present rather than only their manager. What reporting obligations attach to that data is a different question, and contracts is where it lives. That one stalls a room, and it is worth building into a scenario deliberately.
Legal, for the questions the contracts do not settle.
Communications. Somebody has to write what is said to customers, staff and regulators, and doing that badly is its own incident. Joint guidance from CISA, the FBI and four partner agencies recommends designating an incident lead, a communications lead and a spokesperson with predefined approval paths, and recommends exercising that structure through tabletop exercises specifically. Where a plan names those roles and no exercise ever tests them, the plan is untested at its most visible point.
Human resources, for anything involving a person rather than a system.
Finance, for emergency spending and for the question of what the outage is costing.
Physical security, wherever premises, media or access badges are involved.
The application owner, who is the only person who knows what a system is actually used for and therefore whether an outage is survivable.

If the organization uses a managed service provider, they should be in the room. For many small organizations the provider is the response capability, and an exercise that models their part rather than involving them is exercising a guess. Their contract is also the thing that determines how fast they arrive, which is worth discovering in a planning conversation rather than at two in the morning.

What a missing role actually costs, and one it does not

The value of a role is not uniform, and it is worth being precise rather than arguing that everyone should attend.

Take reporting obligations, which are often used to argue for legal counsel. For a defense contractor this is one of the clearer areas rather than one of the murkier ones. The clause is DFARS 252.204-7012, and it is unusually specific about all three. It defines what counts as a reportable cyber incident. It requires the contractor to rapidly report one to DoD, which paragraph (a) defines as within 72 hours of discovery. And it requires, separately, that images of the affected systems and the relevant monitoring data be preserved for at least 90 days from the report, so that the department can ask for the media or decline it. The reporting portal is dibnet.dod.mil and can be visited in advance, so an organization can know before any incident exactly what it will be asked for.

That makes the obligation a preparation question rather than a legal one, and the exercise finding is usually not "we needed a lawyer" but "nobody had read it, and nobody had looked at the form." Both are fixable in an afternoon by somebody who is already on staff.

A related observation belongs in the report when it applies: an organization that has been in the defense industrial base for years and has never filed a report is not describing an unusually quiet decade. Laptops get stolen. People click things. A clean history of that length more often means the reporting threshold was never understood than that nothing ever crossed it.

Contracts is the opposite case. "Which agreements cover the data on this system" is not knowable from the IR plan, not knowable by IT, and not knowable at half past five in the evening. That absence changes what the room can do.

What another role costs

Half a working day, at least, once the pre-exercise briefing is counted. That is the honest number, and it is per person.

For a small organization this is not a rounding error. Six people out for half a day is a meaningful piece of a week's capacity, and the people whose presence would help most are frequently the hardest to release, because they are the same people holding operations together while the exercise runs.

What that cost is not, in practice, is travel. Organizations do not fly people in for a tabletop exercise, and they are right not to: with travel either side it becomes a three-day commitment plus expenses, for half a day of participation. Where somebody is not local, having them attend remotely is nearly as good and costs a fraction as much. A remote participant is a slightly diminished participant rather than an absent one, and that trade is almost always better than dropping the role.

There is a second cost that is easier to miss. A room that is too large stops being a discussion. Participants with no part in the current inject disengage, and a disengaged participant is not merely idle. They are reading email, and the injects continue while they do. When the discussion reaches something they could have contributed to—and it does, because that is why they were invited—they have missed the moment. The exercise then records a gap that was not really a gap, or misses one that was.

A room that is too large also changes character: the people still engaged start performing for an audience rather than working the problem.

So more roles is not monotonically better, even ignoring the money.

Deciding rather than inviting

The useful reframing is that the scenario determines the roster, not the org chart. This is not a principle to be taken on trust—it is visible in any scenario library that has been built deliberately, and it is worth looking at before planning a particular exercise.

The scenario library Ultimate TTX builds for subscribers shows the pattern plainly. Across those scenarios three roles appear almost everywhere:

A ransomware with exfiltration scenario reaches ten roles, including the helpdesk, backup administration and endpoint administration, because the response spans first contact, containment, restoration and disclosure.
A business email compromise scenario reaches five, and finance is central to all of them. Backup administration has no part in it at all.
An insider data theft scenario brings in human resources and drops IT operations.
A physical access and removable media scenario needs physical security, which appears nowhere in the financial or identity scenarios.
A service provider compromise scenario needs network operations and the application owner, and much of it is about somebody else's environment.

Build one standing invitation list and reuse it, and some exercises carry passengers while others are missing the person who mattered. Read the scenario first; it will tell you who has to be there.

Two practical devices help.

Name the role, not the person. The question is which functions have to be represented; who fills each one is a scheduling problem after that. It also makes substitution visible, and substitution matters: a deputy who does not hold the authority will answer as though they do.

Absence is a finding. When a role cannot be released, the exercise still runs, and the report records that the response was modeled without it. That is more useful than quietly proceeding, because "we could not free the only person who can authorize an emergency shutdown" deserves management's attention, and it recurs during real incidents for exactly the same reason.

One role is not on the scenario's list, and it belongs in every room: whoever would coordinate the response. The scenario decides which functions the incident touches; coordination is not one of those functions, it is the thing that sequences the rest of them. Sending a deputy is worse here than anywhere else on the roster, because the exercise cannot then observe whether the arrangement works—it observes a stand-in improvising one. Why that role matters, and what it has to not be doing, is a subject of its own.

Communications is the other role worth arguing for beyond first principles. Where a scenario requires the organization to say anything to anyone outside the response team, the function that says it should be present rather than represented by whoever is nearest—and the joint guidance Communicating Under Pressure, published by six agencies in September 2026, recommends exercising communication plans through "simulated tabletop exercises" and naming communications teams specifically.

The roster shapes the report, not just the day

Choosing the room is also choosing what the report can say, and it is worth thinking about at planning time rather than discovering afterward.

A finding about a role that was not represented is weaker, and it should be: nobody from that function was there to explain what actually happens. The report has to say so, and a reader is entitled to discount it. Conversely, a finding that rests on two functions contradicting each other in the room—the backup administrator and the person who would be woken up, say, or IT and contracts—is among the strongest material an exercise produces, and it is only available if both were present.

The recommendations divide along the same lines. A finding whose fix is political, requiring somebody senior to settle a disagreement between two groups, lands very differently depending on whether both groups were in the room and heard it emerge.

This is also the strongest argument for management attending rather than being briefed afterward. The recommendations needing an executive decision are exactly the ones that arrive in a report as a surprise, and an executive who watched the disagreement happen does not have to be persuaded that it is real.

The failure mode worth avoiding

The version that wastes everyone's time is the room built from whoever was available, running a scenario chosen independently of them. It produces a pleasant discussion, a report with thin findings, and a general impression that tabletop exercises are a compliance ritual.

Who is in the room is part of the scenario design. Choosing deliberately is most of what separates an exercise that finds something from one that fills a requirement.

Ultimate TTX plans facilitated exercises from the scenario outward, including which roles have to be in the room for it to be worth running.

"How long would this take you?"

2026-09-08T00:00:00-06:00

It is a regular question in an exercise, and it is worth asking often. Restoring that server: how long? Finding which accounts the attacker used: how long? Getting the replacement hardware: how long?

The answers are optimiztic. Not occasionally, and not only from the overconfident. They are optimiztic in the same direction, from careful people, about work they have done before, and sometimes they are wildly optimiztic.

The reason is not carelessness. Everybody estimates on the assumption that things go well, and an incident is by definition the situation in which things are not going well. Skill offers no protection here and may work against it: an experienced administrator's estimate is drawn from a real memory of doing the work competently. During the incident those conditions are gone, and what replaces them is worse than merely different: the same work, under time pressure, in front of an audience, with consequences attached.

The estimate is the task; the incident is the conditions

The number is usually the work itself, measured on a good day, by someone who has done it. What it leaves out is everything the incident puts around the work.

Start with the room. During an incident the office fills with people who need answers: managers asking what is happening, end users asking when their systems will be back, somebody stating—or shouting—what the downtime is costing per hour. A colleague, part-way through an incident, had to tell his own boss and several others to leave the area so that he and his team could concentrate. That is a reasonable thing to have to do and a difficult thing to do, and the time it takes to reach the point of doing it is not in anybody's estimate.

It is also, as it turns out, the recommended practice. Joint guidance from CISA, the FBI and the Australian, Canadian, New Zealand and UK cyber centers tells organizations to run communications through designated liaisons, scheduled syncs and a single intake path, explicitly in order to shield engineers from external interruptions. Nobody should have to improvise that at the moment it is needed, and an organization that has not arranged it in advance is relying on somebody being willing to throw their own manager out of the room.

The interruptions themselves cost more than people expect, and this has been measured. In a field study of ordinary computer work, Iqbal and Horvitz logged what happened after email and instant messaging alerts. Where somebody answered an email alert immediately, an average of sixteen and a half minutes passed before they were working again in the same program and document they had left, against an overall rate of close to four alerts an hour. In 27% of cases they had still not returned to it two hours later.

The authors are careful about what that measures, and it is worth repeating: the clock stops when the person is back in the software they were using, which is not the same as being back in the task. Getting the window in front of you again is the first step of resuming work, not the completion of it. The real cost of the interruption is therefore larger than the figure, not smaller.

There is a mechanism behind that number, and it makes the incident case worse rather than better. Leroy's work on attention residue describes what persists: cognitions about the first task that continue after somebody has stopped working on it, switched to a second, and is now working on the second. Those lingering thoughts are a cognitive load, and the resources they consume are not available to the task actually in front of the person.

What decides how much residue there is turns out to be whether the first task was finished. Across two experiments, people who were stopped mid-task carried the unfinished one with them and did measurably worse on what came next; people who finished first did better.

That is the incident case exactly. Interruptions during an incident almost never arrive at a clean stopping point—the interruption is what makes the task unfinished, and the person carries the restore, or the log search, or the half-formed theory about what happened, into the conversation with the director and back out again.

Leroy also found a way out that an incident cannot offer. Finishing the first task under time pressure produced less residue than finishing it at leisure, because working against the clock narrows what somebody considers, which leaves them more confident it is done and better able to stop thinking about it. Time pressure helps only in combination with completion. During an incident there is plenty of the first and very little of the second.

So the measured figures should be read as a floor rather than a model. Ordinary office work, interrupted at ordinary moments, is the benign case.

One task, several bottlenecks

Interruption stretches whatever the work turns out to be. The next problem is that the work itself is larger than the person answering has in mind—and the two compound rather than compete. A four-hour job that is really a nine-hour job, done in a day full of interruptions, is not thirteen hours; it is nine hours of work delivered across a stretch of calendar nobody predicted, with the resumption cost paid at every one of the boundaries.

So the second reason technical estimates come in short: the person answering has one step in mind, and the work has several, each with its own limit.

Restoring from backup is the clearest case. The details depend entirely on the backup method, but the candidates for the constraint include:

Read speed from the device holding the backup, which is a property of both that device and the backup server's own I/O system.
The backup server itself, which has to do the work of serving the restore.
Network bandwidth between the backup server and the system being restored.
Write speed on the system being restored, which is frequently the limit nobody thought about, because the estimate was framed as a question about backups.

Whichever of those is slowest sets the time, and the person giving the estimate is rarely thinking about all four.

Off-site backup copies complicate this in a way worth separating out. Holding a copy somewhere else is good practice and the right answer for disaster recovery, since a backup in the same building as the original is not a backup against several of the things most likely to destroy both. It also frequently means the restore path runs over a much narrower link than the local one. An organization can be entirely correct to keep the off-site copy and still be badly wrong about how long restoring from it takes, and the estimate given in the room is almost always the local number.

Few organizations have ever calculated their real maximum, and it is never the number on the manufacturer's data sheet. That figure describes ideal conditions no production system reproduces.

It can be done, though, and the exercise finds the people who have done it. In a recent exercise the head of the server team was a participant. Asked about restore time, he said plainly that he could not speak for the network bandwidth between the systems—and then said that he had calculated the I/O bandwidth himself when specifying the backup server. That is exactly the right answer: a real number for the part he owns, and an explicit boundary where his knowledge stops. It is also uncommon.

Then there is a compounding effect, in the class of incident where it applies. Where the response requires restoring many systems at once—ransomware and destructive malware are the obvious cases—simultaneous restores contend for the same shared resources: the backup server, the storage behind it, the network between them. Depending on the hardware and the topology, that contention can be considerably worse than proportional, and an estimate derived from restoring one system will not survive it.

Hardware that no longer exists

"How long to get the replacement?" deserves asking on its own, because the answer has moved.

The first question is whether cold spares exist. If they do, the estimate is about installation. If they do not, the estimate is about somebody else's supply chain, which is a different kind of number entirely: one the organization does not control and probably has not checked recently.

It has been a bad few years to be checking late. The build-out of AI data centers has pulled memory and other components toward buyers who order them by the container, and lead times on ordinary enterprise hardware have stretched accordingly. An organization planning a routine server refresh now queues behind hyperscalers for the same parts.

The second question is subtler, and it catches the people who did check. Hardware model lifespans are short, often measured in months and rarely much beyond a year. The exact model in the rack may simply not be purchasable, whatever the lead time. The replacement task is then not procurement but engineering: working out what will function correctly in this situation, with this software, in this rack, on this power budget—and doing it under incident conditions, which is the worst circumstance in which to make a design decision.

The estimate also assumes skill that may not be there

Searching logs is a good second example, because the constraint is not only hardware.

Suppose the organization has a SIEM and the logs are in it. Somebody is asked to establish the initial entry vector. That is not a narrow lookup: it means searching across gigabytes, often considerably more, over a period nobody has bounded yet.

Two things then decide the answer, and neither is in anyone's estimate. The first is how well the SIEM indexes the data being searched, which varies enormously between products—and the ones that hold up are expensive and need capable hardware underneath them, so this is a purchasing decision made years earlier that presents its bill during an incident.

The second is whether the person searching knows the query that returns what is needed and nothing else. If they do, it is minutes. If they do not, it is a broad query, thousands of returned entries, and a person reading them, which is not the same task at all. The tooling is identical in both cases. The difference is whether a particular skill happens to be present at that moment, on that shift.

Dependencies on external people or organizations

Any step that depends on somebody outside the team carries a hidden assumption: that they will respond immediately, and that they will be competent when they do.

Where the dependency is a service provider, the assumption is usually contradicted by the contract. A managed service provider has an agreed maximum response time, and it is entirely proper for them to use it—a response inside the committed window has met the agreement. It is also unlikely that the first response contains everything needed, so the real elapsed time is several exchanges long, each governed by the same clock. Internal groups frequently have service levels to other internal groups, with the same effect and less visibility.

Then there is the question of who is even at work. A recent exercise reached the question of which contracts governed the data held on a particular system. That is not an IT question, and the people who can answer it keep office hours.

How much that costs varies more than any other delay in this article, and the range is worth stating rather than assuming the worst. Some organizations can raise somebody in Contracts at ten at night—a small company where everyone has everyone's mobile number, or one that has thought about which non-IT roles an incident needs and put them on a call list. Others cannot, and the answer waits for the morning. The figure offered in that room was an hour, which assumed reaching the right person at once and getting a complete answer first time; the useful correction is not a bigger number but the two questions underneath it.

Can this person be reached out of hours at all, and has anyone checked? A name on a plan is not a phone number that gets answered, and the reasons it might not be answered have changed in a way most call trees have not caught up with.

Phones now silence themselves. A sleep schedule or a focus mode turns the ringer off every night automatically, and the exception list is a handful of close contacts—family, not the on-call rotation. Adding somebody to that list is a deliberate act that nobody does for a colleague they might need once. And even where the exception exists, the phone is frequently charging in another room, which defeats it completely.

None of that is carelessness. It is the default configuration of an ordinary phone belonging to somebody with a healthy relationship to their work. The consequence is that "we can call him" needs testing rather than assuming: has anyone actually rung that number at eleven at night and seen what happens? An organization that has tested it knows whether it has an out-of-hours path. One that has not is holding a list of numbers and a hope. And is the first answer likely to be the complete one? A contracts question rarely is: the document has to be found, the relevant clause read, and its application to this specific system decided, which is frequently a second conversation with somebody else. Even the organization that can reach its Contracts lead at ten at night is usually looking at more than an hour before it has an answer it would act on.

The number that describes work nobody has done

Some estimates describe a capability demonstrated only under conditions unlike the ones that will apply.

Kenneth spoke with someone able to image a remote system's storage and capture a memory image remotely. A genuinely useful capability, and the person is competent to do it. He has not, however, done it against a remote system: the timing he would quote comes from doing it across a local network.

That is not an idle distinction. The whole point of the capability is reaching something that is not on the local network, and the link is exactly the variable that has never been exercised. The estimate is honest, from a skilled person, about work they can genuinely do—and it describes a different job from the one that will be asked of them.

The question is a probe, not a measurement

The estimate is not really the point. What it is made of is the point, and the follow-up questions are where the exercise earns its day:

Has anyone here done this at this size?
When was the last time?
What has changed since?
Which part of it is the slow part, and how do you know?
Who else has to be available for it, and what happens if they are not?

What the exercise cannot do is produce the true number. There is no time in an exercise to run a restore or a search and measure it, and a room that spends its afternoon trying has stopped doing the exercise. That is a real limit and it should be said plainly rather than glossed over.

What the exercise can establish is whether a defensible number exists anywhere. A room that can answer the questions above has one, or knows where it is written down. A room that cannot has discovered something more useful than a number: that its recovery planning rests on a figure nobody has tested, which is a finding, and one nobody had to run a stopwatch to reach.

The second useful move is asking two people separately. The backup administrator and the person who would actually be woken up frequently give different answers, and the difference is rarely a disagreement about technology. It is usually that one of them is including the conditions and the other is not.

The after-action item writes itself

This is one of the rare exercise findings with a clean, cheap recommendation attached, and the recommendation is not "revise the estimate". It is do the thing and time it.

Pick the steps the plan depends on, carry them out under conditions as close to real as can be arranged, and record what they actually took. The timing is only half the return. The other half is confirming that the expected data is really there: that the backup contains what everyone believes it contains, that the logs retain the period that would need to be searched, that the remote capture works against something remote.

Where an IR plan states time estimates for its steps—and plans increasingly do—those numbers need real tests behind them, and the conditions of the test need recording somewhere alongside the estimate. An unsupported number in a plan is worse than no number, because it will be relied on by people who assume somebody measured it. "Forty minutes" means something quite different depending on whether it was one system or twelve, a local network or a remote one, a practiced operator or whoever was on shift.

A note on how to ask

Ask for the number before discussing the difficulties, not after. Once the room has spent five minutes on everything that could go wrong, the estimate that follows is contaminated by the discussion, and the interesting artifact—the number people carry around in their heads and plan with—is gone.

Ask, write it down, then open the discussion. The first answer is the one the organization has actually been relying on, and the distance between it and the second answer is the finding.

Ultimate TTX runs facilitated exercises built to surface exactly this kind of gap, on the day, in front of the people who can act on it.