"How long would this take you?"
It is a regular question in an exercise, and it is worth asking often. Restoring that server: how long? Finding which accounts the attacker used: how long? Getting the replacement hardware: how long?
The answers are optimiztic. Not occasionally, and not only from the overconfident. They are optimiztic in the same direction, from careful people, about work they have done before, and sometimes they are wildly optimiztic.
The reason is not carelessness. Everybody estimates on the assumption that things go well, and an incident is by definition the situation in which things are not going well. Skill offers no protection here and may work against it: an experienced administrator's estimate is drawn from a real memory of doing the work competently. During the incident those conditions are gone, and what replaces them is worse than merely different: the same work, under time pressure, in front of an audience, with consequences attached.
The estimate is the task; the incident is the conditions
The number is usually the work itself, measured on a good day, by someone who has done it. What it leaves out is everything the incident puts around the work.
Start with the room. During an incident the office fills with people who need answers: managers asking what is happening, end users asking when their systems will be back, somebody stating—or shouting—what the downtime is costing per hour. A colleague, part-way through an incident, had to tell his own boss and several others to leave the area so that he and his team could concentrate. That is a reasonable thing to have to do and a difficult thing to do, and the time it takes to reach the point of doing it is not in anybody's estimate.
It is also, as it turns out, the recommended practice. Joint guidance from CISA, the FBI and the Australian, Canadian, New Zealand and UK cyber centers tells organizations to run communications through designated liaisons, scheduled syncs and a single intake path, explicitly in order to shield engineers from external interruptions. Nobody should have to improvise that at the moment it is needed, and an organization that has not arranged it in advance is relying on somebody being willing to throw their own manager out of the room.
The interruptions themselves cost more than people expect, and this has been measured. In a field study of ordinary computer work, Iqbal and Horvitz logged what happened after email and instant messaging alerts. Where somebody answered an email alert immediately, an average of sixteen and a half minutes passed before they were working again in the same program and document they had left, against an overall rate of close to four alerts an hour. In 27% of cases they had still not returned to it two hours later.
The authors are careful about what that measures, and it is worth repeating: the clock stops when the person is back in the software they were using, which is not the same as being back in the task. Getting the window in front of you again is the first step of resuming work, not the completion of it. The real cost of the interruption is therefore larger than the figure, not smaller.
There is a mechanism behind that number, and it makes the incident case worse rather than better. Leroy's work on attention residue describes what persists: cognitions about the first task that continue after somebody has stopped working on it, switched to a second, and is now working on the second. Those lingering thoughts are a cognitive load, and the resources they consume are not available to the task actually in front of the person.
What decides how much residue there is turns out to be whether the first task was finished. Across two experiments, people who were stopped mid-task carried the unfinished one with them and did measurably worse on what came next; people who finished first did better.
That is the incident case exactly. Interruptions during an incident almost never arrive at a clean stopping point—the interruption is what makes the task unfinished, and the person carries the restore, or the log search, or the half-formed theory about what happened, into the conversation with the director and back out again.
Leroy also found a way out that an incident cannot offer. Finishing the first task under time pressure produced less residue than finishing it at leisure, because working against the clock narrows what somebody considers, which leaves them more confident it is done and better able to stop thinking about it. Time pressure helps only in combination with completion. During an incident there is plenty of the first and very little of the second.
So the measured figures should be read as a floor rather than a model. Ordinary office work, interrupted at ordinary moments, is the benign case.
One task, several bottlenecks
Interruption stretches whatever the work turns out to be. The next problem is that the work itself is larger than the person answering has in mind—and the two compound rather than compete. A four-hour job that is really a nine-hour job, done in a day full of interruptions, is not thirteen hours; it is nine hours of work delivered across a stretch of calendar nobody predicted, with the resumption cost paid at every one of the boundaries.
So the second reason technical estimates come in short: the person answering has one step in mind, and the work has several, each with its own limit.
Restoring from backup is the clearest case. The details depend entirely on the backup method, but the candidates for the constraint include:
- Read speed from the device holding the backup, which is a property of both that device and the backup server's own I/O system.
- The backup server itself, which has to do the work of serving the restore.
- Network bandwidth between the backup server and the system being restored.
- Write speed on the system being restored, which is frequently the limit nobody thought about, because the estimate was framed as a question about backups.
Whichever of those is slowest sets the time, and the person giving the estimate is rarely thinking about all four.
Off-site backup copies complicate this in a way worth separating out. Holding a copy somewhere else is good practice and the right answer for disaster recovery, since a backup in the same building as the original is not a backup against several of the things most likely to destroy both. It also frequently means the restore path runs over a much narrower link than the local one. An organization can be entirely correct to keep the off-site copy and still be badly wrong about how long restoring from it takes, and the estimate given in the room is almost always the local number.
Few organizations have ever calculated their real maximum, and it is never the number on the manufacturer's data sheet. That figure describes ideal conditions no production system reproduces.
It can be done, though, and the exercise finds the people who have done it. In a recent exercise the head of the server team was a participant. Asked about restore time, he said plainly that he could not speak for the network bandwidth between the systems—and then said that he had calculated the I/O bandwidth himself when specifying the backup server. That is exactly the right answer: a real number for the part he owns, and an explicit boundary where his knowledge stops. It is also uncommon.
Then there is a compounding effect, in the class of incident where it applies. Where the response requires restoring many systems at once—ransomware and destructive malware are the obvious cases—simultaneous restores contend for the same shared resources: the backup server, the storage behind it, the network between them. Depending on the hardware and the topology, that contention can be considerably worse than proportional, and an estimate derived from restoring one system will not survive it.
Hardware that no longer exists
"How long to get the replacement?" deserves asking on its own, because the answer has moved.
The first question is whether cold spares exist. If they do, the estimate is about installation. If they do not, the estimate is about somebody else's supply chain, which is a different kind of number entirely: one the organization does not control and probably has not checked recently.
It has been a bad few years to be checking late. The build-out of AI data centers has pulled memory and other components toward buyers who order them by the container, and lead times on ordinary enterprise hardware have stretched accordingly. An organization planning a routine server refresh now queues behind hyperscalers for the same parts.
The second question is subtler, and it catches the people who did check. Hardware model lifespans are short, often measured in months and rarely much beyond a year. The exact model in the rack may simply not be purchasable, whatever the lead time. The replacement task is then not procurement but engineering: working out what will function correctly in this situation, with this software, in this rack, on this power budget—and doing it under incident conditions, which is the worst circumstance in which to make a design decision.
The estimate also assumes skill that may not be there
Searching logs is a good second example, because the constraint is not only hardware.
Suppose the organization has a SIEM and the logs are in it. Somebody is asked to establish the initial entry vector. That is not a narrow lookup: it means searching across gigabytes, often considerably more, over a period nobody has bounded yet.
Two things then decide the answer, and neither is in anyone's estimate. The first is how well the SIEM indexes the data being searched, which varies enormously between products—and the ones that hold up are expensive and need capable hardware underneath them, so this is a purchasing decision made years earlier that presents its bill during an incident.
The second is whether the person searching knows the query that returns what is needed and nothing else. If they do, it is minutes. If they do not, it is a broad query, thousands of returned entries, and a person reading them, which is not the same task at all. The tooling is identical in both cases. The difference is whether a particular skill happens to be present at that moment, on that shift.
Dependencies on external people or organizations
Any step that depends on somebody outside the team carries a hidden assumption: that they will respond immediately, and that they will be competent when they do.
Where the dependency is a service provider, the assumption is usually contradicted by the contract. A managed service provider has an agreed maximum response time, and it is entirely proper for them to use it—a response inside the committed window has met the agreement. It is also unlikely that the first response contains everything needed, so the real elapsed time is several exchanges long, each governed by the same clock. Internal groups frequently have service levels to other internal groups, with the same effect and less visibility.
Then there is the question of who is even at work. A recent exercise reached the question of which contracts governed the data held on a particular system. That is not an IT question, and the people who can answer it keep office hours.
How much that costs varies more than any other delay in this article, and the range is worth stating rather than assuming the worst. Some organizations can raise somebody in Contracts at ten at night—a small company where everyone has everyone's mobile number, or one that has thought about which non-IT roles an incident needs and put them on a call list. Others cannot, and the answer waits for the morning. The figure offered in that room was an hour, which assumed reaching the right person at once and getting a complete answer first time; the useful correction is not a bigger number but the two questions underneath it.
Can this person be reached out of hours at all, and has anyone checked? A name on a plan is not a phone number that gets answered, and the reasons it might not be answered have changed in a way most call trees have not caught up with.
Phones now silence themselves. A sleep schedule or a focus mode turns the ringer off every night automatically, and the exception list is a handful of close contacts—family, not the on-call rotation. Adding somebody to that list is a deliberate act that nobody does for a colleague they might need once. And even where the exception exists, the phone is frequently charging in another room, which defeats it completely.
None of that is carelessness. It is the default configuration of an ordinary phone belonging to somebody with a healthy relationship to their work. The consequence is that "we can call him" needs testing rather than assuming: has anyone actually rung that number at eleven at night and seen what happens? An organization that has tested it knows whether it has an out-of-hours path. One that has not is holding a list of numbers and a hope. And is the first answer likely to be the complete one? A contracts question rarely is: the document has to be found, the relevant clause read, and its application to this specific system decided, which is frequently a second conversation with somebody else. Even the organization that can reach its Contracts lead at ten at night is usually looking at more than an hour before it has an answer it would act on.
The number that describes work nobody has done
Some estimates describe a capability demonstrated only under conditions unlike the ones that will apply.
Kenneth spoke with someone able to image a remote system's storage and capture a memory image remotely. A genuinely useful capability, and the person is competent to do it. He has not, however, done it against a remote system: the timing he would quote comes from doing it across a local network.
That is not an idle distinction. The whole point of the capability is reaching something that is not on the local network, and the link is exactly the variable that has never been exercised. The estimate is honest, from a skilled person, about work they can genuinely do—and it describes a different job from the one that will be asked of them.
The question is a probe, not a measurement
The estimate is not really the point. What it is made of is the point, and the follow-up questions are where the exercise earns its day:
- Has anyone here done this at this size?
- When was the last time?
- What has changed since?
- Which part of it is the slow part, and how do you know?
- Who else has to be available for it, and what happens if they are not?
What the exercise cannot do is produce the true number. There is no time in an exercise to run a restore or a search and measure it, and a room that spends its afternoon trying has stopped doing the exercise. That is a real limit and it should be said plainly rather than glossed over.
What the exercise can establish is whether a defensible number exists anywhere. A room that can answer the questions above has one, or knows where it is written down. A room that cannot has discovered something more useful than a number: that its recovery planning rests on a figure nobody has tested, which is a finding, and one nobody had to run a stopwatch to reach.
The second useful move is asking two people separately. The backup administrator and the person who would actually be woken up frequently give different answers, and the difference is rarely a disagreement about technology. It is usually that one of them is including the conditions and the other is not.
The after-action item writes itself
This is one of the rare exercise findings with a clean, cheap recommendation attached, and the recommendation is not "revise the estimate". It is do the thing and time it.
Pick the steps the plan depends on, carry them out under conditions as close to real as can be arranged, and record what they actually took. The timing is only half the return. The other half is confirming that the expected data is really there: that the backup contains what everyone believes it contains, that the logs retain the period that would need to be searched, that the remote capture works against something remote.
Where an IR plan states time estimates for its steps—and plans increasingly do—those numbers need real tests behind them, and the conditions of the test need recording somewhere alongside the estimate. An unsupported number in a plan is worse than no number, because it will be relied on by people who assume somebody measured it. "Forty minutes" means something quite different depending on whether it was one system or twelve, a local network or a remote one, a practiced operator or whoever was on shift.
A note on how to ask
Ask for the number before discussing the difficulties, not after. Once the room has spent five minutes on everything that could go wrong, the estimate that follows is contaminated by the discussion, and the interesting artifact—the number people carry around in their heads and plan with—is gone.
Ask, write it down, then open the discussion. The first answer is the one the organization has actually been relying on, and the distance between it and the second answer is the finding.
Ultimate TTX runs facilitated exercises built to surface exactly this kind of gap, on the day, in front of the people who can act on it.