Ranking what you recommend
Turning observations into recommendations is one job. Putting them in an order the reader can act on is a second one, and it is where a report either earns its next quarter's attention or spends it.
The obvious arrangement is a single list in severity order. It is what the first version of a report tends to be, and it does not survive the second.
A recommendation is attached to a risk
Every recommendation exists because something could go wrong. Saying what would close the gap without saying what the gap exposes the organization to leaves the reader with a shopping list and no way to rank it.
Naming the risk also changes what happens when the recommendation is declined, and this is the part worth being explicit about. A recommendation that is not acted on is a risk the organization has accepted. That may be entirely reasonable: the risk may be small, or the fix may cost more than the exposure justifies. What makes it reasonable is that somebody decided it, on stated grounds, and the decision exists somewhere it can be revisited.
What happens instead, when the risk is not named, is that the same decision gets made silently. Nobody weighed anything. The recommendation was simply not convenient, and the acceptance is invisible until the thing it covered happens. An explicit acceptance and an implicit one leave the organization in the same technical position and in very different positions in every other respect.
This is the point at which an exercise report meets the organization's risk assessment, which is where risks are supposed to be enumerated and rated in the first place. A finding that maps onto a risk already on that register is easy to act on. A finding with no counterpart there is telling the organization something about its register as well as about its incident response.
Grouping beats ordering, and documentation is its own group
The reason is a fact about what exercises actually produce. Working through a recent report, most of the recommendations turned out to be documentation defects: a point missing from a procedure, a step in the wrong place, an instruction that no longer matches the environment. Individually small, cheap to fix, and—the part that matters—approved at a level far below the person the report is addressed to. Somebody owns that runbook and can change it this week.
In a single severity-ordered list those get scattered among the expensive structural recommendations, and they inherit the reader's sense of what acting on the document costs. A fix that needed one person and an afternoon waits for a program that needs a budget cycle.
So the recommendations are grouped by what acting on them involves, and severity orders within a group rather than across them:
- High risks. The response departed from its own plan, or the plan itself is unsafe. These need a decision, and often a change in how the organization behaves under pressure rather than a change to a document.
- Documentation. Ranked high, moderate and low inside the group. One kind of work, one approval level, and a section that can be handed to the person who owns the documents.
- Exercise updates. What the next exercise has to do differently, given what this one could and could not reach.
- Testing and timing. The values the organization has to go and measure—restore rates, call-out times, how long a step actually takes—and record as it revises its procedures.
The documentation group is not the low-severity group, and saying so in the report is what makes the arrangement work rather than looking like a demotion. Establishing the date of the initial compromise before starting to restore from backups is a documentation change: a step in the wrong order in an existing procedure. It is also the difference between restoring the business and restoring the intrusion, because a team that has not dated the compromise cannot know which backup is clean. It is a documentation recommendation and it is a serious one, and a flat list ordered by severity would have separated it from every other fix that lands on the same person's desk.
Severity still has to be rated, and the rating has a known weakness
Severity comes from risk, and rating risk from outside the organization has a structural weakness worth stating plainly. The facilitator rates what the exercise revealed, and the exercise reveals a fraction of what is true.
The mechanism is not that anybody withheld anything. A mitigation exists, and it simply never comes up: the room was working a network problem when the finding is about endpoints, so the endpoint control nobody mentioned was never relevant to the discussion they were having. The facilitator cannot ask every question that would surface every control—there is not time, and asking them would be leading the room, which produces a worse exercise than a slightly wrong severity rating does. The participants are the ones who know which mitigations they hold and when each applies, and that knowledge only surfaces if the exercise happens to walk past it.
So a rating can be wrong, and the organization is the only party able to correct it. That is worth saying in the report rather than leaving the reader to discover it. The cleanest correction would be the organization's own risk assessment, read alongside the findings—though asking for it does not always work, since not every company will share that with an outside consultant, and some do not share it with their own staff.
Alongside the severity, it is worth recording the kind of change that would mitigate the finding. The grouping above is coarse, and inside a group the kind is what tells the reader what they are being asked for. Two findings of equal severity can be a morning's configuration work and an eighteen-month argument about how the company operates, and a reader planning next quarter needs to know which is which. Where more than one route exists, naming them beats picking one, since the organization knows its own constraints better than the facilitator does.
A handful of kinds recur, and each fails in its own way:
- Tooling. Something needs buying, configuring or turning on. This is the easiest kind to write and the easiest to over-recommend, because a product is a satisfying answer to a problem that may not be product-shaped. It is worth being suspicious of a tooling recommendation that would paper over something structural. And tools are not free once bought: they frequently do not talk to each other, they can be genuinely unpleasant to use, and a bad tool in the middle of a response makes the response slower rather than faster. Which tool is right depends on the organization, and the facilitator usually does not know enough to say.
- Training. Frequently the honest reading of a technical-looking failure, and it splits into two recommendations that cost very different amounts. In the first, the capability exists in the organization and not in the person who needed it at the time—somebody knows how to do this and was not the one on shift. That is scheduling, documentation and a training plan. In the second, nobody in the organization has the knowledge, which is not a training gap but a capability gap wearing its clothes: it closes by acquiring the skill deliberately, or by hiring, or by contracting for it in advance rather than discovering the absence mid-incident. Saying which of the two it is matters, because a recommendation to "train the team" aimed at the second one cannot succeed.
- Process and documentation. Rarely just "nothing is written down". More often the procedure exists and is wrong in a specific way: it specifies actions that no longer fit because the environment changed underneath it; a step is in the wrong place and needs to happen earlier or later than it says; or it was written by an expert for an expert, and the person who actually has to follow it at two in the morning is the most junior member of the team. Each of those is a different repair.
- Contractual. The gap is at a provider boundary and closes through an agreement rather than through anything the organization can configure.
- Political. Somebody senior has to settle it and absorb the friction. Not always a disagreement between groups: sometimes what the finding actually asks for is a change in how the company behaves, which is a culture change wearing a technical costume, and no amount of tooling or training substitutes for it.
That last kind deserves care. A recommendation that needs an executive decision may be beyond what the person holding the report can settle alone, and writing it as though it were theirs to action is how it ends up sitting there. Naming who has to decide makes it actionable—but an external facilitator often cannot, because who really holds that decision is internal knowledge that a day in the room does not reveal. Where it is not known, the honest form is to state that resolving the finding requires a decision above the level of the people who were present, and let the organization supply the name.
Where this sits
Every recommendation here started as an observation, and getting from one to the other is the subject of its own article: what it means to account for every observation, and what to do with the ones that resist becoming anything.
Ultimate TTX runs facilitated exercises where the report is ordered for the person who has to act on it.