X · Governance & ObservabilityEmerging★

Postmortem Pattern Mining

also known as Incident Corpus Mining, Retrospective Map-Fold

Mine a corpus of thousands of written postmortems through a staged model pipeline that summarises, classifies, analyses, and aggregates so that recurring incident causes surface as one short report.

Context

A mature engineering organisation accumulates years of incident postmortems, each a long free-text document written by whoever ran the response. The single most valuable thing in that archive is not any one document but the trend across all of them: which causes keep recurring, which mitigations keep failing, where the same class of outage returns under a new name. Reading the whole archive to extract that trend is a quarterly chore nobody finishes, so the corpus grows while the cross-document signal stays buried.

Problem

No reviewer can hold thousands of long, inconsistently-written postmortems in working memory at once, and the recurring pattern only becomes visible when the whole corpus is compared. Reading them serially is too slow to keep current, sampling a handful misses the long tail, and a single pass over the concatenated text overflows any context window and blurs distinct incidents into mush. The organisation is forced to choose between never extracting the cross-incident trend or paying for a manual read that is stale before it finishes.

Forces

  • The signal lives in the aggregate, but every model call can only see a small slice of the corpus at once.
  • Free-text postmortems are written to no fixed schema, so they must be normalised before they can be counted or compared.
  • A model summarising or classifying one document can fabricate a cause or miscategorise it, and a fabricated row corrupts the aggregate silently.
  • Reprocessing the full corpus on every run is expensive, yet skipping documents biases the trend toward whatever was processed.

Example

A platform team has six years of postmortems in a wiki. The map stage extracts a normalised record per document; a reviewer samples fifty of them and corrects two miscategorised causes; the classify stage snaps causes onto a fixed taxonomy; the analyse stage ranks them and finds that a quarter of all severe outages trace to the same unguarded config-reload path; the aggregate stage writes a one-page report naming that path as the top recurring cause, with links back to the nineteen postmortems behind the claim.

Diagram

Solution

Therefore:

Treat the archive as a map-fold problem. A per-document map stage sends each postmortem to a model that summarises it and emits a normalised record — cause category, affected component, trigger, mitigation, severity — against a fixed taxonomy. A classify stage snaps free-text causes onto that taxonomy so distinct documents become comparable rows. An analyse stage clusters the rows and ranks recurring causes by frequency, recency, and severity. A final aggregate stage drafts a one-page report of the dominant trends and patterns. Because a single hallucinated or miscategorised record poisons the count, a human reviewer samples the per-document records before the aggregate stage runs, and the report cites the underlying records so any claimed trend traces back to specific postmortems.

What this pattern forbids. The aggregate report may assert only trends that trace back to cited per-document records; a claim not backed by sampled, taxonomy-classified records is not allowed into the report.

The smaller patterns that complete this one —

  • usesMapReduce for Agents★— Split an oversize task into independent chunks, process each in parallel, then aggregate.

And the patterns that stand alongside it, or against it —

  • complementsDecision Log★★— Persist the agent's reasoning trace alongside its actions so post-hoc review can explain why.
  • complementsLineage Tracking★★— Track which prompt version, model version, and data sources produced each agent output.
  • conflicts-withAgent Confession as Forensics✕— Anti-pattern: after an agent-caused incident, the team treats the agent's confabulated self-narrative as the forensic record and root cause, even though the self-report is generated rather than remembered and can be flatly wrong.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance