AI Matrx Logo

Incident Postmortems

Brightlane Pay · Reliability · Denver, CO

Sigrid Halvorsen heads reliability at a payments company and uses this to run postmortems the same way every time and chase their action items.

Any businessComplianceBoards3 tables · 14 views · 1 form · 1 row action · 1 agent
Use this template

By status

Action items
Open3
Test replica promotion every week in stagingDatabase failover did not promote a replicaPrevent
Alert fourteen days before any certificate expiresLogin errors after a certificate renewalDetect
Automate certificate renewal with a staging dry runLogin errors after a certificate renewalPrevent
In progress4
Run a monthly region-loss drillCard payments failing for one regionPrevent
Retry queue alert when depth passes 1000Payout emails delayed by six hoursDetect
Move email sending to its own worker poolPayout emails delayed by six hoursPrevent
Write the manual promotion steps into a runbookDatabase failover did not promote a replicaMitigate
Done4
Add a regional health check to the payment routerCard payments failing for one regionDetect
Fail over card traffic automatically when errors pass 5%Card payments failing for one regionMitigate
Add an index to the transactions tableDashboard slow for all customersPrevent
Alert when dashboard p95 passes 3 secondsDashboard slow for all customersDetect

Tables

Action items

11 sample rows · 9 columns

The follow-up work each incident produced, by type, owner and due day.

ActionIncidentTypeOwnerDueStatusCreated
Add a regional health check to the payment routerCard payments failing for one regionDetectBruno CastellanosSep 3, 2026Done—
Add an index to the transactions tableDashboard slow for all customersPreventKeiko Tanaka-RuizSep 13, 2026Done—
Fail over card traffic automatically when errors pass 5%Card payments failing for one regionMitigateAmara NwachukwuSep 18, 2026Done—
Alert when dashboard p95 passes 3 secondsDashboard slow for all customersDetectAmara NwachukwuSep 20, 2026Done—
Move email sending to its own worker poolPayout emails delayed by six hoursPreventBruno CastellanosOct 9, 2026In progress—
Retry queue alert when depth passes 1000Payout emails delayed by six hoursDetectPatrick OseiOct 14, 2026In progress—
Run a monthly region-loss drillCard payments failing for one regionPreventSigrid HalvorsenOct 18, 2026In progress—
Write the manual promotion steps into a runbookDatabase failover did not promote a replicaMitigateKeiko Tanaka-RuizOct 20, 2026In progress—
Test replica promotion every week in stagingDatabase failover did not promote a replicaPreventKeiko Tanaka-RuizOct 21, 2026Open—
Alert fourteen days before any certificate expiresLogin errors after a certificate renewalDetectAmara NwachukwuOct 23, 2026Open—
Automate certificate renewal with a staging dry runLogin errors after a certificate renewalPreventAmara NwachukwuNov 7, 2026Open—

Incidents

5 sample rows · 16 columns

Every incident: how severe, how long, who led it and what came out of it.

IncidentIncident numberSeverityStatusStartedResolvedCommander
Login errors after a certificate renewal—SEV2MitigatedOct 17, 1:15 PM—Sigrid Halvorsen
Database failover did not promote a replica—SEV2ResolvedOct 10, 4:10 AMOct 10, 5:50 AMKeiko Tanaka-Ruiz
Payout emails delayed by six hours—SEV3Postmortem doneSep 21, 12:30 PMSep 21, 7:00 PMAmara Nwachukwu
Dashboard slow for all customers—SEV2Postmortem doneSep 7, 3:40 PMSep 7, 5:15 PMAmara Nwachukwu
Card payments failing for one region—SEV1Postmortem doneAug 17, 8:05 PMAug 17, 10:20 PMSigrid Halvorsen

Also included

  • Mark doneRow action

Related templates