Communication Tools Internally: We primarily used Slack for real-time communication, enabling quick collaboration and integrations.
Externally: Our main external communication method was email for engaging with stakeholders.
Communication Framework Here is an excellent framework on how to communicate effectively during incidents:
Conditions: The current scenario—what's happening right now?
Actions: The dance of deeds—what's being done?
Needs: Resources, time, or assistance required. What do you need?
Example: "Need the network team to join the bridge and assist with load balancer logs."
Example: "Require 10 minutes to dive into logs for insights."
This framework helps keep communication simple and removes emotional responses that sometimes enter these conversations.
Guidance for Incident Severity Response Here's some guidance for additional resources depending on the severity of an ongoing incident.
Dispatch The group must mobilize and respond to the ongoing disruption. Generally, this is the on-call person responding to help mitigate. We dispatch them to take action; their help is mandatory in solving the issue.
Building Dispatch Groups: On-call schedules generally indicate who the people are. If you don't have that available, start building a manual roster in a spreadsheet.
Inform Stakeholders outside of engineering may need to be informed of ongoing disruptions. This group generally includes customer support, press relations, senior leadership, critical business partners, legal, etc. These teams likely have their own process for dealing with an incident and require less technical information.
Notify The notification groups are unnecessary to resolve an incident or interact with those fixing it. Generally, this involves broad communication to an email distribution or public chat channel, informing everyone of the ongoing activity. This communication group may offer additional resources due to being notified.
You can automate a significant portion of this workflow by leveraging Slack, external status pages, emails, or even SMS directly. Let me know if I can help
