Building incident playbooks for machine identity breaches

Pull up your SOC’s compromised account playbook and read through the first 15 minutes. Most SOCs begin their response by calling or messaging the user via Teams to ask whether the 2:00 am sign-in from an unfamiliar country was indeed them. Their subsequent actions depend upon what the user responds with. Try that with a service account, and you’ll be waiting a long time. The same goes for an API key buried in a build pipeline, or an OAuth token somebody granted to a SaaS integration back in 2022 and hasn’t thought about since. Once one of those is stolen, your analysts are running a script that was written with people in mind, and the cracks tend to show well before the first shift handover. There are also far more of them than most boards realise. CyberArk’s research puts machine identities at roughly 82 for every human identity. Very few sit behind MFA, and a fair number hold far more access than the job they were built for. I’d add a problem that gets less airtime, which is that their traffic lives in the background noise your monitoring was tuned to filter out. Where the human playbook stalls Ownership is usually the first thing to slow you down. When a person’s account is compromised, HR can tell you who they are and who their manager is before you’ve finished your coffee. A service account pulling data it shouldn’t comes with no such shortcut, so the analyst burns the first hour working out which team built it. More often than anyone likes to admit, that trail ends at a contractor who left the business a year earlier. Say you do find an owner; containment still isn’t the clean step it is for a person. Lock a user out and one employee loses an afternoon. Pull a credential that a payments integration depends on, though, and you might take a customer-facing service down, which no analyst wants on their name at 11 p.m. without sign-off. So, they go hunting for somebody senior enough to approve the outage while the attacker carries on. Detection is the part people underestimate. An attacker with a valid token makes authenticated calls to the same endpoints the integration hits every day, and a patient one will even keep to its usual hours. Your impossible-travel rules were built around people moving between cities, so they have nothing to compare against here. Figure 1: One alert split into two paths. Ashish Mishra What recent breaches show Cloudflare’s write-up of its Thanksgiving 2023 security incident is worth a quarter of an hour of any CISO’s time. The attackers came in with one access token and three service account credentials lifted during the earlier Okta support breach. Cloudflare had rotated thousands of credentials after Okta, yet missed those four because the team believed, wrongly, that nothing was using them. The alert that finally fired came more than a week after the attackers first logged in, when they added one of those service accounts to an administrator group. Microsoft’s case turned on a test application nobody had cleaned up. Its guidance for responders on the Midnight Blizzard attack describes a password spray against a legacy test tenant account with no MFA. Inside that tenant sat an old test OAuth app that still held elevated access to the corporate environment, and the attackers used it to get into corporate email accounts. Drift is the case I’d put in front of a board. In August 2025, attackers got hold of the OAuth tokens that Salesloft’s Drift chatbot used to talk to Salesforce, and for roughly ten days they pulled records out of one customer’s Salesforce instance after another. Google’s threat intelligence team, which tracked the campaign closely, concluded the attackers were mostly after credentials. Once the data was out, they went digging through it for AWS access keys and passwords, anything that might get them into the next company along. I don’t think any of the three needed a clever attacker at the front door. In each case, a credential was issued for a piece of work and then left alone long after the work had finished. Getting the groundwork in first It is a relatively simple task to create a playbook. However, a playbook relies upon an inventory. In many organizations, this inventory consists of a service account list generated from Active Directory where half of the description fields have been left blank. Each time you identify a high-privilege non-human identity, document the team responsible for that identity, and provide a brief explanation (one or two sentences) describing what will fail should you remove that identity. Many people do not include the explanation as to why something would fail if removed, since obtaining this information requires contacting the application team for clarification; this contact would otherwise be initiated by your analyst at 3:00 AM while the attacker remains active within your network. Next, sort those identities by blast radius. I wouldn’t put a read-only reporting token on the same response path as a cloud admin service principal, and I doubt your auditors would either. For the top tier, settle in advance who’s allowed to authorise an immediate revoke even if it knocks something over, and put that role in the playbook by name. Executives must own that call. If you leave it with whoever’s on shift, you’ll get a polite wait for permission while the token stays live. For SOCI-regulated entities, there’s an extra edge to this, because the 12-hour window for reporting a significant cyber incident to ASD can be running while that conversation drags on. Triage that fits a machine No one can provide a response to “was this you?”, thus, replace it with questions that can be answered by the log files. A rapid means of verification is location: where do the credentials usually sign in from, and where are they signing in from at present? Logging is readily available via Cloud Audit Logs and Workload Identity Platforms, if someone turned the logging on. Permissions take longer but tell you more. Line up what the identity holds against what it has used over the last few months, because attackers tend to go straight for high-risk scopes that have been sitting idle. I’d give persistence the same attention. With a privileged service principal in hand, an attacker will often mint new secrets or register an app of their own, and none of that goes away when you revoke the original credential. Containment without breaking the business The “trigger” for rotation and revocation should be written separately in the playbook. Rotation would be appropriate when you are reasonably confident that the attacker has not escalated and the organization cannot afford to lose the integration. Revocation is the obvious choice if you find evidence of an ongoing attack; this will be clear enough so there will be no debate at midnight. Pre-staging your replacement credentials will make revocations tolerable. When the team that owns the application has previously tested how to generate a new credential and then store it in their secret management system, revoking old credentials can cost you about ten minutes of downtime instead of being a full-day disruption. Ask the team to practice generating and storing new credentials during a low-traffic week. Do not find out mid-incident that the runbook references a vault location that was renamed last year. Scoping and recovery To understand the full scope of an incident, look at the entire lifespan of the machine credential, not just the period after the alert was generated. Because machines may be able to go undetected as being compromised for many months prior to discovery, the first unusual activity reported will most likely not be the first unusual activity performed. Also, keep in mind that “drift” is a great way to remember to see what information the identity has access to, and if anyone has placed credentials into that identity. If so, each addition of a secret will require you to add a new rotation to your list. Recovery is when you fix what caused the problem in the first place. For example, if a static secret was compromised, determine if that workload can use short-lived tokens using Workload Identity Federation; or perhaps a short-lived token can be generated on demand (Just-In-Time) and will expire automatically. Recovery is a tedious, non-sexy task, and most of the time it requires financial support from an entity outside of the Security Operations Center (SOC). Before finalizing the playbook, conduct a tabletop simulation with the application owners in attendance along with the Security Operations Center (SOC). Select your highest-privilege integration and simulate a stolen-token event (from first alert through recovery). During the simulation, when a decision needs to be made regarding who approves revoking the token, and revoking the token will disrupt production, observe how much time it takes for the team to identify who should make this approval.