FBI-Confirmed Water Utility Lockouts Trigger OT Incident Command

The water incidents were covered on August 17 as remote PLC manipulation without verified malware. Today’s delta is FBI confirmation across seven states, operator lockouts, changed IP addresses/passwords, and reported pressure loss and flooding. Decide whether this is now a coordinated safety campaign or still a set of loosely related intrusions. Give water operators the three decisions they must make today, and name the evidence that would disprove your coordination assessment.

Halil, I assess this as one coordinated campaign of opportunity at moderate confidence—but not yet a proven campaign intentionally designed to cause physical harm. FBI/EPA link incidents since July 27 across at least seven states to internet-facing MicroLogix 1100/1400 PLCs; CISA reports the repeated password/IP-change lockout pattern, with pressure loss, flooding, boil-water notices, and sustained manual operations. That places initial impact at Purdue Level 1, with consequences reaching the physical process. The evidence establishes a common campaign pattern, but not common command-and-control, attribution, or safety intent. FBI/EPA (opens in new tab) | CISA (opens in new tab)

Operators have three decisions today:

1. Can the process remain safely under remote control? Independently verify pressure, tank levels, pumps, and valves; move affected zones to approved local/manual operation if PLC state or operator access is untrustworthy.
2. Which access paths must be isolated now? Remove direct PLC internet exposure and contain indirect Level 2–3 paths—including engineering workstations and vendor access—without interrupting required control or safety communications. Validate segmentation changes before deployment.
3. How will recovery be sequenced? Preserve configurations and network evidence, compare logic/IP/passwords with known-good baselines, restore trusted credentials, and return one process zone at a time under operator supervision. Do not perform an IT-style reset or patch cycle while the plant is unstable.

What would disprove coordination: forensic timelines showing unrelated opportunistic scanners; no shared source infrastructure, accounts, tooling, or distinctive configuration changes; materially different initial-access methods; or credible evidence that separate actors independently exploited the same exposed devices. Because this playbook is simple, identical outcomes alone are not enough to prove central direction.

We already established that patching alone is insufficient for CVE-2026-59310, CVE-2026-12569, and CVE-2026-65400. Rank vCenter, Windchill/FlexPLM, and exposed Macs by the threshold for assume-compromise handling today. Separate verified exploitation artifacts from ransomware, Clop-scale, and actor claims, and give one decisive hunt condition per platform that should trigger rebuild or credential rotation.

1. vCenter — lowest assume-compromise threshold. Shadowserver reports identified victims compromised through CVE-2026-59310. Assessment: an exposed, vulnerable vCenter during the exploitation window deserves incident handling, not patch-and-watch; it controls the virtualization estate. Decisive trigger: any unexplained persistence such as a malicious cron entry or reverse_ssh artifact means rebuild the appliance and rotate vCenter, SSO, service-account, and reachable ESXi credentials.

1. Windchill/FlexPLM. PTC published an observed command-and-control IP (216.152.148.54) and a custom C2 command-header indicator. Decisive trigger: evidence that this header reached the application—or unauthorized server-side code executed—means rebuild affected nodes and rotate application, database, integration, and service-account secrets. The Clop label does not establish Clop-scale exfiltration, ransomware deployment, or compromise of every exposed instance.

1. Exposed Macs. CISA lists CVE-2026-65400 as exploited; reported cases involved root access and Monero miners. Decisive trigger: a Screen Sharing event with session_username: root should trigger rebuild and rotation of local administrator credentials and credentials used from that Mac. Exposure alone warrants isolation and hunting, but its likely blast radius remains below an infrastructure controller or enterprise PLM server.

The verified picture is exploitation plus host artifacts. Ransomware deployment, mass Clop-scale theft, and named-actor attribution remain unverified here. Don’t let campaign branding outrun the evidence.

Treat TheHatman’s Azure/Entra theft claims and Wiz’s observation of rogue device registration in nearly one in seven Entra environments as separate evidence streams. What identity-control failure is actually demonstrated, what remains alleged, and which two Entra changes and two audit queries should a CISO approve tonight to prevent a compromised account from converting device enrollment into durable Microsoft 365 access?

Halil, FACT: Wiz demonstrated a tenant-control failure: Entra’s Device Registration Service lets a holder of a compromised identity register a rogue device, potentially satisfying poorly designed device-based access policies. Wiz observed this in nearly one in seven environments over 90 days. That demonstrates abuse of legitimate tenant trust—not an Azure platform vulnerability. UNKNOWN: TheHatman’s claims remain allegations; the available reporting does not independently establish the acquisition path, affected tenants, or connect the claimed dataset to this device-registration technique.

Approve two changes tonight:

1. Create Conditional Access for the “Register or join devices” user action, requiring the phishing-resistant authentication strength—FIDO2/passkey or certificate-based authentication—not SMS or push approval.
2. Restrict device registration/join permissions to a dedicated enrollment group. Remove ordinary and privileged accounts unless enrollment is part of their role. Also verify that Microsoft 365 access policies require a managed, compliant device, rather than treating registration alone as trust.

Run these two Sentinel queries:

``kusto AuditLogs | where TimeGenerated > ago(7d) | where LoggedByService == "Device Registration Service" | where OperationName == "Add device" | extend UPN=tolower(tostring(InitiatedBy.user.userPrincipalName)), IP=tostring(InitiatedBy.user.ipAddress), DeviceId=tostring(TargetResources[0].id), Device=tostring(TargetResources[0].displayName) | project AddTime=TimeGenerated, UPN, IP, DeviceId, Device, Result | order by AddTime desc ``

``kusto let NewDevices = AuditLogs | where TimeGenerated > ago(7d) | where LoggedByService == "Device Registration Service" | where OperationName == "Add device" | extend UPN=tolower(tostring(InitiatedBy.user.userPrincipalName)), DeviceId=tostring(TargetResources[0].id) | project AddTime=TimeGenerated, UPN, DeviceId; SigninLogs | where TimeGenerated > ago(7d) and ResultType == 0 | extend UPN=tolower(UserPrincipalName), DeviceId=tostring(DeviceDetail.deviceId) | join kind=inner NewDevices on UPN, DeviceId | where TimeGenerated >= AddTime and TimeGenerated <= AddTime + 24h | project AddTime, TimeGenerated, UPN, DeviceId, IPAddress, ResourceDisplayName, AppDisplayName, ConditionalAccessStatus | order by AddTime desc ``

Resolve the prior disagreement over LiteLLM 1.82.7/1.82.8: should teams roll back to 1.82.6, move only to a separately verified clean release, or freeze execution entirely? Define the evidence that distinguishes download, installation, Python execution, and confirmed exfiltration; then give a minimum CI/CD containment and secret-rotation decision tree without repeating the unverified 2,500-organization or 434,000-pipeline estimates as compromises.

Halil, freeze potentially affected environments immediately, then restore only from a separately verified artifact. Version 1.82.6 is a rollback candidate—not automatically trusted. Use it only if its wheel hash matches a pre-incident artifact and its contents have been reviewed. Otherwise, deploy a later clean release after independently comparing source, wheel contents, provenance, and hashes. The confirmed failure is in the distributed PyPI artifacts; whether the maintainer account, build system, or publishing credentials failed remains unknown.

The evidence ladder is:

• Download: proxy, PyPI-mirror, or cache logs show retrieval. This does not prove installation.
• Installation: pip/image-layer records, SBOM, dist-info/RECORD, or filesystem evidence confirms the version. In 1.82.8, look specifically for litellm_init.pth; 1.82.7 placed the payload in proxy_server.py.
• Python execution: for 1.82.8, any Python interpreter start with that site-packages active may trigger the .pth payload—no LiteLLM import is required. For 1.82.7, evidence must show the malicious proxy module was imported or executed.
• Confirmed exfiltration: an IOC-domain connection alone proves contact, not stolen data. Require captured outbound content, equivalent endpoint telemetry, or subsequent verified use of an exposed credential. The documented domains include models.litellm[.]cloud and checkmarx[.]zone.

Minimum CI/CD decision tree: download only → block both versions, purge caches, and inspect downstream artifacts; installed but demonstrably never executed → destroy the environment/image and rebuild, with rotation generally limited to credentials whose non-exposure cannot be established; Python executed → revoke and rotate every secret reachable by that job or runtime, including CI, cloud, registry, API, and signing credentials; IOC traffic, telemetry gaps, or confirmed exfiltration → treat those secrets as compromised, revoke before rebuilding, review their use, and replace runners rather than cleaning them in place. Lock the replacement by hash, generate an SBOM, and reject unexpected .pth files during artifact inspection.

Close the room operationally. Sara assesses the seven-state water activity as a coordinated campaign of opportunity at moderate confidence, not proven intent to cause physical harm. Alex puts exposed vulnerable vCenter at the lowest assume-compromise threshold, followed by Windchill and then exposed Macs. Marcus separates demonstrated Entra device-registration abuse from unverified TheHatman scale. Tomas says LiteLLM 1.82.6 is only a candidate if independently verified, not automatically safe. Build a 24-hour sequence for a resource-constrained CISO across these four subjects. Push back against blanket isolation or blanket secret rotation: state the evidence gate for each disruptive action and identify what can safely wait.

0–2 hours — establish exposure and preserve evidence. FACT: Public reporting describes active exploitation of vCenter CVE-2026-59310, Clop web shells on Windchill, and exploitation of a macOS Screen Sharing flaw. ASSESSMENT: Put internet-exposed, vulnerable vCenter first: restrict management access immediately, preserve logs and persistence artifacts, then canary-test and deploy the fixed release. Isolate or rebuild only when exploitation artifacts, unauthorized root activity, or integrity loss appear. Apply the same gate to Windchill—web shells or unauthorized data access trigger isolation and incident response; exposure alone triggers access restriction and validated patching. For Macs, disable exposed Screen Sharing now, but isolate endpoints only upon miner execution, persistence, or suspicious outbound traffic.

2–8 hours — run identity and water lanes in parallel. Entra device-registration abuse is demonstrated; TheHatman’s claimed scale remains unverified. Review registration and authentication records, remove confirmed rogue devices, revoke associated sessions, and reset affected identities. Do not rotate every tenant secret: rotate an application or service-principal credential only when logs show access, export, or use by the compromised identity. For water operations, review remote-access accounts and preserve IT/OT logs; disconnect an OT asset only after unauthorized remote access, configuration change, or loss of view/control. Otherwise, keep operations running while restricting external administration and preparing manual control.

8–24 hours — contain LiteLLM without destroying the build estate. Freeze LiteLLM upgrades and identify where packages executed, especially CI/CD runners. UNKNOWN: I found no independent evidence here that version 1.82.6 is safe, so treat it as a candidate pending repository and artifact verification. Isolate a runner and rotate only credentials reachable from it if the suspect package executed or credential access is evident; a lockfile reference without execution does not justify either action. Enterprise-wide isolation, blanket password/API-key rotation, non-exposed Mac remediation, and unaffected internal application patching can wait until after this first-day containment cycle.

Give the board-level business call for the water-utility incidents only. With operator lockouts, changed network settings, and reported pressure loss/flooding—but no proven destructive intent—set concrete thresholds for activating manual operations, emergency procurement, public communications, and service-continuity escalation. Avoid a wide cost range; identify the single operational metric the board should receive every two hours.

The FBI/EPA report incidents across at least seven states since July 27, with operator disruption, pressure loss and flooding. With no proven destructive intent, treat this as an operational emergency—not yet a catastrophe declaration.

• Manual operations: activate immediately after any confirmed operator lockout or unauthorized setting change; also when an independent gauge shows pressure outside the utility-approved band for five minutes.
• Emergency procurement: activate when trusted automated control cannot be restored within four hours, or when a required PLC, modem or communications spare is unavailable onsite.
• Public communications: issue a holding notice within 30 minutes if pressure loss or flooding reaches customers, or if pressure, treatment or water-quality variables cannot be independently verified. Coordinate any health advisory with the relevant authority.
• Service-continuity escalation: open the emergency operations structure and request mutual aid when at least 10% of customers are affected, any hospital or fire-service supply is endangered within two hours, or manual operation is expected to exceed one eight-hour shift.

Every two hours, give the board one metric: Verified Safe-Service Coverage—the percentage of customer connections receiving water at acceptable pressure while treatment and quality are independently confirmed. Below 99% requires executive monitoring; below 90% triggers full continuity escalation. These are proposed internal decision thresholds, not universal regulatory standards.

Claim: Your plan keeps vCenter online unless exploitation artifacts, unauthorized root activity, or integrity loss are found, and limits water-asset disconnection to evidence of unauthorized access, configuration change, or loss of control. Counter-evidence: Alex says an internet-exposed, vulnerable vCenter during the known exploitation window already crosses the incident-handling threshold because vCenter controls the virtualization estate. Pierre says the water events are an operational emergenc... [arg clipped]

Halil, Alex is right: I set the vCenter threshold too high. FACT: CVE-2026-59310 is in CISA’s KEV catalog, enables network-based arbitrary code execution, is actively exploited, and has no workaround. ASSESSMENT: An internet-exposed, vulnerable vCenter enters incident handling immediately. Network-isolate it from untrusted and nonessential paths, preserve evidence, hunt for persistence, then deploy the applicable fixed release through an expedited canary. Rebuild if integrity cannot be established; rotate credentials only if credential-store access, administrator misuse, or downstream ESXi access is found.

Pierre’s operational concern also lowers the water threshold. CISA and the FBI report degraded operations, lost monitoring/control, flooding, boil-water notices, and sustained manual operation. Therefore, an internet-facing PLC should be disconnected from the public internet now—even without local compromise evidence. That does not mean shutting down the OT process or isolating every water asset: full process isolation requires unsafe state changes, unauthorized configuration, or inability to maintain safe control, coordinated with operations staff.

For customer-managed GitLab exposed to CVE-2026-19478, give the immediate unauthenticated GraphQL restriction, fixed-version deployment, and repository-integrity checks required today. Distinguish public-project manipulation exposure from proven compromise, and name the evidence that should trigger downstream release invalidation, credential rotation, and full incident response.

Halil, today’s order is: restrict, preserve, patch, verify. At the reverse proxy, deny unauthenticated access to /api/graphql; where complete denial is operationally impossible, at minimum block unauthenticated POSTs and GraphQL mutations/directives or allowlist trusted networks. Preserve proxy, GitLab application, audit, and GraphQL logs before upgrading. Deploy 18.11.11, 19.0.8, 19.1.6, or 19.2.4, according to the installed release branch.

An affected internet-facing version with public projects establishes exposure, not compromise. GitLab says the flaw can let unauthenticated users modify or delete public projects and user data; NVD currently records exploitation as “none.” Check public repositories against trusted clones or backups: commits, branches, tags, protected-ref settings, release assets, package-registry objects, project/user metadata, .gitlab-ci.yml, build scripts, webhooks, deploy keys, and artifact hashes or signatures. Correlate discrepancies with anonymous GraphQL requests and audit events.

Alex’s threshold is the right one: invalidate downstream releases when unauthorized activity touches commits, tags, CI configuration, packages, release assets, or provenance—or when integrity cannot be reconstructed for the relevant release window. Rotate credentials if altered code executed in CI, access controls or deploy keys changed, or runners could reach tokens, signing keys, cloud credentials, or registries. Trigger full incident response for confirmed unauthorized mutations/deletions, persistence, credential use, modified releases deployed downstream, or unexplained integrity failures with inadequate logs.