Attackers turned OpsShield’s trusted update channel into a supply-chain attack
CSO Online ·
The kinks in the software supply chain are once again being exposed, with attackers now targeting OpsShield’s update process. This week, threat actors used compromised credentials to push malicious updates out to cPGuard customers. They then accessed data stored on the vendor’s servers and in its billing portal accounts, and made copies of at least three of OpsShield’s cloud servers. The company said in an advisory that it has “shut the attacker out of every system we know they reached, and … are rebuilding the ones we cannot fully trust.” It has also released updates and urges customers to update to the latest version of cPGuard, change passwords, and generate new SSH keys. Ultimately, the breadth of this unsanctioned access underscores the continued fragility of the software supply chain, as well as the vulnerability of security tools themselves. “The real danger here is that attackers were able to turn a trusted security update mechanism into a vector to break into servers,” said Erik Avakian , a technical counselor at Infotech. “The software that’s supposed to protect those servers became the source that compromised them.” The anatomy of the attack OpsShield specializes in server protection and web security software. Its cPGuard product is a security suite, malware scanner, and firewall designed for Linux web hosting servers. The company disclosed that, between November 1 and 2, attackers were able to gain unauthorized access to the OpsShield update distribution infrastructure, and, for about 70 minutes, pushed out a malicious update. The threat actors began by probing OpsShield’s billing portal, registering a customer account under a false identity, and ordering a free cPGuard trial. They then sent crafted requests that allowed them to gain the ability to run commands on the server to collect data. They used stolen access keys to publish fake updates that ran as root (the highest level of admin privilege), and to copy sensitive files and make copies of three OpsShield cloud servers. About 2,950 customer servers ran the fake update, exposing password hashes, root SSH keys, MySQL root passwords, and shell history, OpsShield reported. Attackers also accessed about 7,900 billing portal accounts containing customer names, addresses, emails, and hashed passwords. OpsShield says it is still reviewing the exact request that provided access, and has patched two possible entry points and verified related vendor components “line by line.” It closed the attacker’s account, revoked the stolen update-storage key as well as “every cloud API token and access key on our account.” The company also removed the attacker’s secure shell (SSH); deleted the clones of its servers created by the attacker plus any snapshots; turned on stronger login protection and replaced every SSH key and password; reset all billing portal login sessions and rotated all keys; and began rebuilding its billing, licensing, and internal management servers with tighter PHP and firewall settings. The company has also published clean updates: v5.88.26, which removes the hidden root shell; plus a follow-up which removes the scheduled task left by the attacker. Further, it began re-issuing all cPGuard license private and public keys so that the copies the attacker gained access to stopped working. The company advised admins to: Update cPGuard to the latest version, then confirm that /etc/cron.d/i93677 , /var/tmp/.i93677/ and /var/tmp/.cpg91511_sh are gone; delete them if not; Change root passwords and every local user password on the server; Generate new root SSH keys and remove old ones from every server that trusted them; until the keys are changed, treat the servers as compromised; Change MySQL root passwords and any database passwords and rotate any password, API key, or token existing in root shell histories; Reset the OpsShield billing portal passwords and turn on two-factor authentication. Finally, it said, “if we told you the attacker logged in to your server by hand, rebuild it from a clean backup taken before 1 November.” OpsShield being transparent, but the impacts are still real “Frankly, few vendors write a breach disclosure this candid,” said Frank Dickson of Dickson Research. “OpsShield deserves a lot of credit for that.” However, he pointed out, the worst kind of breach is the one a security tool delivers. Update channels are left unlocked on purpose, and security tools run as root because they have to see everything, “so the fake update ran as root too.” For a hosting provider, one server can hold hundreds of customer websites and databases. A MySQL root password opens all of them, and a root SSH key grants access to every other server that trusts it. Shell history also often contains typed passwords and tokens. The downstream customer may be hit hardest while knowing the least, Dickson pointed out, as most website owners have no idea which security agent their hosting provider runs. Every server that updates itself trusts the vendor’s pipeline. OpsShield describes itself as a small team selling to smaller hosts at a low price, which is the uncomfortable truth of the supply chain, Dickson said. “Your exposure is set by the smallest vendor holding root on your servers, not the largest.” Avakian agreed that this serves as a reminder that an enterprise’s software supply chain is only as secure as the systems, identities, and processes behind it. “You can do everything you can to secure your own infrastructure and still get compromised through a third-party supplier you trust.” What’s particularly concerning here is that the attackers didn’t have to go after thousands of customers one by one; they only had to compromise a single distribution mechanism to turn the vendor’s own trusted infrastructure against its customers, he noted. Third parties extend risk, which is a fact that organizations still underestimate, he said. “Enterprises need to pay much more attention to third-party and supply chain risk management.” Zero-trust a must-have Avakian also pointed out that the impact on downstream customers could go well beyond initial compromise, as stolen credentials can be used to carry out further attacks. Enterprises should take a zero-trust approach that includes their software suppliers and the updates they rely on. “Just because an update comes from a trusted source doesn’t mean you can assume it’s safe,” he said. Enterprises should take a closer look at vendor risk, protect privileged credentials, and examine software integrity. Also, continuously monitor for unexpected behavior after an update and be ready to isolate systems quickly if and when something goes wrong. “You have to plan for the possibility that even a trusted supplier can be compromised,” he emphasized. “And you need to practice those types of scenarios as part of a regular table-top exercise program.” Dickson advised starting with an inventory, and knowing every vendor agent running with root or administrator privilege, as each one is an update channel. Customers should ask vendors where update publishing keys live and what separates them from customer-facing systems, watch outbound traffic from servers, keep secrets out of shell history, and treat SSH keys as credentials with an expiration date. Downstream customers should ask their host directly whether it runs cPGuard, then change their own passwords. “The most dangerous software on your server may be the software you trust to update itself,” he said. “Choose those vendors as if they hold root, because they do.” This article originally appeared on NetworkWorld .