Salesforce Bruised over Heroku Breach Response

Heroku, a cloud platform-as-a-service (PaaS) frequently used in Salesforce devops pipelines, was acquired by Salesforce in 2010. On May 3, 2022, Salesforce Incident Response sent out a carefully worded email alert to all owners of Heroku accounts. In the email, Salesforce stated that they will reset the passwords of all Heroku accounts on May 4. The email gave no details about a Heroku breach, except to refer readers to a thread on status.heroku.com. The email also warned that any passwords previously used on Heroku should not be used on any other system.

In this post I take you through the breach, and what Heroku still needs to do. I will also describe why this breach says you need to evaluate your devops secrets management strategy today!

Salesforce Incident Response Email Sparked Criticism

The May 3^rd email unleashed a firestorm of criticism about the lack of an adequate response from Heroku about a security breach which had started two weeks earlier.

On April 12 GitHub informed Heroku that a cybersecurity threat actor had stolen the OAuth token which controls Heroku’s GitHub integration. Then, on April 15 GitHub published a blog post about the breach, and Heroku notified customers in the support thread. Heroku then shut down the connection with GitHub. Heroku later published some mitigation recommendations in the thread. However, no further information about the nature of the breach was available when the password reset email went out.

On May 5, Heroku posted new information about the breach in the support thread. In a shocking chronology, the company detailed a chain of events that led to the threat actor accessing authentication systems. A leading cybersecurity publication, Bleeping Computer, penned the headline “Heroku admits that customer credentials were stolen in cyberattack.”

Response from Heroku

On Friday, May 6, Bob Wise, the newly installed Heroku general manager and a Salesforce EVP, made a post. He cited several missteps while handling the breach response. After acknowledging problems, Mr. Wise said they intend to perform better. The post did not quell the firestorm, and left questions about the GitHub integration.

Mr. Wise said in the post that Heroku had detected no intruder activity since April 14. And, Heroku had not observed any threat actors accessing customer accounts or decrypting environment variables. He stated that they have received feedback on the May 5 post where they unveiled too much information all at once, weeks after Heroku had learned details about the breach.

Mr. Wise claimed to have the situation under control and said, “the integration between Heroku and GitHub is part of the magic of using Heroku.” He went on to say that Heroku will update users “over the next several weeks” on the GitHub integration. In the meantime, the Heroku-GitHub connection remains down.

Wait, Heroku is Salesforce?

Part of the anger and confusion expressed by Heroku users is because they are getting notices about a Heroku breach from Salesforce Incident Response. Up until now, all Heroku users have communicated with Heroku using a pure Heroku identity. Why at this important moment did the email come from Salesforce? Has there been some mysterious behind-the-scenes activity?

Social media conversations about the Heroku breach often digress into lamentations about the “good old days” of Heroku and complaints about Salesforce’s handling of the company. One possible source of the complaints is that since its heyday, Heroku seems to have experienced a brain-drain. Social media is peppered with former employees who complain about rudderless management and product development.

In the mid-2010s, Heroku pioneered the idea of lowering the cognitive load on developers. This was when, even after the Salesforce acquisition, Heroku “just worked” and made deploying cloud native applications easy. But Heroku has fallen behind in compact, developer-friendly code deployment systems. A recent discussion on Hacker News cited PaaS alternatives like Vercel, DigitalOcean, Render.com, and Fly.io as Heroku alternatives.

Heroku Had a History of Weak Defenses

As the new Heroku GM, Mr. Wise has his work cut out for him. He needs to overcome years of poor oversight in cybersecurity. A management reckoning must be underway with this existential security breach.

It appears this incident has been poorly handled partly due to the history of Heroku cybersecurity practices. One insider who worked in Salesforce Incident Response until 2020, and now works in the federal government, spoke confidentially to SalesforceDevops.net about Salesforce security management practices. Unlike other Salesforce clouds, Heroku incident management was handled “at arm’s length.”

The resources devoted to Heroku cybersecurity in the years before this incident did not match the resources put into other Salesforce clouds or divisions, according to the former Salesforce cybersecurity analyst. Heroku security staffers lacked the resources required for a proper cybersecurity posture at the time. “It seems like they only had one or two guys, maybe not even full-time [cybersecurity staff],” said the analyst.

Salesforce usually funds key operational functions like security and platform to a luxurious extent. The former Salesforce analyst confirmed that Salesforce Incident Response is well funded with generous staffing, excellent recruiting, access to key systems, and 24/7 global management. Apparently, in the years before this incident, the Heroku cybersecurity team was off on a cybersecurity and operations island.

Waiting for GitHub Restoration

It is disconcerting that Heroku has yet to restore the GitHub connection. Without further information we are left to speculate about possible explanations. People are left to wonder if they still may not know exactly what happened and how to stop it in the future.

Even though Heroku has not detected any activity since April 14, that does not mean the threat actor cannot perform further exfiltration activities. Due to this and other factors, the nature of the Heroku attack is worrisome for the former Salesforce cybersecurity analyst. “The man in the middle is the ever-present ghost sitting on your shoulder, and you never know when he’s listening,” he said ominously.

Avoid the Double Tap Cyberattack

Since user authentication information was exfiltrated from Heroku, every user must completely re-generate account passwords and associated environment variables. And the former Salesforce analyst warned that running mitigation activities just once may offer insufficient protection, however.

When a threat actor uses the “keys to the kingdom” for exfiltration, a one-and-done attitude to mitigating a breach may not be enough. After full functionality is restored, or an acceptable retrospective is produced by Heroku, mitigation procedures should be rerun, advised the cybersecurity analyst.

Threat actors have been known to remain undetected in breached systems. They wait for users of breached systems to regenerate credentials in a corrupted system. When a threat actor remains hidden for a secondary attack, it is known as the Double Tap Cyberattack. It should be emphasized that there is no evidence this is happening to Heroku.

The double tap cyberattack is named after a tactic used in the Iraq and Afghanistan wars. A double tap bombing attack occurs when an enemy terrorizes a civilian population by first bombing crowded public areas such as markets. That is the first tap. The enemy then waits a short period of time and attacks the first responders and civilians searching for survivors for the second tap.

Devops uses Secrets Management

OAuth tokens and environment variables are frequently used within Salesforce devops pipelines. Using a scripted command server, such as GitHub Actions, Azure DevOps, or even Opsera or Copado, sends the secrets over the Internet.

Whenever authentication information must be handled in a pipeline, that activity is called secrets management. Authentication information is used in devops to log onto Salesforce, access source code repositories, run tests, perform cybersecurity scans, and access cloud-based services. In other words, secrets management is everywhere in devops.

Cyberattack Fast Recovery is Critical to Enterprise Resiliency

This Heroku breach, man-in-the-middle attacks, and the theoretical notion of a double tap cyberattack all highlight the potential vulnerability of your secrets management practices. The former Salesforce cybersecurity analyst emphasizes the need for users to be ready with a safe and responsive secrets management strategy. “Setting up static things doesn’t work. When faced with a cyberattack of my authentication data, I have two questions. What do I need to know? And how do I recover fast?”

Fast recovery is a key emphasis in the US government’s civilian cybersecurity doctrine. Since 2021 CISA and the White House have emphasized the need for critical infrastructure to have plans to recover quickly from an unanticipated cyberattack.

This is based on the realization that cyber attackers have the upper hand in cyberwar. Attackers have access to zero-day exploits and use unforeseen attack methods, especially when attacks are state-sponsored. So, you are always at risk of being taken down by a malicious actor.

The proper response is to go beyond the active and passive measures designed to keep out intruders. You need to assume your secrets will eventually get exfiltrated. For application recovery resiliency you need a plan and a secrets management infrastructure ready to recover quickly from an attack.

Salesforce Gets a Bruise from a Troubled Child

Without a doubt, the Heroku breach is a negative note for Salesforce cybersecurity and a red alert for every Heroku user. However, the problem appears to be more with corporate governance than a generalized Salesforce problem. After allowing the division to languish on a corporate island for years, the company is now faced with an embarrassing cybersecurity incident. But it appears that Salesforce is moving in the resources needed to fix the situation. Let’s hope Heroku is fully resolved soon.

And, figuring out how Heroku moves forward isn’t going to be easy. Heroku clearly has two separate customer bases. The first customer base consists of the cloud native developers who love all application deployment features. It should be noted that a fair number of those customers don’t care about Salesforce.

The other Heroku user base consists of Salesforce users and the Salesforce platform itself. Many Salesforce customers have integrated Heroku into their IT infrastructure. And Heroku supports features in Salesforce, such as Salesforce Functions. Heroku is also frequently used by advanced Salesforce AppExchange packages. Based on these dependencies, it seems unlikely that Salesforce will abandon Heroku. The appointment of a new general manager is a good sign that changes may be underway.

Heroku Breach Offers Lessons and Cautions

Currently, there is still cause for concern about the Heroku breach. The company still needs to restore the GitHub connection and produce a reasonable technical retrospective on the incident. After that, it would be advisable for customers to re-run any mitigation procedures.

Hopefully this event will increase awareness about how secrets management permeates devops. If your authentication information is stolen, do you have a way to quickly change and re-deploy it in your devops pipeline?