Heroku Breach Update, Opsera Delivers Devops Secrets Management
Heroku, a division of Salesforce, last week announced status updates on their GitHub-related OAuth security breach. This extremely serious incident exposed customer passwords and data to a cybersecurity threat actor. In a status update and blog post, Heroku says it has identified the users who had been impacted by the attack, and they have been notified. In this post, I analyze the Heroku response, and interview an enterprise user on how one Salesforce devops vendor, Opsera, delivers a HashiCorp Vault integration that does better secrets management.
Heroku Says the Coast Is Clear, But…
In a short May 19, 2022 blog post, this time co-authored by two engineering staff members instead of division leader Bob Wise, Heroku offered up few explanations and mysteriously put a security choice into the hands of Heroku users.
Heroku opened the post by focusing on the scope of an OAuth token’s access to user data. Heroku said that while they don’t need the entire repo scope from GitHub, that is what they have been requesting. Apparently, this has something to do with the breach, because they further stated that they intend to implement IETF RFC8705 to restrict access in conjunction with GitHub.
Then, mysteriously, Heroku seems to offer users a cybersecurity choice. Without giving a retrospective into what caused the problem, the company stated they have a plan to restart the GitHub integration and will do so soon. Then, at the end of the post, they say “The choice is yours” to use the restarted service because they will not have yet implemented RFC8705.
Linking to the incident status page, Heroku reveals some disturbing details of what happened. Last week, on Monday May 16 the company said it identified that the threat actor had “downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI.”. And another group of users “had their Heroku tokens exposed in a config var for a pipeline.”
Heroku believes it is near the end of this incident. They claim they have identified the users whose configuration information was copied, and everyone impacted has been notified via email. They continue to reiterate the claim that no suspicious activity has been detected since April 14, 2022.
Opsera Uses HashiCorp Vault to Guard User’s Devops Secrets
The Heroku shenanigans is a warning for all devops practitioners about what happens when secrets management goes wrong. While this incident appears to be the work of an advanced threat actor, it shows how losing control of a 2K byte text file can bring a business to its knees.
Running a devops pipeline with a scripted command server can touch several different systems. All these systems have different authorization and access schemes. And the command server must use the right authentication mechanisms and data to make the connection to an external service.
Managing the API keys, authentication tokens, and other secure authentication information to make those connections is what is called secrets management. The cloud native world has been dealing with secrets management for years. One leading cloud native product that manages secrets is HashiCorp Vault. And one top Salesforce devops vendor, Opsera, now bundles Vault directly into its low-code SaaS devops platform.
What is HashiCorp Vault?
Vault is a popular identity-based secrets and encryption management system. To run the open-source version, users provision a server to run a Vault server instance. HashiCorp also offers to host Vault instances for users as a paid software-as-a-service (SaaS).
A web GUI is used to define the various secrets used in a project. Then, in a devops pipeline the Vault CLI is used to retrieve secrets and put them into environment variables, for example. Vault also has an API and a large variety of use cases and advanced features.
Opsera-Vault Integration
Opsera is a low code SaaS devops tool. It lets users design devops pipelines graphically, which puts pipeline management within reach of business creators. A key feature of Opsera is that it manages a scripted command server, which runs on a cloud-based server that accesses external systems.
Opsera tightly integrates Vault into its credential invocation system. When activated, the Opsera-Vault integration provisions and manages a private Vault instance for the Opsera customer. Next, when authentication information is used within an Opsera pipeline, it is automatically retrieved from the Vault server instance. Exiting Vault customers can bring their own instance into Opsera. The Opsera-Vault integration is currently available to all Opsera customers for no charge.
Gorkey Vemulapalli, who is Sr. Director, Precision Medicine Data & Systems in Office of the Chief Digital Officer for the City of Hope medical center in Los Angeles, is familiar with the risks of storing secrets. “We needed an option to store the secrets in one place, to help manage our risk against breaches,” said Mr. Vemulapalli in a recent interview with SalesforceDevops.net.
City of Hope, which is a major cancer and disease research center with clinics throughout Southern California, has worked in the last few years to consolidate on-premises devops efforts. Now, the billion-dollar enterprise is converting those projects to Opsera. Mr. Vemulapalli says they use Opsera “to build all of our cloud native apps and then deploy them into Kubernetes on a public cloud.”
Using Vault plus Opsera meets City of Hope’s cybersecurity encryption requirements. “Vault takes care of encryption for data-at-rest, but Opsera securely fetches the data and then encrypts it in transit,” added Mr. Vemulapalli. He also mentioned that using Vault was transparent and integrated to how Opsera worked.
Heroku and Salesforce Cybersecurity Wake Up Call
The handling of the Heroku cybersecurity incident is still nothing to brag about. It appears that specific users who were impacted by the breach have been identified and notified. The company has a plan for enhancing OAuth posture, but it curiously left it up to users to decide if they still should use the GitHub integration in the meantime. And the integration is still not up, so the whole thing isn’t over yet.
The lingering doubts expressed by the community and the poor response from Salesforce continues to call the viability of Heroku into question. One must wonder about the future of the service, and whether it can be a platform for Salesforce integration and enhancements. Salesforce already has a good relationship with AWS, so the company does have alternatives available.
In this post I covered one Salesforce devops vendor who is thinking ahead about the challenges posed by threats like those encountered by Heroku. As Salesforce becomes a key business and critical system platform, users will need to manage multi-system access automatically, and more securely, in more pipelines. Each system integration needs credentials which need to be securely stored. Manual methods for secrets management just don’t cut it. Let us hope that more Salesforce devops vendors add secrets management to their feature list soon.
