Build Better Apps with Salesforce Devops
Check out the video and script from my Salesforce Devops presentation to the inaugural Pakistan Dreamin’!
Slide 1
Welcome everyone to Pakistan Dreamin’ 2021 sessions. I’m Vernon Keenan from SalesforceDevops.net coming to you today from Oakland, California.
I work as a senior information technology industry consultant. I’ve been in the industry a while. I started out when I earned a Biomedical Engineering degree at Northwestern University where I programmed a PDP-8 with punched paper tape and measured insulin in rats.
In my long career I have been a teacher, SPSS programmer, database administrator, clinical researcher, technology journalist, product marketing manager, market researcher, management consultant, and industry analyst. Most recently he is a telecom operator, cloud architect, Go devops engineer and Salesforce Developer/Architect.
Today’s topic is Build Better Apps with Salesforce Devops. Let me tell you a little bit about how I came to see devops as the key to unlocking Salesforce as an app delivery platform.
Until very recently, Salesforce definitely suffered from being in a developer backwater. I could go through the litany woes I had about Salesforce development in the mid 2010’s, but it all boiled down to speed.
Using tools like change sets, Eclipse, and Developer Console I found it extremely frustrating to do things like copy metadata between orgs. Salesforce development was, and pretty much still is, very slow.
This was pointed out to me in the extreme when I crossed over to the cloud native world and basically ported much of an existing Salesforce org to a microservices architecture. I used the strangler pattern to make a bunch of microservices in Go.
I was kind of blown away and embarrassed by how much more advanced software engineering techniques are in Go and Kubernetes when compared to Salesforce. Obviously there is a fundamental difference, and you really can’t compare the two, but I knew that Salesforce was getting left behind.
Then we have a global pandemic and the demand for digital projects skyrockets, and all of sudden the speed with which one builds a Salesforce app becomes a big deal! That is when I noticed a unicorn job description popping up all over the place — Technical Architect with DevOps Engineering.
That led me to find the Top 5 Salesforce devops platform companies who are essentially trying to productize that unicorn job description. Then I realized no independent analyst was covering the space, so I started the site SalesforceDevops.net in April, 2021. You can go there to get more information about me and read many of the articles that will form the basis for today’s talk.
Time for the session. Let’s commit!
Slide 2
I’ve spent 2021 curating and cataloging the companies, products and services that help Salesforce customers deliver enterprise applications. Here is what I’ve learned and I am ready to share!
It is my goal to give you a general view of what devops is. Then we will get more specific and try to define Salesforce Devops.
Next, we will get into some of the build-vs-buy issues and learn a bit about building your own devops pipeline from scratch.
Then I’ll take you through the Top 5 Salesforce Devops platforms, and I’ll give you my quick take on each of them.
Finally, we will try to elevate ourselves to seeing the next thing in devops – App Devops.
Slide 3
If you are “doing devops” today that says you are managing how to build the means of production for your organization. If you want to exist as an enterprise in the digital domain, then you need DEVOPS to have the automation capabilities that delivers enterprise app updates and continuously improves app value.
The origin of the devops concept started in 2007 when its main push was to find ways to obliterate barriers between app developers and system operators. Developers became frustrated with the time it took to iterate on software builds when dealing with operations teams.
This diagram from Azure DevOps website portrays it beautifully. It shows how better communication results in the collaboration and teamwork necessary to manage, develop, deliver and operate a live application. It shows the main benefit of using devops, which is to have a mechanism that allows teams to respond quickly to user behavior.
Another key to the success of DevOps is the idea of metrics and observability, and how that dovetails with agile management philosophies, such as tying metrics into objectives and key results. In fact, I’d say that Devops and Agile have been kissing behind the barn, and a new unified app production perspective is evolving which I call Devops 3.0.
Slide 4
To talk about how devops has changed over the years, I’ve divided its history into three separate eras. I call this the Devops 1.0 to 3.0 evolution.
As I mentioned, devops was conceived in early 2007, and in those early days the need to break communication and cultural barriers was the main subject of discussion. Then, in the last decade we have steadily seen a progression where those barriers have come down, and many organizations are now on a road to faster development cycles and a better ability to respond to user behavior.
Let’s take a little tour of these devops eras to see where Salesforce fits into the picture.
Slide 5
DevOps 1.0 grew in power as cloud computing made it easier for a developer to spin up new IT infrastructure. In the early 2010’s, Docker accelerated DevOps 1.0 by creating deployment artifacts that behave like self-contained virtual machines. Today, Docker containers are the main artifact type generated by DevOps 1.0 build automation processes.
Jenkins emerged as the first free open-source software (FOSS) implementation of a DevOps 1.0 command server in 2012. Jenkins uses web hooks and scripting agents that automatically compile and link source code into an artifact. Artifacts are then continuously delivered to a production environment. Jenkins is still a leading pipeline orchestration system and command server in 2021.
Jenkins-driven continuous delivery accelerated development, which enabled technology companies to update their cloud native apps faster. Companies delivered small changes to an app continuously. This allowed them to see how audiences reacted to small changes in an app. This allowed technology companies to use continuous delivery to learn more about the psychology of their users.
DevOps 1.0 emerged as a transformational force in the technology industry. Guided by its old motto of “Go fast and break things,” Facebook used continuous delivery to fine tune their products. Facebook, and other app developers, grew their markets by learning how to addict users to their app. They used continuous integration to experiment with small incremental changes to amplify desired user behavior.
Slide 6
DevOps 1.0 as practiced by technology companies evolved into a new set of job functions that I call DevOps 2.0. This was done as the need for online system integration services spread globally throughout the 2010’s.
After seeing how Silicon Valley used DevOps 1.0 to dominate the Internet with innovative apps, big organizations around the world started using the same tools. DevOps 2.0 blossomed when corporate IT teams started using software used by Netflix, Uber, Airbnb, and Twitter. Eventually many organizations began to effectively copy how Silicon Valley delivered cloud native applications.
The increased interest in cloud native computing and DevOps 2.0 ignited an explosion of FOSS projects. This created new communities and companies to support pivotal projects like Jenkins, Apache Kafka, and Kubernetes. By the mid-2010’s enterprises had access to many of the cloud native goodies that Silicon Valley still uses to build and deliver technology products.
In 2021, most big enterprises now use bits and pieces of cloud native computing and DevOps 2.0 to improve their existing enterprise apps and delivery techniques. The demand for a “DevOps Engineer” that assembles the FOSS bits and pieces needed for a solution is stronger than ever.
It seems like the original idea of DevOps 1.0, which was to give developers continuous delivery tools, has changed. Today, most corporate DevOps Engineers no longer perform any coding or development activities. Instead, they provide a service within app production teams by using implementation skills in cloud native computing.
Slide 7
The pandemic has inspired quick adoption of new management practices. Many of these are centered around the Japanese Kanban board and process flow management practices. The shortcut term “agile management” now describes this management philosophy.
In the last decade, agile management has demolished some traditional corporate structures because agile doesn’t respect traditional hierarchical communication structures. Agile organizations create ad hoc and permanent teams to deliver Objectives and Key Results (OKR). This methodology finds its roots in the “management by objective” techniques that started in Japanese manufacturing factories in the 1960’s.
Based on scientific methodologies, OKR-driven management seeks to create measurable outputs in the processes that achieves a key objective, which is also measurable. For example, Salesforce app production projects could have an OKR like “Deliver a field service app that improves service rep response times by 30%.”
Just like how the term “devops” is now an overly generic way to refer to enterprise application delivery, the term “agile” has evolved to represent the management techniques inspired by the Kanban board and other OKR-related topics.
The general concept of “devops” and “agile” is becoming somewhat blended as well. Devops and agile share the philosophies of continuous delivery and user testing, enhanced communication, and advanced project management tools.
Slide 8
Now that we’ve thoroughly discussed devops, and the cloud native world that it helped to grow, let’s see how it relates to Salesforce Devops.
One reason why Salesforce devops has gained prominence is because it represents the convergence of three huge business forces. First, Salesforce is now a business transformation platform, and not just a CRM. Since the platform is strategic for many more businesses, executives need a governance harness and better visibility into app production activities.
Next, we have the cloud native world of “traditional” devops which we have been discussing.
Then in 2017, Salesforce became accessible to the cloud native world. That is when SFDX-CLI was introduced and that really made Salesforce devops take off. It allowed LINUX scripts and cloud native tools to now interact with configuration information in Salesforce.
That subsequently allows cloud native automation tools like Jenkins and Azure Devops to act as a command server in a release management pipeline orchestration scheme.
Slide 9
I made this relationship classification system to help figure out if all the boxes are checked when you are building a continuous app delivery system. This diagram gives you an idea of the complexity involved in a Salesforce devops solution. So, the good news is that we have good roadmaps for how to build a devops solution, but the bad news is that there are a lot of things to do and a lot of systems that need to be tied together.
That’s where one of the Top 5 Vendors come into play. Salesforce devops platform vendors are all working to check these boxes off for you, so they are worth a look if you are getting started.
Slide 10
Now, let’s take that mess of products and reorganize them into this stack. This summary shows how Salesforce devops is first controlled by the governance functions, which collects metrics and data from the app production teams. Next, these teams of architects, developers, admins, business analysts, and constituents use a consistent set of automation and deployment tools to continuously optimize apps.
One key thing that makes this different for Salesforce app is the curious way in which we must manage metadata. A key function in Salesforce devops is that metadata must be ingested, analyzed, queried, and dynamically updated. When updating a live org, this is the only way for an app production team to work together. There must be intelligent metadata management in any Salesforce devops platform in order to optimize your release management processes.
Slide 11
Now we are expanding the pyramid to show a system architecture diagram for a full-fledged Salesforce devops solution, pretty much like what you get with one of the Top 5 solutions we will discuss.
Here we can see some of the more familiar names and details that go into creating a full devops solution. The elements are stacked in order of functional granularity, and generally are related to adjacent layers.
I call it an Industry map because if you go to SalesforceDevops.net and go to the Map section you can basically click into any of these categories and see the companies that I’ve curated.
Slide 12
Now, here is another view of the Salesforce devops industry. Ben McCarthy of SalesforceBen and I worked together to design this Period Table of Devops Solutions using the SalesforceDevops.net database. This will be published September 27 on his site, so you are getting a world premiere preview of this new infographic.
You can see all the categories I laid out in the ontology and map illustrations. If a company has a product in a category, it gets a badge. For example, you can see the “G” symbol for Gearset showing up in multiple categories since they are a leading Salesforce devops platform.
I know this is an interesting graphic, so I’ll go through all the categories, and you can get a chance to check it out.
Repository Management – These companies have Salesforce-specific source code management solutions.
Release Management – The complex process of using repositories to deploy new releases to your Salesforce org is supported by these companies.
Cybersecurity – These companies help you integrate key application security features directly into your development pipeline.
Data Backup & Recovery – These companies help you backup your org’s data and to integrate backup processes into your devops pipelines.
Profile Management – These companies offer another key utility in devops pipelines: the ability to copy and manage profiles.
Data Transformation – These companies offer export, translate and load (ETL) tooling and the ability to integrate data ingest and export into your devops pipelines.
Testing Tools – Testing is a multi-layered concern, and these companies help you use modern methods to increase app delivery reliability.
Sandbox Management – These companies offer advanced Salesforce sandbox management features, including data masking and managing privacy concerns.
Observability – These companies support the use of metrics, incident management, and graphical analysis for application performance management.
Pipeline Orchestration – These companies offer a “command server” which runs on Salesforce or a cloud native application which manages the execution of processes that perform release management and other devops functions. Also known as a “CI/CD tool.”
Engineering Services – These are companies who, in addition to offering software services, have professional services practices for devops system development.
Application Management Lifecycle – These are the companies who make the tools used to manage software teams.
Value Stream Management — These companies have advanced agile management tools we will discuss more later.
So, be sure to watch out for that article which will come out the week after Dreamforce and go to SalesforceDevops.net right to find these products and companies.
Slide 13
Here is another fun slide. This portrays what goes into a DIY or a platform-based devops solution.
Most Salesforce devops activity centers around what I call a command server, which is also referred to as the CI/CD engine. The command server is a program that manages system activities during Salesforce release management. Using a command server to automate the release management cycle is a real benefit that decreases Salesforce deployment times.
A Salesforce devops command server runs either on Salesforce, on-premises, or as a cloud native application. For example, both Copado and Flosum run their platform on top of Salesforce, while AutoRABIT, Blue Canvas, and Gearset all have their own cloud native apps, which is called Public Devops Clouds in this diagram.
The top 5 platforms also allow for integration of private command servers into how the platforms manage Salesforce releases. In this diagram I show some Salesforce architects create their own command server by running it on private enterprise resources. This is where the enterprise often uses Azure DevOps or Jenkins to set up a private devops command server.
Each enterprise IT environment is unique, and this illustration shows all the enterprise devops services deployed on-premises. Other implementations may spread those services around to Salesforce, public clouds, and public devops clouds.
Let us drill into the main parts of this diagram.
Slide 14
Now, let’s continue being practical and talk about some of the ways to set up a devops workflow for developers.
Allow me to say that it is essential that developers be working in SFDX-CLI and VS Code daily. The days of Developer Console and Eclipse are over. If you have not yet figured out how to get going with VS Code, that should be job number 1 in your devops journey.
One major problem in enterprise app delivery is setting up the individual workstations to support Salesforce VS Code integration. I have a favorite technique to get around this issue by focusing my setup efforts on a LINUX server. As I show in this diagram it is deployed in the same network in which I have my workstation. I then use VS Code Remote SSH to use my Windows or Mac workstation as a kind of terminal client. This setup, plus GitHub configuration sync, allows developers to get up and going on a brand-new workstation in a fraction of the time compared to manual setup.
Slide 15
I mentioned command line interfaces, or CLI’s, because these are super-important. CLIs provide a connection between cloud systems like Salesforce and LINUX scripting.
Now, when I talk about LINUX scripting, I specifically mean BASH scripting on either the Redhat, Debian or Alpine LINUX platforms. It is important to have a standard scripting execution engine when developing your own Salesforce devops scripts.
I already mentioned SFDX-CLI, and it is fundamental in Salesforce devops for a couple of reasons. It not only provides a way to read and write metadata to live Salesforce org, but it is also a platform from which independent software vendors may deploy other developer-related tooling. For example, DigitSec S4, which is a suite of cybersecurity solutions, performs an application security scan using an SFDX-CLI extension.
There are literally hundreds of CLIs you could include in your Salesforce devops scripts. My favorite Open-Source packages are DX@Scale, which comes from Accenture in Melbourne, Australia, and CumulusCI which comes from Salesforce.org. Each of these packages have CLIs, or a whole set of CLIs, with which you manage deployments.
Slide 16
Let us talk more about what I call the Devops Command Server. I like to use that wording because I find the term “CI/CD server or tool” to be off-putting and intimidating. I like Command Server because it is more understandable.
This is where we get into the build-vs-buy issues. Here I am illustrating three of the most popular ways to automate Salesforce devops pipelines if you are in more of a build mindset.
The name Azure DevOps is slightly misleading, because you don’t need an Azure cloud account. In fact, Azure DevOps is a windows server that comes on an ISO. And most sites with Microsoft licensing agreements already have access to the ISO for free. Plus, the software is usable and should be familiar to most users with IT admin experience.
Jenkins has the distinction of being the oldest devops command server, so it has the biggest user base. Cloudbees, which is a massive devops services company based on Jenkins, supports Salesforce. That means there are some big Salesforce devops projects out there using Jenkins.
Finally, GitHub Actions is the latest kid on the block and is easy to use. There is widespread adoption and it is easy to get started. And, we are seeing it used on Salesforce devops projects.
Slide 17
If you are a big company, then you might want to figure out how to deploy centralized IT resources to developers. This way they don’t have to use cloud services for everything. The result can be much cheaper and more secure, and it lets you experiment with scripting resources and some of the other techniques I’ve mentioned.
Slide 18
I wanted to touch on those cloud services on the right-hand side of my system integration diagram.
By Salesforce Platform I’m referring to all of the different cloud services in the Salesforce family.
Public Clouds refers to other SaaS platforms, such as AWS, GCP or Microsoft Azure. It could also be another service provider besides Salesforce, like ServiceNow. Or it could be cloud versions of SAP or Oracle. I’ll talk more about app devops in a bit.
And the Public Devops Clouds are providers like GitHub, Cloudbees, AutoRABIT, Gearset, or Blue Canvas. These companies have public cloud native apps which interact with your Salesforce org on a cloud-to-cloud basis.
Slide 19
Remember how agile and devops were kissing behind the barn, well that is a reference how devops and agile management really depend on one another, and in fact devops can be seen as the engine of an agile management philosophy. And this whole thing is expressed as the Scaled Agile Framework or (SAFe), which is a set of organizational and workflow patterns for implementing agile practices at an enterprise scale.
To help this process along, more Japanese management techniques related to Kanban have emerged. These techniques are called value stream mapping and value stream management.
Focusing on process analysis, Value stream mapping starts at the app production team level. It assesses the way elements within a complex project interact to achieve an operational objective.
The value stream mapping process generates a graphical product called the value stream map. Planners create maps that describe current and desired value stream maps. Doing so provides a guide for process improvement. Also, these diagrams often seem a little strange. It is because they retain the same iconography and symbols used in the original Toyota value stream maps. I hope that we see more adaptations of these symbols for digital app production teams.
Value stream management is the “devops automation harness,” or the tools and practices that measure the ongoing dynamics of individual value streams. To become more widespread, value stream mapping and management tools need to be integrated directly into a Devops 3.0 platform.
Slide 20
With the automation tools available to developers today, there is much to be gained by focusing on developing automation techniques before getting busy with code.
I cannot overemphasize this point. You might spend days or weeks dreaming about how you would automate this or that. The thing is, if you don’t have a way to kick off some sort of automated process based on a code commit, or other developer action, you are wasting too much developer time.
If you cannot deploy a change set via a script running on a command server today, then you are wasting a team member’s time by making them log on to do the same task.
Remember how I mentioned how SFDX-CLI gives us a touch point between the LINUX scripting world and the Salesforce cloud? Well, another huge way to take advantage of cloud native computing and all its goodies is to use a GIT-compatible source code repository. So, you really need to get your code up on GitHub or another cloud-based repository. If you don’t want to go to the cloud, I recommend an open-source project called GITEA to host it privately.
Slide 21
OK – Let us go into a description of the top 5 vendor-based Salesforce devops solutions
Slide 22
When looking at these solutions here are the essential features you want to compare. Some of the packages use GIT behind the scenes automatically, while others require that you manage the repo yourself.
The next big thing to consider is how will this package fit into your release workflow, and can it be used to help optimize workflow in your app production teams. All of the packages have different approaches to specifying how to move code through a release process. Remember, you don’t necessarily have to replicate your current release management scheme. Some of the vendors might be able to help you optimize it.
Finally, a key competitive differentiator in the space is what does the service provider do with all that metadata that gets continuously ingested. Some of the service providers do important things like perform a census of metadata usage, create dependency maps, keep teams from overwriting metadata, and anticipate errors before deployment.
Slide 23
Most of the Top 5 vendors also offer integrated versions of these features which we have discussed.
Sometimes a vendor like Blue Canvas doesn’t support these features but has an open platform that encourages you to integrate other vendors’ solutions using CLIs and scripts.
Slide 24
I do not have a comparison matrix for these companies quite yet. Maybe next year!
I do have some basic characterizations I can share with you, though. AutoRABIT has a cloud-native server that ingests metadata and helps to intelligently manage the release process. They are focused on the financial services sector and support several AppExchange packages like nCino.
Blue Canvas, like I mentioned, doesn’t really address the top part of the SalesforceDevops.net industry map like the other four companies in the roundup. They have a cloud native architecture that uses the Go programming language that translates into better speed and flexibility using their cloud native app to ingest Salesforce metadata.
The Blue Canvas app also runs a command server, and manages branch-based development. Blue Canvas does not seem to need as much up-front consulting assistance compared to the others and can achieve a faster return on investment.
Copado is by far the market leader in the Salesforce devops platform space. They have the biggest global footprint, with a strong presence in Europe. Their architecture is to put everything onto Salesforce, so Copado is mainly a Salesforce-based application. They recently got a Series C investment that put them over a $1B dollar valuation.
Copado is acquiring companies like crazy. They also seem to have some of the biggest Salesforce accounts in their corner. They just put out a nice series of bite-sized videos on YouTube, so go check that out if you’re interested.
Maybe it is just because they are so big, but I’ve heard complaints about Copado’s price, complexity, and speed. Since they are getting so much big enterprise business right now, I’d be cautious using them for smaller projects.
Flosum is another Salesforce Platform-based solution, like Copado. They have a strong USA presence. They compete pretty much head-to-head with Copado, and I believe that Flosum addresses some of the complaints I just mentioned about Copado.
Gearset is one of my favorites because they do so much to educate the industry and teach about devops, and their founder Keven Boyle is a real leader in the industry. On the downside, I believe most of their support resources are in Cambridge, England, so that can present some issues for US-based users.
You can go to the Gearset website and get a whole bunch of basic information on Salesforce devops, so thumbs up for that. Gearset also has a cloud native architecture, like AutoRABIT and Blue Canvas. I’ve heard about a fair number of consultants who use Gearset for org discovery and documentation, and that Gearset may be better for smaller projects.
Slide 25
As systems become more intertwined, a need for a holistic view I call APP DEVOPS is emerging. This seems like a natural evolution as the need for multi-cloud orchestration increases. Some Salesforce-compatible orchestration and command servers already offer limited multi-cloud capabilities.
Slide 26
Let us go through this architecture diagram starting from the bottom and going on up to the top. The bottom layer is hyperconverged infrastructure, which represents compute, storage, software, networking, and telecommunications resources required to build a giant cloud service like AWS.
The cloud service providers then resell those services by offering APIs to provision and use the hyperconverged architecture.
Application service providers then use the cloud service provider APIs to provision and operate cloud native applications running in the cloud.
Next is the Cloud Native Devops layer, which delivers cloud native apps to be run by application service providers.
Remember the DevOps eras from earlier? This is where DevOps 2.0 engineers assemble scripts, command servers, online repositories, and other resources that take source code artifacts and then create run-time artifacts. Those artifacts are then deployed into a cloud native application that uses cloud service providers APIs.
The next layer represents the cloud native apps. In this diagram it shows Salesforce and “other” applications. These applications offer programmatic and low-code application development capabilities that create enterprise apps.
The “other” applications could be big enterprise packages like SAP and Microsoft Dynamics 365. Sometimes these other applications are customer data platforms which help to round out a Salesforce 360 strategy.
The next layer, App Devops, is the technical and human interface between SaaS service providers and the digital objectives of the enterprise. Which is pretty much how we have been discussing Salesforce Devops.
So, ideally App Devops will give us ways to intelligently manage the flow of information and metadata between multiple high-level cloud native applications. After all, do you think we are going to have SAP Devops, Dynamics Devops, and Salesforce Devops forever? Eventually there will be a convergence.
Slide 27
In summary, I hope that I’ve shown you how building up expertise in Salesforce devops will help you get things done faster and more efficiently.
As you go forward, you will first be confronted with that build-vs-buy decision. If you have the chance, try to find teams who are starting out and get them to use build automation from the very beginning.
But, if you don’t have cloud native expertise, then the Salesforce-specific devops products from one of the Top 5 vendors may be a safer choice.
I’d like to thank the sponsors for producing PakDreamin 2021. See you next time!
