testRigor Eases Salesforce Testing

testRigor is a Salesforce-specific online testing service that uses a human-based perception model to specify visual tests in plain English. testRigor uses an online testing engine designed to improve the long-term reliability of tests. testRigor’s tester-friendly Chrome extension and online service lets Salesforce platform owners create tests faster and more reliably than traditional Salesforce testing methods.

Salesforce testing is usually overlooked until the lack of testing shuts down a critical enterprise app. By moving test development and use to an earlier phase of app production, test automation is likely to reduce the cost and frequency of defect resolution. In fact, studies show defect resolution during production to be over 100 times more expensive than if the defect didn’t make it out of the devops pipeline.

Advanced testing techniques help to detect when a Salesforce org’s functionality unexpectedly changes due to a Salesforce version update. Testing also mitigates another common source of system failure when there are changes to an external system. To manage app delivery risk, Salesforce systems need a form of continuous testing to check for Salesforce UI changes and failures in external integrations.

To see how a Salesforce app production team could use a newer testing tool, SalesforceDevops.net picked testRigor as a leading example of next-generation Salesforce testing tools. In this review of testRigor, we will see why testRigor’s technology is next-generation, and then I’ll share my experiences using the tool in my own Salesforce developer pipeline.

Keenan Vision Scorecard Rating

testRigor gets a 4.0 Keenan Vision Rating score. Here is how the product scored on each component. Check this article for an explanation of the score.

• Ease of Use: 3
• Features: 4
• How Much Does Vern Recommend: 4
• Value: 5
• Support: 4

Keenan Vision Rating: 4.0

Pros: Practical tool for extending Salesforce testing capabilities; 2nd generation scripting overcomes

Cons: Needs better system integration tools

Salesforce Testing Activities

There are several types of Salesforce testing activities. These activities occur throughout Salesforce devops activities. Here is a summary of how I look at Salesforce testing activities during enterprise application delivery.

Here is a quick summary of these different phases of testing.

• Unit Testing – These tests perform automatic checks of Salesforce language files used to improve code reliability. Developers define, run and maintain unit tests.
• Integration Testing – Also known as QA or Functional Testing, integration tests perform high-level checks of system functionality.
• End-to-End Testing – These tests are extensive run-throughs of business process implementations, messaging, and related database modifications. A separate testing team creates End-to-end tests to avoid testing the “happy path” preferred by the development team.
• Smoke Testing – In the spirit of the American expression “where there’s smoke there is fire,” smoke tests quickly survey a system to signal if further tests are needed.
• Functional Monitoring – These tests, which are usually also smoke tests, are run periodically after a full deployment to check that interrelated systems and external services are running properly.

Each of these phases consist of a suite of tests. And some individual tests may be in more than one suite. For example, some integration tests may be suitable for use in smoke testing or functional monitoring.

Salesforce Testing is Different

Most Salesforce testing activities are about what is happening on the screen from a human perspective. But the way the Salesforce streams HTML and CSS code to the browser greatly complicates the job of an automatic visual testing service.

Here are some of the reasons why the way Salesforce streams HTML and CSS breaks DOM-based testing.

• Developers have very little control over the Document Object Model (DOM) used in Salesforce HTML and CSS, which is frequently a reference point for traditional testing tools.
• The use of a Shadow DOM by Salesforce Lightning Web Components further complicates the use of testing tools.

Because of these difficulties, there is a market for testing service providers who employ groups of manual testers who work with English-like descriptions of what to do on the screen, use their human visual perception to perform tasks, and then manually record the results of each test.

Selenium Tests Are Unstable

To avoid the use of manual testers, test automation is required. Until recently, Salesforce test automation systems still relied on DOM coordinates for test execution. This is because despite being unstable, Salesforce DOM-based testing is still used for integration and end-to-end tests performed during an app’s development phase.

Older test automation activities often use the Selenium browser automation project. Selenium scripts use references to the XPATH or DOM location of specific elements.

Creating, running, and maintaining Selenium scripts is generally considered an expert-level activity. This is because the scripts depend on exact specifications of element names and location paths. For example, Selenium scripts must exactly specify the name of an HTML element within the current DOM.

DOM-based testing does work during an app’s development phase. But, most DOM-based visual tests will stop working the next time Salesforce changes its HTML and CSS stream. This is unfortunate because DOM-based integration tests created during development cannot be reliably reused for Salesforce smoke testing and functional monitoring.

When Salesforce renders a screen, Salesforce changes the actual CSS and HTML streamed to the browser almost continuously. This happens at least three times a year with the normal Salesforce release cycle. And it also happens throughout the year as Salesforce rolls out visually-imperceptible updates to their HTML and CSS.

• 

• 

• 

• 

testRigor: What You See Is What You Get

The system uses specification-based tests plus several other novel approaches to Salesforce visual testing in its second-generation, DOM-less testing solution. Instead of relying on an accurate recording of DOM-based interactions to play back a specific script, testRigor uses a cloud native app that emulates user interactions with Salesforce. It then uses machine learning and advanced techniques to respond to screen elements like how a user would perceive them.

For example, here is a line from a testRigor script I generated for this review.

click “Products” roughly below “Mark Stage as Complete”

An equivalent way to locate this element on the Opportunity record layout would be to use this JavaScript expression. It refers to the HTML “A” element which needs to be clicked for the test to proceed.

document.querySelector(“#header_2073\\:0 > a”)

The “header_2073” portion of this code documents the element’s place in a long list of similar elements. The number and order of HTML elements is almost certain to change in a new release of Salesforce, even if the visual appearance does not change.

By using specification-based testing to overcome DOM oddities, testRigor compares favorably to other behavior or specification-based testing systems in the cloud native world such as the open source project Cucumber.

testRigor Features

Here is a quick rundown of the leading testRigor features.

• Mobile Friendly — Mobile apps in Salesforce Experiences are supported
• Reusable Rules — Frequently used testing sequences, such as “Create new Lead” are easily created as macros for later reuse
• Test Data Generation — Unique data values may be generated for use in data entry forms and retained for later use in the test script
• Pipeline Integration — testRigor has a public API for test suite execution and results monitoring
• Email Testing — Email delivery may be validated when sent to a special domain
• SMS and Phone Testing — Using a Twilio account lets testRigor test SMS delivery
• API Testing — testRigor can test external APIs directly

Using testRigor

testRigor is a cloud native application that works like a modern B2B SaaS app. Accordingly, users log onto the the portal to manage test suites, manage individual tests, and debug the execution of new test scripts. 

After I signed up for the services at https://testrigor.com, the first step was to manually add all of testRigor’s IP addresses to the authorized IP address list for the org. testRigor should make this process a little easier by putting those IP addresses into their new AppExchange package. After setting up the security and authentication the next step is to access testRigor’s page in the Google Chrome Web Store (link) and install the testRigor Chrome extension.

Test Definition

Setting up a new test was simple and could be an entry-level skill in a Salesforce app production team. To start a new definition I first logged onto the Salesforce org I wanted to test. I used testRigor on Salesforce production and sandbox orgs.

To start a new test I clicked on the testRigor extension and then started a new recording. I next performed a business-level function, which in my case was creating a new opportunity and quote. When I finished, I clicked back on the testRigor Chrome extension to stop the recording. That causes a testRigor window to pop up where I can edit, save, or discard the new script.

For new definitions, I clicked on the Save button to store the new script as a test the testRigor cloud. Next, I switched my browser to the testRigor app and began to edit the script. This is a required step because testRigor will often record superfluous mouse activities, such as hovering over a button.

After saving a new test, I ran the test over and over from the testRigor app. Each time I updated my script to correct an error or misinterpretation. I then checked the results of the tests, and repeated until the test sequence was satisfactory. 

Making an End-to-End Test

For this review, I went through the process of creating a single end-to-end test that incorporated external system integrations and used email delivery. I did this on a production org so that I could reuse this test for a smoke test or functional monitoring.

It took a few customer support experiences for me to feel comfortable using the testRigor scripting system, but I eventually got the feel for it. Going through the iterative process of fixing errors in my script was a bit tiresome, but making new tests really does seem like a trainable skill for a worker starting a career in Salesforce development.

testRigor in a Devops Pipeline

testRigor provides a crude, albeit effective method for incorporation test suites into devops pipelines. For each test suite, testRigor provides a secure REST API call to launch a test suite, execute a retest, and check the status of a submitted test.

I clicked on the “CI/CD Integration” menu option to access the HTTP function calls and saw a BASH script with the appropriate CURL commands to access the REST API. To actually use this code requires manual integration of a script in a devops pipeline sequence. It would be nice if testRigor provided a CLI or perhaps an SFDX-CLI plugin to provide a similar level of functionality.

Conclusion: testRigor Eases Test Creation and Maintenance

Salesforce testing is currently limited to popular Salesforce AppExchange packages or vertical industry packages. Most enterprise users have yet to use testing to increase the reliability of their pipelines. This is because Salesforce tests made using traditional methods, like with the Selenium web browser automation project. This often causes a test to execute improperly when a new version of Salesforce is released. So, to overcome this problem manual testing is still used heavily in the Salesforce ecosystem.

testRigor elevates Salesforce test automation from figuring out the DOM to a more human-like semantic model. By using an English-like screen action specification language, platform owners who use testRigor may create more tests and use them in more productive ways. testRigor helps managers run more smoke tests and functional monitoring tests. This all helps enterprises increase their release cadance and confidence, which is a primary goal of Salesforce devops.

Check out the Video Review!