Introducing AI Gateway for Salesforce: Managed Access for OpenAI API

I am thrilled to announce the open-source project AI Gateway for Salesforce. The project, which is designed to let you safely dip your toes into the generative AI waters, incorporates OpenAI’s state-of-the-art text generation models within your Salesforce environment. This project recognizes the concerns of senior IT leaders about widespread use of ChatGPT within their organizations. AI Gateway for Salesforce aims to give IT leaders a controlled and monitored alternative to using ChatGPT for small developer teams.

This gateway enables Salesforce customers to generate text with OpenAI API directly within Salesforce. The prompts and answers are stored in Salesforce custom objects. A remarkably powerful tool, AI Gateway for Salesforce allows small teams to begin experimenting with prompt engineering in everyday productivity tasks. The project is still experimental and is not intended to be used by end users at this time.

What is AI Gateway for Salesforce?

AI Gateway for Salesforce uses a custom Salesforce object named Prompt__c to create and store prompts. The Prompt layout uses a Flow that performs OpenAI chat completions API using the gpt-4 and gpt-3.5-turbo models. The response from the API is then stored in the PromptAnswer__c object.

The project is a bare-bones implementation of the OpenAI Chat Completions API. It is intended to be a template for other organizations to experiment with OpenAI API. The OpenAI user interface is written as a very simple Salesforce Flow, so it should be easy for admins and developers to customize the user interface to meet their needs.

OpenAI Privacy and Data Security

Privacy and data security are crucial considerations for enterprise AI. If you are using OpenAI in your organization, then it is important to understand the OpenAI privacy policy and API data usage policies.

First, you should understand there is a difference between how OpenAI treats API customers versus ChatGPT customers. If you are saving chats in ChatGPT, then OpenAI will use your data for research. On the other hand, according to OpenAI’s promises, if you are using the API to communicate with OpenAI, then they won’t use your data for training. OpenAI does have a 30-day policy where API data is retained for safety and engineering purposes and then deleted.

While OpenAI says they follow this data regimen, it doesn’t mean they will not screw up or fail to abide by their own rules. And, OpenAI is completely domiciled in the United States, so I’m sure some international users aren’t happy about that.

It’s not perfect, and some enterprises will not tolerate OpenAI risk at any level. But OpenAI API has clear guidelines and policies on data handling, and it won’t be used for model training. So, if you can tolerate those terms, then AI Gateway for Salesforce allows you to send data to OpenAI servers without worrying about the data misuse that could happen on ChatGPT.

Personally Identifiable Information (PII) and OpenAI

Please do not put PII into your AI Gateway prompts or answers. Besides sending that information to OpenAI, it is recorded into the Prompt__c and PromptAnswer__c custom objects. You must use placeholders if you’re trying to generate personalized information.

Managing PII internally is a major challenge with AI apps. Salesforce recently said they will automatically manage PII masking in the upcoming AI Cloud.

Not everybody will be willing to wait for Sales and Service Cloud integrations from Salesforce. There is a new product in the AppExchange called GPTfy, and it has similarities with AI Gateway. But GPTfy uses a PII filter to keep your PII out of the OpenAI API calls.

Enterprise Concerns and Opportunities for Generative AI

Just because you can feel better about OpenAI not using your data, this doesn’t mean your AI problems are solved. There are plenty of other concerns when it comes to generative AI. As we learned when Salesforce launched its new AI Cloud, we still have a way to go using large language models (LLMs) in the enterprise. The risk for confident fabrications and other LLM mistakes is still too high. And foundational LLMs are incapable of explaining themselves. So, that means using a tool like AI Gateway for Salesforce throughout an enterprise without restrictions would be a terrible idea!

So, why did I put this tool out there if it is potentially risky? Every company has an individual or team that’s chomping at the bit to get their hands on OpenAI for Salesforce applications. I made AI Gateway to give small organizations or senior IT teams the ability to say Yes to those teams. Storing the prompt library in Salesforce gives teams an easy way to distribute and secure saved prompts. Using the OpenAI API keeps your data out of training. And keeping the answers in a database gives you an audit trail. Using AI Gateway can encourage the development of prompt engineering skills in Salesforce developers, which will be a critical capability in the coming months.

Generative AI Accelerates Software Projects

There is something special about the combination of generative AI and software engineering. Because of that, I especially encourage the use of AI Gateway among teams of developers and architects. The foundational model features of GPT-4 allows it to perform extraordinary feats of logic as a thought partner. And, by creating elaborate prompts with specific Salesforce language examples, it generates large blocks of code with minimal hallucinations or other mistakes.

And using generative AI in software engineering has multiple built-in safety features. All generated code must be inspected by a developer before it is put into a project. And Salesforce languages must be complied, which validates the accuracy of the code.

There is a Wall Street Journal article that quotes an MIT professor who said “I think there is a risk of accumulating lots of very shoddy code written by a machine.” Well, that works completely against my limited experience! As I already stated, there are built-in safety features. And it’s been my experience using GPT-4 to refactor code is very helpful in eliminating technical debt.

How to Get AI Gateway for Salesforce

Visit https://github.com/vkeenan/ai-gateway to get more information on installing and using AI Gateway for Salesforce. You can install AI Gateway for Salesforce into a new scratch org from the repo. Or you can host the unmanaged package in one of your developer or sandbox orgs. Use this link to install the unmanaged package:

https://login.salesforce.com/packaging/installPackage.apexp?p0=04tHs0000011h8l

AI Gateway has several, rather complicated post-installation setup steps involved to securely integrate with OpenAI API. So, please read the GitHub README file and the main VisualForce page in the Prompt Engineering app to get the details on the needed External Credential and Permission Set setup steps.

Using AI Gateway for Salesforce

Here is a quick overview of how to use AI Gateway for Salesforce.

Pick A Prompt

To use the AI Gateway after installation, access the Prompt Engineering app in the Salesforce application menu. Then click the Prompt tab to see a list of prompts.

Send A Prompt

Clicking on the prompt name puts you into a Flow called OpenAI Prompt. There are Parameter, Prompt, and System Prompt fields you can edit before sending it to OpenAI. Here are what the different fields mean.

• The Parameter field uses a JSON format to list variables that will be replaced in Prompt when executed. By being explicit how one may parameterize a prompt, I believe you can increase the utility of that prompt by others.
• The Prompt field is where you make your request. This is functionally equivalent to entering text into a ChatGPT prompt.
• The System message is used to restrict, enhance, and shape the responses generated by the Prompt message. This is where you use your skills to create long, elaborate prompts. And there are formal prompting techniques you can use to draft the System prompt. These advanced techniques include few-shot and chain-of-thought prompting. In the GitHub Repo and the sample prompts, I have included several examples of using elaborate system prompts for code generation.

In the “Market Research” prompt example, the Parameter field is filled out by entering “Cloud Backup Services” as the research topic.

View Prompt Answer

Unlike when using ChatGPT, there is no word-by-word feedback on the response with the Flow interface. AI Gateway for Salesforce waits until OpenAI is done generating text and then displays the result. It can take up to two minutes for the result to come back, and I have occasionally experienced timeouts. When the API call is finally returned, the result is displayed on the screen and saved in the PromptAnswer__c custom object.

Let the Games Begin

I had a lot of fun doing this project. My personal motivation was to have a safe place where I can put my frequently used prompts. Now I have a safe and secure place to put my prompts, and I can execute prompts in a simple user interface that saves the results. As a result, I am a more efficient prompt engineer.

AI Gateway for Salesforce is among the first of what will eventually be dozens or even hundreds of AI-infused apps to appear on the Salesforce AppExchange. The feelings of excitement about AI and the unprecedented popularity of OpenAI ChatGPT will continue to deliver entrepreneurial innovations. The challenge for Salesforce is to deliver their fully integrated vision of AI Cloud and the GPT updates to all the Salesforce Clouds on time.

I made AI Gateway for Salesforce thinking about the software development teams that can benefit from LLM technology. I believe developer and architect LLM access is the area that Salesforce customers can safely invest in LLM technology starting today. But AI Gateway isn’t just for developers. Let your imagination run wild! But please do that in a sandboxed environment without customer data. I can’t wait to hear what you are doing with it!