Why Make a Chatbot Service?

In 2023, conversational AI products like ChatGPT exploded in popularity as they became smarter, more capable and flexible. How has conversational AI gotten so much more robust over the course of just a year? The answer lies in the exponential growth of the core large language models (LLMs) that power these products. By training an LLM on text data, you can teach that model how to reliably take in a new text input and respond in a helpful, human-like way.

LLMs are a game changer in a wide range of applications, but critically for conversational AI, they make it possible for tech to interact naturally with users while also performing tasks like answering questions or recording data. This technology is the major facelift that chatbots have needed for a long time. Gone are the days of the sad, pre-programmed chatbots that can never seem to understand what you need unless you type in the exact keywords they’re looking for: these robots can now actually help users just like a human would. It’s much easier to get an LLM set up, connect it to your company’s database to inform its answers, and teach it how to send any data they’ve collected back to you.

Furthermore, while chatbots have traditionally only seen use in customer service, the chatbots that are utilizing LLMs are so powerful that they’re now seeing use in all kinds of applications. Here at PressW, we’ve personally built chatbots to:

• Talk to prospective customers to get their contact information and tell them more about the company’s product

• Help company employees effortlessly find and summarize information that would otherwise be lost in a sea of files in their company’s databases

• Search the web for new developments in the company’s field, create reports based on their findings, and quickly break down any important takeaways for employees

With little to no tuning, we’ve seen chatbots perform all of these tasks far faster than most humans. Now, we’d like to give you a step-by-step guide on how to create your own chatbot and provide it with its own specialized tools so that you can build a perfect LLM-powered assistant to automate your own workflows.

The Core Pieces of a Chatbot Service

There are a few key pieces that you’ll need to build in order to transform an LLM into a fully-fledged chatbot:

• Prompts

• Tools and integrations

• A user interface and hosting setup

We’ll start by breaking these pieces down at a higher level first, and then get into the nitty-gritty of what tools we generally use to make these.

Prompting

“Prompting” is the act of adding extra instructions to an LLM’s input in order to get your ideal response. There are a few different levels of prompting that you’ll need to think about:

System Level Prompting

If you’re working with your own LLM or a model that doesn’t already have a system level prompt, you’ll have to create one for your LLM. The system level prompt acts as the general ruleset that defines how your chatbot behaves, and will be passed to your LLM along with your chatbot’s inputs every time you call it. This is the place to define your chatbot’s tone of voice, what areas of expertise it should talk about (or avoid), and any other broad rules it should always abide by. Perhaps the most common one we see here is simply “you are a helpful chatbot” - however, you can add as much as you’d like here, from broad to extremely specific instructions.

Keep in mind that these rules will always apply to your chatbot, so if you want its behavior to change depending on factors besides the user’s input, you should define that in a different part of the LLM’s prompting.

If you’re using an open-source LLM or a third party’s model like OpenAI’s ChatGPT, it will likely already have a system prompt baked into the model. Don’t worry - you can still change how these services behave for your use case by adding those instructions to your other prompts like the conversation control flow prompts.

Conversation Control Flow Prompting

If you want your chatbot to actively pursue an agenda beyond answering the user’s questions, such as discussing certain topics with users or asking them for information that you’d like to collect, you’ll have to write prompts asking the LLM to do so. For example, let’s make a basic chatbot to ask for the user’s contact information. We’d prompt the LLM like so:

System Level Prompt:
"You are a helpful chatbot that always responds to the human user politely."

Conversation Control Prompt:
"You are trying to collect the user's contact information. 
If the user just inputted a piece of information that's not in the "Collected Information" JSON yet, please add it to the JSON and thank the user.
Then, please ask the user to provide a piece of their information to fill in the first empty slot of the JSON. 
Output your response as a JSON with your message to the human user under the key "output_message" and the collected information JSON under the key "collected_information"."

Collected Information JSON:
{
	"name": "",
	"email": ""
}

Chat History JSON:
[{
   "ai_chatbot": "Hello! Could I get your name please?"
}]

Human User Input:
"My name is Cam Rider."

Our LLM should output a JSON with its response to the user as well as a constantly updating list of the information it’s collected for us.

{
	"output_message": "Thank you Cam! Could I please get your email?",
	"collected_information": {
		"name": "Cam Rider",
		"email": ""
	}
}

By updating the chatbot’s prompt with the newest breakdown of the information collected from the user, we can easily configure it to ask for new information instead of having to use a completely different model.

Tool Usage Prompting

While optional, chatbot “tools” allow us to power up our chatbots by giving them the ability to trigger other code. Let’s see an example of what that looks like by telling our chatbot how to get information from the internet using a function to search the web and get the text from the top result:

System Level Prompt:
"You are a helpful chatbot that always responds to the human user politely."

Conversation Control Prompt:
"You are trying to answer the human user's questions as accurately as possible.
If you'd like to answer the user's question, you can output a JSON with the key "response" and a corresponding value with your message back to the user."

Tool Usage Prompt:
"If the human user asks you a question, you can use the online lookup tool to look it up online by outputting a JSON with the key "lookup" and a corresponding value containing what you'd like to look up."

Chat History JSON:
[]

Human User Input:
"Who is the CEO of PressW?"

Now, when we trigger our LLM, we’ll get an output depending on what the LLM thinks is most appropriate:

{
	"lookup": "PressW CEO"
}

We catch that the LLM is looking for an answer online instead of responding to the user, and after calling code to search the internet for us, we send the result back to the LLM:

Chat History JSON:
[{
	"human_user": "Who is the CEO of PressW?"
	"lookup": "PressW CEO"
}]

Lookup Result:
"Tarun Thummala, PressW CEO"

Now, the LLM is confident in its answer and can respond back to the user:

{
	"response": "The CEO of PressW is Tarun Thummala. Do you have any other questions that I can help you with?"
}

All in all, this is just scraping the surface of how capable LLMs are and how you can use them to make your chatbot a superstar. By creating intelligent prompts, changing them to fit your use case, and processing the LLM’s output to inform your next steps, you’ll have a chatbot automating your company’s workflows in no time.

Tools and Integrations

This is a good time to segue into how you should build the actual integrations and tools you’ll likely want to give your chatbot. While these aren’t technically necessary, keep in mind that no matter how much prompt engineering you do, the base LLM is only capable of receiving a message and spitting out a response. Creating integrations and tools will allow your chatbot to effectively directly interact with your code.

Tools

Tools are built for your LLM to be able to use in response to a user’s input. When the LLM thinks its more appropriate to call one of the tools you give it instead of responding with a message for the user, it’ll write a message based on its prompting to indicate that it wants to call your tool. After catching that the LLM’s message is meant to trigger your tool, your code can then take in the LLM’s message and do whatever you’d like with it. Sometimes, you’ll want to pass the results of your code back to the LLM so it can make an informed decision with the information it received from calling your tool. On the other hand, you may want to manually send a response back to the user instead.

Let’s take a look at a twist on the task we took a crack at earlier, getting a user’s information. If we have a lot of questions for the user, and we want to give the chatbot additional information about the question and what answers are acceptable, we can create a tool to properly capture data and update the chatbot’s prompting each time it does:

System Level Prompt:
"You are a helpful chatbot that always responds to the human user politely."

Conversation Control Prompt:
"You are trying to collect the user's contact information. 
Please ask the user for their name."

Tool Usage Prompt:
"If the human user has provided you with both their first and last name, please call the Data Capture tool by outputting a JSON with the keys ("first_name", "last_name") with their corresponding values.

Chat History JSON:
[{
   "ai_chatbot": "Hello! Could I get your name please?"
}]

Human User Input:
"My name is Cam Rider."

Let’s see what the LLM outputs:

{
	"first_name": Cam,
	"last_name": Rider
}

Since the LLM output a JSON instead of a message for the user, we should have code that catches that and sends the information we’ve collected off to our company’s database. After that, all we have to do is update the LLM’s prompt before putting the ball back in its court:

System Level Prompt:
"You are a helpful chatbot that always responds to the human user politely."

Conversation Control Prompt:
"You are trying to collect the user's contact information. 
After thanking them for providing their name, please ask the user for their email."

Tool Usage Prompt:
"If the human user has provided you with a valid email, please call the Data Capture tool by outputting a JSON with the key ("email") with its corresponding values.

Chat History JSON:
[{
   "ai_chatbot": "Hello! Could I get your name please?"
}]

Human User Input:
"My name is Cam Rider."

Alternatively, if we want to send a curated message straight back to the user instead and skip a second LLM call, we can update the LLM’s prompt without calling it again. We’ll manually output our message to the user, and when the user responds, we’ll send the user’s message to the LLM like we normally would.

Integrations

You can also accomplish many workflows by adding pre- and post-processing steps before sending a message to an LLM. One of the most well-known LLM integrations is document retrieval (or retrieval-augmented generation, aka RAG), where you insert additional information into the LLM’s prompt based off the user’s message. After hooking up our code to our company’s database and putting the files we want the chatbot to reference in a vectorstore (a fancy appendix for computers), we look for excerpts that are semantically similar to the user’s input and insert them into the LLM’s prompt:

System Level Prompt:
"You are a helpful chatbot that always responds to the human user politely."

Human User Input:
"What was our latest lunch and learn about?"

Related Context:
"Lunch and Learn Schedule:

February: Parsing Through Tables and PDFs for LLMs by Cam Rider"

This lets our LLM answer the question right away without calling a tool:

"The latest lunch and learn was conducted by Cam Rider, who presented on how to parse tables and PDFs for large language models."

As you can see, both tools and integrations make it possible to do so much more with chatbots.

User Interface and Hosting

If you just want to build a chatbot for your personal use and chat with it using local development tools, you absolutely can. However, if you want to give other people access to your chatbot’s functionalities while still maintaining control of how it works, you’ll have to host your chatbot yourself and give it a proper user interface. While these may seem like trivial tasks at first, they can have a big effect on how quickly your chatbot responds and how a user will engage with your chatbot.

Before you finish building your chatbot and start creating a basic user interface and cloud setup for it, consider:

• Do you want your chatbot to be able to reference past parts of the conversation?

• Do you want users to be able to upload documents for the chatbot to analyze?

• Should users be able to regenerate responses if they want a slightly different answer or aren’t satisfied with the response your chatbot initially generated?

If you want your chatbot to be able to do any of these, or support any workflows outside of sending messages back and forth, you’ll have to create the functionality to support them in the frontend, the cloud, and the backend.

Conclusion

While that’s just a high-level view of what pieces you might want your chatbot to have, we hope that helps you get a sense of where to start! If you want to try out making a chatbot but don’t have the time or technical knowledge to do so, check out our article on creating a basic chatbot with OpenAI’s GPTs. We’ll also be releasing an in-depth look at the pros and cons of using an OpenAI GPT vs your own custom chatbot soon, so keep an eye out!

Appendix - What We Use to Build Chatbots

• LLMs: While we’re constantly on the lookout for new LLMs to incorporate into our products, we generally use OpenAI’s models to power our products. Currently, they’re the most powerful models on the market, and data that goes through their API won’t ever be sent to third party companies or used to train their models. Azure also hosts alternative versions of OpenAI’s models where your data will be protected by Microsoft’s even stricter privacy policies.

• Prompt building: LangChain is our package of choice for connecting to third party LLMs and building LLM pipelines. In addition to simplifying system and tool prompting, it has a lot of other great tools that we constantly use to make our chatbots better. For example:

  • LCEL, which allows you to stream LLM responses token-by-token to your user interface so users see responses faster.

  • Langsmith, which allows us to save LLM results for data analysis and see intermediate results if we build pipelines where LLMs call tools or other LLMs.

• Integrations and tools: We’ll break these down into a few categories:

  • Document downloading and parsing: Unstructured is a great starting point since they let you download files from other services like Google Drive and break them up based on document type and format, all in one step.

  • Vector stores: We commonly use Chroma, Qdrant, and FAISS.

  • Web search: We usually default to Google Serper, but LangChain supports a few other options like SerpAPI as well.

  • Custom tools: We usually build our backends and our custom tools using Python.

• Server hosting: We like using Google Cloud the most, but AWS and Azure are both perfectly acceptable alternatives. Azure places a higher priority on cybersecurity, but their user interfaces can be harder to navigate.

• User interface: We make all of our frontends with NextJS and Typescript.