Website Chatbot Workflow

This workflow allows you to crawl a website and turn it into a RAG-based agent. This allows you to produce a chatbot that can answer questions from your website.

For more detailed information about what this workflow is doing, see the crawl-to-rag documentation in GitHub.

Creating the workflow

For general information on creating workflows in ApertureDB Cloud see Creating and Deleting Workflows.

Configure your workflow by selecting:

• Which instance to use. If you only have one instance, there will be no options to select.
• Set the URL of your website where the crawl should start. For example, https://docs.aperturedata.io/. You can include multiple URLs with spaces between them.
• Choose an LLM provider from the list. Currently the choice is between OpenAI, Together, or Groq, but more options may be added.
• Enter an API key for the chosen LLM provider. To acquire an LLM key, go to: OpenAI, Together, Groq. For your security, we recommend that you create a new API key rather than reusing one.
• The chatbot will be secured using a token you specify to prevent anyone else from accessing your server. Choose a good password string and make sure that you remember it.

Once you have filled in the fields, click "Submit". Your workflow will be created and will start running.

See the results

The API

The RAG agent provides an API; see the documentation for more details. To connect to the API, use the URL for your server with /rag at the end. To authenticate, supply the bearer token in an authorization header:

Authorization: Bearer <token>

You can also explore the API using OpenAPI (sometimes referred to as "Swagger") by going to /rag/docs on the server.

The demo user interface

In addition to the API, we provide a simple HTML UI for demonstration purposes. When you first access it (again via /rag) it will look rather plain, just a title and a blue button. The blue button opens the chat window, but the first time you try to use it, you will instead be asked to log in using your token. Your token is then stored in a cookie, so you should not need to enter it again.

After logging in, the display will look a little more interesting. On the left will appear various debugging elements that show what is going on behind the scenes. This includes:

• Config information about your server
• The rewritten query. As part of the RAG agent, the LLM rewrites the user query in the context of the chat history. This not only fixes typos, but allows for clever tricks like pronoun resolution.
• The chat history. Every time you ask a question, the agent generates a summary of the conversational context. This is used in query rewriting.
• Context documents. A key part of RAG is that the agent retrieves documents from the database that are related to the (rewritten) query. The LLM uses these to provide an answer.

You can now open the chat window and type your question. The answer will appear in the chat window, and the debug information on the left will be updated.