Edge AI SDK/GenAIChatbot

Introduction

GenAI Chatbot is a next-generation conversational AI assistant designed to provide natural, context-aware interactions. At its core, it utilizes efficient Small Language Models (SLMs) and supports the direct import of models fine-tuned within GenAI Studio, enabling easy deployment and immediate use of custom models in the chatbot.

The chatbot features advanced capabilities, including:

• Audio processing (Speech-to-Text [STT] and Text-to-Speech [TTS]).
• Retrieval-Augmented Generation (RAG).
• An embedded vector database (VectorDB).
• Model Context Protocol (MCP) tools, which enable AI agents to perform actions.

GenAI Chatbot features a flexible configuration suitable for diverse application scenarios and is optimized for embedded platforms such as the Ryzen AI 8000, NVIDIA Jetson Orin Nano, and Jetson Orin AGX.

How To

Download Models from Ollama (Support Device: Nvidia)

• Go to the page https://www.ollama.com/search and click on the "Models" tab as shown in Icon 1.
• Use the search bar shown in Icon 2 to find the model you want to download.
• In Icon 3, locate the name of the model.

 

 

• After clicking in, you'll see the name and version of the model you need to download.
• Please note that the model size depends on your hardware resources.
• Make sure your hardware matches the specifications shown in the table above.

 

 

• Go back to GenAI-Chatbot and create new chat window, as shown in Icon 1.
• Enter the model name you just saw into the search bar in the chat window, as shown in Icon 2.
• Then, click the download button at the location marked in Icon 3.

 

 

• After clicking the download button, a notification will pop up.

 

 

• After the download is complete, a notification will appear confirming the completion.

 

 

• Next, you'll be able to find the model you downloaded in the model selection menu.

Delete Models on GenAI-Chatbot (Support Device: Nvidia)

• ⚠️ Delete Models For AMD

 

• Click on icon 1 "User" to enter the Admin Panel (icon 2).
• After entering "Settings," click on icon 3 "Models."
• Click the download icon (icon 4) on the right.

 

• After the window pops up, find the option to delete the model at the location of icon 1.
• After selecting the model you want to delete, click on icon 2 "Delete."

 

• Click "Confirm".

 

• A notification will pop up after a successful operation.

Download SLM Models from GenAI Studio

• Click GenAI Studio Hub from the left menu.
• Enter the URL of your GenAI Studio.
• Click the "Save" 
• Once the configuration is successful, a notification will appear as shown at icon 4.
• Displays a list of all models supported by GenAI Studio.
• Click on icon 6 to download the desired model.

 

 

• After the download is complete, a notification will appear, and the icon will change to a completion icon.

 

• After the model has downloaded, you can select this LLM Model in new Chat 

 

Create a new Knowledge

• Client the Workspace from the left menu.
• Go to the Knowledge.
• Click the + icon to add a new knowledge.

Here are sample files: * PDF: tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf , * Text: 10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt  

 

• Enter the title.
• Enter the goal or description.
• Click the "Create Knowledge" button to finish.

 

• Click the "+" icon.
• Click the "Upload files" to upload files.

 

• Select the files you want to use.

 

• After the upload, the files will be displayed and a success notification will appear.

 

Create a Chatbot Assistant with RAG ( Knowledge )

• Go to the Models tab.
• Click the "+" icon, to add a new model.

 

• On the add new model page, fill in and select the required fields shown in the red box:
  • Click the "+" icon, to add a new model.
  • * Title, * Subtitle, * Base Model, and * System Prompt.

 

Continuing from the previous page,

• Click the "Select Knowledge" to select the Knowledge you just created,
• Click the "Save & Create" to save and create.

 

• After successful creation, a notification will appear at icon 1.
• Then, click on the model ( icon 2 ) to enter the model chat. Start the Assistant chat.

 

• The chat window will display that the model in use is the Knowledge model you created.

 

• After starting the conversation, you will see that the model retrieves information from the Knowledge you created in its responses.

 

Configuring TTS with Azure AI Speech API

Create an Account on Azure AI Speech API

1. Create an Azure account

• Get started with Azure’s free account: new users receive $200 credit for 30 days and free access to popular services.

• AI Speech – Text-to-Speech: 500,000 neural characters per month for free accounts

2. Set Up the Azure Speech Service

• In the Azure Portal, click on "Create a resource".

• Click on icon 1: "AI + Machine Learning"
• Then, click on icon 2: "Speech"

• Click on “Start” at the position marked by the red box.

• Click "Create" and fill in the necessary details:
  • Subscription: Choose your Azure subscription.
  • Resource Group: Select an existing group or create a new one.
  • Region: Choose a region close to your location.
  • Name: Provide a unique name for your Speech resource.
  • Pricing Tier: Select Free F0.
  • Click on icon 2: "Review + create"

• After confirming the information, click the “Create” button.

• After entering the overview page, click on the name link of the resource you created under "Resource."

• In the left-hand menu, click on "Keys and Endpoint".
• Note down the Key1 or Key2 and the Endpoint URL; you'll need these to authenticate your API requests.

 

Setup the Azue Text-to-Speec in GenAI Chatbot

• Click the"Admin Panel."

 

• Click the Audio
• Select the "Text-to-Speech Engine" with "Azure AI Speech".

 

• Enter the Azure AI Speech API token in API Key.
• Click the "Save" icon.
• Finally, you will see a success notification.

Evaluate the Benchmark of Each Chatbot Response

• An information button is provided next to each response. Clicking it reveals detailed performance and inference statistics for that response, including token counts, processing speed, and computation time. This allows developers to monitor and optimize system performance in real time.

Using MCP tools

1. Verify the default MCP server in the GenAI-Chatbot tools

• Click User.
• Select Admin Panel.
• Open the Tools section.
• Under Manage Tool Server, ensure that file-utils / mcp-system-info are listed.
• Click Save to apply the changes.

2. Open a new chat and select the tools

• Select New Chat.
• Click the “+” button in the chat window.
• Choose the tool to open, e.g., system-info.
• Verify that the tool icon appears next to the “+” button, with the number of icons corresponding to the number of selected tools.

3. Use the tool to obtain system information

• Example: Ask the LLM model → “What is the current memory usage?”

4. Review the MCP server APIs and available functions

• MCP - System-Info-Server: http://localhost:23952/mcp-system-info/docs

Using GenAI Assistant Mode

1. Configure Agent Settings

• Click User.
• Select Admin Panel.
• Open the Agent section.
• Select model.
• Select tools.
• Click Save to apply the changes.
• Click Test button.

2. Start GenAI Assistant Mode

• The GenAI Assistant Mode will appear, allowing you to invoke tools and start a conversation.

 

Change Suggest content

• Change Suggested Content on the Agent Page.

Example

Creating an Audio + RAG Chatbot for Medication Assistant

1. Configuring TTS with Azure AI Speech API

(1). Create an Account on Azure AI Speech API

(2). Setting Up on GenAI Chatbot

 

 

 

4. Start a voice chatbot assistant with RAG

 

1. Make sure the select model is the RAG model.
2. Click the mic icon to start voice mode.
3. While speaking, the system will show a listening status.
4. After speaking, your question will appear as text.
5. The Medical Chatbot Assistant will response and play the answer with audio.

Add new mcp server on the Genai-Chatbot 

Link