Edge AI SDK/GenAIChatbot

 

Introduction

GenAI Chatbot is a next-generation conversational AI assistant built on the OLLAMA architecture, supporting all models compatible with OLLAMA. Designed for seamless integration with GenAI Studio, it allows users to directly import models that have been fine-tuned within GenAI Studio, enabling easy deployment and immediate use of custom models in the chatbot. At its core, GenAI Chatbot utilizes efficient Small Language Models (SLMs) to provide natural, context-aware interactions. The chatbot features advanced capabilities, including audio processing (Speech-to-Text [STT], Text-to-Speech [TTS]), Retrieval-Augmented Generation (RAG), and an embedded vector database (VectorDB), all within a flexible configuration suitable for diverse application scenarios. It is currently optimized for embedded platforms such as NVIDIA Jetson Orin Nano and Jetson Orin AGX.

 

GenAI Chatbot

Platforms

Device	Hardware	OS	EAS	Note
AIR-030	32/64GB, SSD:128GB	JetPack 6.0	3.3.0	Quantization Type: Q4_K_M Max LLM Model:     32GB <= 34B  ,  64GB <= 70B
EPC-R7300	8GB, SSD: 128GB	JetPack 6.2	3.3.0	Quantization Type: Q4_K_M Max LLM Model: 8GB <= 8B

Function

3.3.0

• GenAI Chatbot
  • Small Language Model
  • Visual Language Model
• Audio:
  • STT
  • TTS
• Integrate with GenAI Studio
  • Download the SLM model from GenAI Studio
• RAG:
  • Embedded Model
  • Vector Database

 

How To

Download Models from Ollama

1. Go to the page https://www.ollama.com/search and click on the "Models" tab as shown in Icon 1.
2. Use the search bar shown in Icon 2 to find the model you want to download.
3. In Icon 3, locate the name of the model.

 

 

1. After clicking in, you'll see the name and version of the model you need to download.
2. Please note that the model size depends on your hardware resources.
3. Make sure your hardware matches the specifications shown in the table above.

 

 

1. Go back to GenAI-Chatbot and create new chat window, as shown in Icon 1.
2. Enter the model name you just saw into the search bar in the chat window, as shown in Icon 2.
3. Then, click the download button at the location marked in Icon 3.

 

 

1. After clicking the download button, a notification will pop up.

 

 

1. After the download is complete, a notification will appear confirming the completion.

 

 

1. Next, you'll be able to find the model you downloaded in the model selection menu.

Download SLM Models from GenAI Studio

1. Click GenAI Studio Hub from the left menu.
2. Enter the URL of your GenAI Studio.
3. Click the "Save" 
4. Once the configuration is successful, a notification will appear as shown at icon 4.
5. Displays a list of all models supported by GenAI Studio.
6. Click on icon 6 to download the desired model.

 

 

• After the download is complete, a notification will appear, and the icon will change to a completion icon.

 

After the model has downloaded, you can select this LLM Model in new Chat 

 

 

Create a new Knowledge

1. Client the Workspace from the left menu.
2. Go to the Knowledge
3. Click the + icon to add a new knowledge 

Here are sample files: * 'tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf , * '10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt  

 

1. Enter the title.
2. Enter the goal or description.
3. Click the "Create Knowledge" button to finish.

 

 

1. Click the "+" icon
2. Click the "Upload files" to upload files.  

 

• Select the files you want to use.

 

• After the upload, the files will be displayed and a success notification will appear.

 

Create a Chatbot Assistant with RAG ( Knowledge )

1. Go to the Models tab.
2. Click the "+" icon, to add a new model.  

 

 

On the add new model page, fill in and select the required fields shown in the red box:

• Title, * Subtitle, * Base Model, and *System Prompt.  

 

 

Continuing from the previous page,

1. Click the "Select Knowledge" to select the Knowledge you just created,

2. Click the "Save & Create" to save and create.  

 

 

After successful creation, a notification will appear at icon 1.

Then, click on the model ( icon 2 ) to enter the model chat. Start the Assistant chat.

 

• The chat window will display that the model in use is the Knowledge model you created.

• After starting the conversation, you will see that the model retrieves information from the Knowledge you created in its responses.

 

Configuring TTS with Azure AI Speech API

Create an Account on Azure AI Speech API

1. Create an Azure account

• Get started with Azure’s free account: new users receive $200 credit for 30 days and free access to popular services.

• AI Speech – Text-to-Speech: 500,000 neural characters per month for free accounts

2. Set Up the Azure Speech Service

• In the Azure Portal, click on "Create a resource".

• Click on icon 1: "AI + Machine Learning"
• Then, click on icon 2: "Speech"

• Click on “Start” at the position marked by the red box.

• Click "Create" and fill in the necessary details:
  • Subscription: Choose your Azure subscription.
  • Resource Group: Select an existing group or create a new one.
  • Region: Choose a region close to your location.
  • Name: Provide a unique name for your Speech resource.
  • Pricing Tier: Select Free F0.
  • Click on icon 2: "Review + create"

• After confirming the information, click the “Create” button.

• After entering the overview page, click on the name link of the resource you created under "Resource."

• In the left-hand menu, click on "Keys and Endpoint".
• Note down the Key1 or Key2 and the Endpoint URL; you'll need these to authenticate your API requests.

 

Setup the Azue Text-to-Speec in GenAI Chatbot

• Click the"Admin Panel."

 

1. Click the Audio
2. Select the "Text-to-Speech Engine" with "Azure AI Speech".

 

1. Enter the Azure AI Speech API token in API Key.
2. Click the "Save" icon.
3. Finally, you will see a success notification.

Evaluate the Benchmark of Each Chatbot Response

• An information button is provided next to each response. Clicking it reveals detailed performance and inference statistics for that response, including token counts, processing speed, and computation time. This allows developers to monitor and optimize system performance in real time.

 

Example

Creating an Audio + RAG Chatbot for Medication Assistant

1. Configuring TTS with Azure AI Speech API

(1). Create an Account on Azure AI Speech API

(2). Setting Up on GenAI Chatbot

 

 

 

 

4. Start a voice chatbot assistant with RAG

 

1. Make sure the select model is the RAG model.
2. Click the mic icon to start voice mode.
3. While speaking, the system will show a listening status.
4. After speaking, your question will appear as text.
5. The Medical Chatbot Assistant will response and play the answer with audio.