Difference between revisions of "Edge AI SDK/GenAIChatbot"
From ESS-WIKI
| Line 65: | Line 65: | ||
[[File:Ollama-1.png|none|900x600px|Ollama-1.png]] | [[File:Ollama-1.png|none|900x600px|Ollama-1.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Go to the page https://www.ollama.com/search and click on the "Models" tab as shown in Icon 1.</span> | |
| − | + | *<span style="font-size:larger;">Use the search bar shown in Icon 2 to find the model you want to download.</span> | |
| − | + | *<span style="font-size:larger;">In Icon 3, locate the name of the model.</span> | |
| | ||
| Line 75: | Line 75: | ||
[[File:Ollama-2.png|none|900x600px|Ollama-2.png]] | [[File:Ollama-2.png|none|900x600px|Ollama-2.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">After clicking in, you'll see the name and version of the model you need to download.</span> | |
| − | + | *<span style="font-size:larger;">Please note that the model size depends on your hardware resources.</span> | |
| − | + | *<span style="font-size:larger;">Make sure your hardware matches the specifications shown in [[#Platforms|the table above]].</span> | |
| | ||
| Line 85: | Line 85: | ||
[[File:Ollama-3.png|none|900x600px|Ollama-3.png]] | [[File:Ollama-3.png|none|900x600px|Ollama-3.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Go back to GenAI-Chatbot and create new chat window, as shown in Icon 1.</span> | |
| − | + | *<span style="font-size:larger;">Enter the model name you just saw into the search bar in the chat window, as shown in Icon 2.</span> | |
| − | + | *<span style="font-size:larger;">Then, click the download button at the location marked in Icon 3.</span> | |
| | ||
| Line 95: | Line 95: | ||
[[File:Ollama-4.png|none|900x600px|Ollama-4.png]] | [[File:Ollama-4.png|none|900x600px|Ollama-4.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">After clicking the download button, a notification will pop up.</span> | |
| | ||
| Line 103: | Line 103: | ||
[[File:Ollama-5.png|none|900x600px|Ollama-5.png]] | [[File:Ollama-5.png|none|900x600px|Ollama-5.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">After the download is complete, a notification will appear confirming the completion.</span> | |
| | ||
| Line 111: | Line 111: | ||
[[File:Ollama-6.png|none|900x600px|Ollama-6.png]] | [[File:Ollama-6.png|none|900x600px|Ollama-6.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Next, you'll be able to find the model you downloaded in the model selection menu.</span> | |
== Download SLM Models from GenAI Studio == | == Download SLM Models from GenAI Studio == | ||
| Line 117: | Line 117: | ||
[[File:Genai-studio-1.png|none|900x600px|Genai-studio-1.png]] | [[File:Genai-studio-1.png|none|900x600px|Genai-studio-1.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Click GenAI Studio Hub from the left menu.</span> | |
| − | + | *<span style="font-size:larger;">Enter the URL of your GenAI Studio.</span> | |
| − | + | *<span style="font-size:larger;">Click the "Save" </span> | |
| − | + | *<span style="font-size:larger;">Once the configuration is successful, a notification will appear as shown at icon 4.</span> | |
| − | + | *<span style="font-size:larger;">Displays a list of all models supported by GenAI Studio.</span> | |
| − | + | *<span style="font-size:larger;">Click on icon 6 to download the desired model.</span> | |
| | ||
| Line 136: | Line 136: | ||
[[File:Genai-studio-4.png|none|900x600px|Genai-studio-4.png]] | [[File:Genai-studio-4.png|none|900x600px|Genai-studio-4.png]] | ||
</div> | </div> | ||
| − | <span style="font-size:larger;">After the model has downloaded, you can select this LLM Model in new Chat </span> | + | *<span style="font-size:larger;">After the model has downloaded, you can select this LLM Model in new Chat </span> |
| | ||
| Line 297: | Line 297: | ||
[[File:Stt-tts-2.png|none|900x600px|Stt-tts-2.png]] | [[File:Stt-tts-2.png|none|900x600px|Stt-tts-2.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Click the Audio</span> | |
| − | + | *<span style="font-size:larger;">Select the "Text-to-Speech Engine" with "Azure AI Speech".</span> | |
| | ||
| Line 304: | Line 304: | ||
[[File:Stt-tts-3.png|none|900x600px|Stt-tts-3.png]] | [[File:Stt-tts-3.png|none|900x600px|Stt-tts-3.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Enter the Azure AI Speech API token in API Key.</span> | |
| − | + | *<span style="font-size:larger;">Click the "Save" icon.</span> | |
| − | + | *<span style="font-size:larger;">Finally, you will see a success notification.</span> | |
== Evaluate the Benchmark of Each Chatbot Response == | == Evaluate the Benchmark of Each Chatbot Response == | ||
| Line 320: | Line 320: | ||
[[File:Delete-1.png|none|900x600px|Delete-1.png]] | [[File:Delete-1.png|none|900x600px|Delete-1.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Click on icon 1 "User" to enter the Admin Panel (icon 2).</span> | |
| − | + | *<span style="font-size:larger;">After entering "Settings," click on icon 3 "Models."</span> | |
| − | + | *<span style="font-size:larger;">Click the download icon (icon 4) on the right.</span> | |
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Delete-2.png|none|900x600px|Delete-2.png]] | [[File:Delete-2.png|none|900x600px|Delete-2.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">After the window pops up, find the option to delete the model at the location of icon 1.</span> | |
| − | + | *<span style="font-size:larger;">After selecting the model you want to delete, click on icon 2 "Delete."</span> | |
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Delete-3.png|none|900x600px|Delete-3.png]] | [[File:Delete-3.png|none|900x600px|Delete-3.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">Click "Confirm".</span> | |
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Delete-4.png|none|900x600px|Delete-4.png]] | [[File:Delete-4.png|none|900x600px|Delete-4.png]] | ||
</div> | </div> | ||
| − | + | *<span style="font-size:larger;">A notification will pop up after a successful operation.</span> | |
Revision as of 03:27, 11 June 2025
Contents
- 1 Introduction
- 2 GenAI Chatbot
- 3 How To
- 4 Example
Introduction
GenAI Chatbot is a next-generation conversational AI assistant built on the OLLAMA architecture, supporting all models compatible with OLLAMA. Designed for seamless integration with GenAI Studio, it allows users to directly import models that have been fine-tuned within GenAI Studio, enabling easy deployment and immediate use of custom models in the chatbot. At its core, GenAI Chatbot utilizes efficient Small Language Models (SLMs) to provide natural, context-aware interactions. The chatbot features advanced capabilities, including audio processing (Speech-to-Text [STT], Text-to-Speech [TTS]), Retrieval-Augmented Generation (RAG), and an embedded vector database (VectorDB), all within a flexible configuration suitable for diverse application scenarios. It is currently optimized for embedded platforms such as NVIDIA Jetson Orin Nano and Jetson Orin AGX.
GenAI Chatbot
| Version | Date | Edge AI / OS | Function | Note & Dependency |
|---|---|---|---|---|
|
3.3.0 |
Coming Soon | |||
|
|
|
How To
Download Models from Ollama
- Go to the page https://www.ollama.com/search and click on the "Models" tab as shown in Icon 1.
- Use the search bar shown in Icon 2 to find the model you want to download.
- In Icon 3, locate the name of the model.
- After clicking in, you'll see the name and version of the model you need to download.
- Please note that the model size depends on your hardware resources.
- Make sure your hardware matches the specifications shown in the table above.
- Go back to GenAI-Chatbot and create new chat window, as shown in Icon 1.
- Enter the model name you just saw into the search bar in the chat window, as shown in Icon 2.
- Then, click the download button at the location marked in Icon 3.
- After clicking the download button, a notification will pop up.
- After the download is complete, a notification will appear confirming the completion.
- Next, you'll be able to find the model you downloaded in the model selection menu.
Download SLM Models from GenAI Studio
- Click GenAI Studio Hub from the left menu.
- Enter the URL of your GenAI Studio.
- Click the "Save"
- Once the configuration is successful, a notification will appear as shown at icon 4.
- Displays a list of all models supported by GenAI Studio.
- Click on icon 6 to download the desired model.
- After the download is complete, a notification will appear, and the icon will change to a completion icon.
- After the model has downloaded, you can select this LLM Model in new Chat
Create a new Knowledge
1. Client the Workspace from the left menu.
2. Go to the Knowledge
3. Click the + icon to add a new knowledge
Here are sample files:
* PDF: tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf ,
* Text: 10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt
1. Enter the title.
2. Enter the goal or description.
3. Click the "Create Knowledge" button to finish.
1. Click the "+" icon
2. Click the "Upload files" to upload files.
- Select the files you want to use.
- After the upload, the files will be displayed and a success notification will appear.
Create a Chatbot Assistant with RAG ( Knowledge )
1. Go to the Models tab.
2. Click the "+" icon, to add a new model.
On the add new model page, fill in and select the required fields shown in the red box:
- Title, * Subtitle, * Base Model, and *System Prompt.
Continuing from the previous page,
1. Click the "Select Knowledge" to select the Knowledge you just created,
2. Click the "Save & Create" to save and create.
After successful creation, a notification will appear at icon 1.
Then, click on the model ( icon 2 ) to enter the model chat. Start the Assistant chat.
- The chat window will display that the model in use is the Knowledge model you created.
- After starting the conversation, you will see that the model retrieves information from the Knowledge you created in its responses.
Configuring TTS with Azure AI Speech API
Create an Account on Azure AI Speech API
- Get started with Azure’s free account: new users receive $200 credit for 30 days and free access to popular services.
- AI Speech – Text-to-Speech: 500,000 neural characters per month for free accounts
2. Set Up the Azure Speech Service
- In the Azure Portal, click on "Create a resource".
- Click on icon 1: "AI + Machine Learning"
- Then, click on icon 2: "Speech"
- Click on “Start” at the position marked by the red box.
- Click "Create" and fill in the necessary details:
- Subscription: Choose your Azure subscription.
- Resource Group: Select an existing group or create a new one.
- Region: Choose a region close to your location.
- Name: Provide a unique name for your Speech resource.
- Pricing Tier: Select Free F0.
- Click on icon 2: "Review + create"
- After confirming the information, click the “Create” button.
- After entering the overview page, click on the name link of the resource you created under "Resource."
- In the left-hand menu, click on "Keys and Endpoint".
- Note down the Key1 or Key2 and the Endpoint URL; you'll need these to authenticate your API requests.
Setup the Azue Text-to-Speec in GenAI Chatbot
- Click the"Admin Panel."
- Click the Audio
- Select the "Text-to-Speech Engine" with "Azure AI Speech".
- Enter the Azure AI Speech API token in API Key.
- Click the "Save" icon.
- Finally, you will see a success notification.
Evaluate the Benchmark of Each Chatbot Response
- An information button is provided next to each response. Clicking it reveals detailed performance and inference statistics for that response, including token counts, processing speed, and computation time. This allows developers to monitor and optimize system performance in real time.
Delete Models on GenAI-Chatbot
- Click on icon 1 "User" to enter the Admin Panel (icon 2).
- After entering "Settings," click on icon 3 "Models."
- Click the download icon (icon 4) on the right.
- After the window pops up, find the option to delete the model at the location of icon 1.
- After selecting the model you want to delete, click on icon 2 "Delete."
- Click "Confirm".
- A notification will pop up after a successful operation.
Example
Creating an Audio + RAG Chatbot for Medication Assistant
1. Configuring TTS with Azure AI Speech API
2. Create a Knowledge
3. Create a Chatbot Assistant with RAG ( Knowledge )
4. Start a voice chatbot assistant with RAG
1. Make sure the select model is the RAG model.
2. Click the mic icon to start voice mode.
3. While speaking, the system will show a listening status.
4. After speaking, your question will appear as text.
5. The Medical Chatbot Assistant will response and play the answer with audio.