Difference between revisions of "Edge AI SDK/GenAIChatbot"
Eric.liang (talk | contribs) |
Eric.liang (talk | contribs) |
||
| Line 38: | Line 38: | ||
| | ||
| − | == Create a new | + | |
| + | == Create a new Knowledge == | ||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-1.png|none|900x600px|Rag-1.png]] | [[File:Rag-1.png|none|900x600px|Rag-1.png]] | ||
</div> | </div> | ||
| − | + | <span style="font-size:larger;">1. Client the Workspace from the left menu.<br/> 2. Go to the Knowledge<br/> 3. Click the + icon to add a new knowledge </span> | |
| − | + | ||
| − | + | <span style="font-size:larger;">Here are sample files: * '[[Media:10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf|tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf]] , * ''''[[Media:10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt|10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt]]''' </span> | |
| + | |||
| + | | ||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-2.png|none|900x600px|Rag-2.png]] | [[File:Rag-2.png|none|900x600px|Rag-2.png]] | ||
</div> | </div> | ||
| − | + | <span style="font-size:larger;">1. Enter the title.<br/> 2. Enter the goal or description.<br/> 3. Click the "Create Knowledge" button to finish.</span> | |
| − | + | ||
| − | + | | |
| − | |||
| − | |||
| − | |||
| − | |||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
| + | | ||
| + | </div> <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-4.png|none|900x600px|Rag-4.png]] | [[File:Rag-4.png|none|900x600px|Rag-4.png]] | ||
</div> | </div> | ||
| − | + | <span style="font-size:larger;">1. Click the "+" icon<br/> 2. Click the "Upload files" to upload files. </span> | |
| − | + | ||
| + | | ||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-5.png|none|900x600px|Rag-5.png]] | [[File:Rag-5.png|none|900x600px|Rag-5.png]] | ||
</div> | </div> | ||
| − | *<span style="font-size:larger;"> | + | *<span style="font-size:larger;">Select the files you want to use.</span> |
| + | |||
| + | | ||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-6.png|none|900x600px|Rag-6.png]] | [[File:Rag-6.png|none|900x600px|Rag-6.png]] | ||
</div> | </div> | ||
| − | *<span style="font-size:larger;"> | + | *<span style="font-size:larger;">After the upload, the files will be displayed and a success notification will appear.</span> |
| + | |||
| + | | ||
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | <div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;"> | ||
[[File:Rag-7.png|none|900x600px|Rag-7.png]] | [[File:Rag-7.png|none|900x600px|Rag-7.png]] | ||
Revision as of 10:06, 29 May 2025
Brief of GenAI Chatbot
Contents
Introduction
GenAI Chatbot is a next-generation conversational AI assistant built on the OLLAMA architecture, supporting all models compatible with OLLAMA. Designed for seamless integration with GenAI Studio, it allows users to directly import models that have been fine-tuned within GenAI Studio, enabling easy deployment and immediate use of custom models in the chatbot. At its core, GenAI Chatbot utilizes efficient Small Language Models (SLMs) to provide natural, context-aware interactions. The chatbot features advanced capabilities, including audio processing (Speech-to-Text [STT], Text-to-Speech [TTS]), Retrieval-Augmented Generation (RAG), and an embedded vector database (VectorDB), all within a flexible configuration suitable for diverse application scenarios. It is currently optimized for embedded platforms such as NVIDIA Jetson Orin Nano and Jetson Orin AGX.
How To
Download SLM Models from GenAI Studio
- Click GenAI Studio Hub from the left menu.
- Enter the URL of your GenAI Studio.
- Click the "Save"
- Once the configuration is successful, a notification will appear as shown at icon 4.
- Displays a list of all models supported by GenAI Studio.
- Click on icon 6 to download the desired model.
- After the download is complete, a notification will appear, and the icon will change to a completion icon.
After the model has downloaded, you can select this LLM Model in new Chat
Create a new Knowledge
1. Client the Workspace from the left menu.
2. Go to the Knowledge
3. Click the + icon to add a new knowledge
Here are sample files: * 'tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf , * '10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt
1. Enter the title.
2. Enter the goal or description.
3. Click the "Create Knowledge" button to finish.
1. Click the "+" icon
2. Click the "Upload files" to upload files.
- Select the files you want to use.
- After the upload, the files will be displayed and a success notification will appear.
- After the Knowledge is created, click on icon 1 to go to the Models page.
- Click on icon 2, the "+" icon, to add a new knowledge model.
- On the add new model page, fill in and select the required fields shown in the red box: Title, Subtitle, Base Model, and System Prompt.
- Continuing from the previous page, click on icon 1 to select the Knowledge you just created, then click on icon 2 to save and create the model.
- After successful creation, a notification will appear at icon 1. Then, click on the model at icon 2 to enter the model chat.
- The chat window will display that the model in use is the Knowledge model you created.
- After starting the conversation, you will see that the model retrieves information from the Knowledge you created in its responses.
Configuring TTS with Azure AI Speech API
Create an Account on Azure AI Speech API
- Get started with Azure’s free account: new users receive $200 credit for 30 days and free access to popular services.
- AI Speech – Text-to-Speech: 500,000 neural characters per month for free accounts
2. Set Up the Azure Speech Service
- In the Azure Portal, click on "Create a resource".
- Click on icon 1: "AI + Machine Learning"
- Then, click on icon 2: "Speech"
- Click on “Start” at the position marked by the red box.
- Click "Create" and fill in the necessary details:
- Subscription: Choose your Azure subscription.
- Resource Group: Select an existing group or create a new one.
- Region: Choose a region close to your location.
- Name: Provide a unique name for your Speech resource.
- Pricing Tier: Select Free F0.
- Click on icon 2: "Review + create"
- After confirming the information, click the “Create” button.
- After entering the overview page, click on the name link of the resource you created under "Resource."
- In the left-hand menu, click on "Keys and Endpoint".
- Note down the Key1 or Key2 and the Endpoint URL; you'll need these to authenticate your API requests.
Setting Up on GenAI Chatbot
- Click the"Admin Panel."
- Click the Audio
- Select the "Text-to-Speech Engine" with "Azure AI Speech".
- Enter the Azure AI Speech API token in API Key.
- Click the "Save" icon.
- Finally, you will see a success notification.
Evaluate the Benchmark of Each Chatbot Response
- An information button is provided next to each response. Clicking it reveals detailed performance and inference statistics for that response, including token counts, processing speed, and computation time. This allows developers to monitor and optimize system performance in real time.
Example
Creating an Audio + RAG Chatbot for Medication Assistant
1. Configuring TTS with Azure AI Speech API
2. Create a Knowledge
1. Client the Workspace from the left menu.
2. Go to the Knowledge
3. Click the + icon to add a new knowledge
Here are sample files: * 'tial_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.pdf , * '10_Essential_Q&As_About_Over-the-Counter_(OTC)_Medication_Use.txt
1. Enter the title.
2. Enter the goal or description.
3. Click the "Create Knowledge" button to finish.
1. Click the "+" icon
2. Click the "Upload files" to upload files.
After the upload, the files will be displayed and a success notification will appear.
3. Create a Chatbot Assistant with RAG ( Knowledge )
1. Go to the Models tab.
2. Click the "+" icon, to add a new model.
On the add new model page, fill in and select the required fields shown in the red box:
* Title, * Subtitle, * Base Model, and *System Prompt.
Continuing from the previous page,
1. Click the "Select Knowledge" to select the Knowledge you just created,
2. Click the "Save & Create" to save and create.
After successful creation, a notification will appear at icon 1.
Then, click on the model ( icon 2 ) to enter the model chat. Start using the audio assistant.
1. Make sure the select model is the RAG model.
2. Click the mic icon to start voice mode.
3. While speaking, the system will show a listening status.
4. After speaking, your question will appear as text.
5. The Medical Chatbot Assistant will response and play the answer with audio.