Difference between revisions of "Edge AI SDK/GenAIChatbot"

From ESS-WIKI
Jump to: navigation, search
Line 39: Line 39:
  
 
= How To =
 
= How To =
−
*'''Download SLM Models from GenAI Studio'''  
+
== '''Download SLM Models from GenAI Studio''' ==
−
[[File:Genai-studio-1.png|left|900x600px|]]
+
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
−
[[File:Genai-studio-2.png|left|900x600px|]]
+
[[File:Genai-studio-1.png|none|900x600px|]]
−
[[File:Genai-studio-3.png|left|900x600px|]]
+
</div>
−
[[File:Genai-studio-4.png|left|900x600px|]]
+
*''Click on icon 1 to open the GenAI Studio Hub page.''
 +
*''At icon 2, enter the URL of your GenAI Studio.''
 +
*''Click on icon 3 to save your settings.''
 +
*''Once the configuration is successful, a notification will appear as shown at icon 4.''
 +
*''Icon 5 displays a list of all models supported by GenAI Studio.''
 +
*''Click on icon 6 to download the desired model.''
  
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Genai-studio-2.png|none|900x600px|]]
 +
</div>
 +
*''Once the download begins, a loading icon will be displayed.''
  
−
*'''Crate a new chatbot assistant with RAG'''
+
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Genai-studio-3.png|none|900x600px|]]
 +
</div>
 +
*''After the download is complete, a notification will appear, and the download icon will change to a completion icon.''
  
−
*'''Speak'''
+
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Genai-studio-4.png|none|900x600px|]]
 +
</div>
 +
*''After the model has finished downloading, click on icon 1 to create a new conversation.''
 +
*''Next, click on icon 2 to select the model you just downloaded and start using it.''
 +
<br clear="all" />
  
−
**Audio to Azure AI Speech API
+
== '''Crate a new chatbot assistant with RAG''' ==
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-1.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-1.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-2.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-3.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-4.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-5.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-6.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-7.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-8.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-9.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-10.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-11.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-12.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Rag-13.png|none|900x600px|]]
 +
</div>
 +
<br clear="all" />
 +
 
 +
== '''Speak''' ==
 +
*Audio to Azure AI Speech API
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Stt-tts-1.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Stt-tts-2.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Stt-tts-3.png|none|900x600px|]]
 +
</div>
 +
<div style="display:inline-block; box-shadow:0 4px 16px rgba(0,0,0,0.15); border-radius:12px;">
 +
[[File:Stt-tts-4.png|none|900x600px|]]
 +
</div>
 +
<br clear="all" />
  
 
== Configuration ==  
 
== Configuration ==  
Line 56: Line 132:
  
 
   
 
   
−
 
−
 
−
 
 
[[Category:Editor]]
 
[[Category:Editor]]

Revision as of 09:58, 20 May 2025

Brief of GenAI Chatbot

Introduction

GenAI Chatbot is a next-generation conversational AI assistant built on the OLLAMA architecture, supporting all models compatible with OLLAMA. Designed for seamless integration with GenAI Studio, it allows users to directly import models that have been fine-tuned within GenAI Studio, enabling easy deployment and immediate use of custom models in the chatbot. At its core, GenAI Chatbot utilizes efficient Small Language Models (SLMs) to provide natural, context-aware interactions. The chatbot features advanced capabilities, including audio processing (Speech-to-Text [STT], Text-to-Speech [TTS]), Retrieval-Augmented Generation (RAG), and an embedded vector database (VectorDB), all within a flexible configuration suitable for diverse application scenarios. It is currently optimized for embedded platforms such as NVIDIA Jetson Orin Nano and Jetson Orin AGX.

SLM Chatbot

The GenAI Chatbot utilizes a Small Language Model (SLM) as its core engine. SLMs provide efficient language understanding and generation capabilities, delivering fast and accurate responses with low computational overhead. This allows the chatbot to operate smoothly on both edge devices and server environments.

Integrate with GenAI Studio

GenAI Chatbot is fully integrated with GenAI Studio, the AI LLM model management platform. Within GenAI Studio, users can:

  • Perform model fine-tuning.
  • Access a variety of model quantization methods for deployment on different hardware.
  • Convert models easily to the formats required by various platforms.
  • Customize and personalize AI models with ease.

This integration significantly simplifies the process of deploying and customizing chatbots for both enterprises and developers.

Audio

  • Speech-to-Text (STT)

GenAI Chatbot supports natural voice input, automatically converting user speech queries into text for processing. The system can flexibly integrate with a variety of Speech-to-Text (STT) engines and providers. By default, it uses the Whisper-base model and supports integration with OpenAI API, Web Browser API, Deepgram API, and Azure AI Speech API, making it easy to connect to different STT services as needed.

  • Text-to-Speech (TTS)

Chatbot responses can be automatically converted to natural-sounding speech using Text-to-Speech (TTS). Users are free to choose different voices and languages. The system supports the Web API by default and can be integrated with OpenAI API, Transformers, ElevenLabs, and Azure AI Speech API, providing flexible options for TTS services.

RAG

GenAI Chatbot supports Retrieval-Augmented Generation (RAG), combining large language models (LLMs) with external knowledge bases to greatly improve the accuracy and depth of answers. This makes it suitable for knowledge-based Q&A, document search, FAQ systems, and various information retrieval scenarios.

  • Embedded
    • The system converts various documents, knowledge sources, or custom content into vector (embedding) representations to enable efficient semantic search.
    • By default, the sentence-transformers/all-MiniLM-L6-v2 model is used for embeddings, balancing computational efficiency and semantic understanding.
    • Supports automatic embedding of multilingual documents and multiple file formats (such as txt, pdf, markdown, html, etc.).
    • Custom embedding models are supported to flexibly meet the needs of different professional domains.
  • VecorDB
    • The generated embeddings are stored in a VectorDB (vector database), supporting large-scale, efficient semantic search and matching.
    • By default, ChromaDB is used as the vector database, offering real-time retrieval and ease of deployment and management.
    • The system can be extended to support other mainstream VectorDBs, such as Pinecone, Weaviate, Qdrant, etc., making it suitable for enterprise or cloud-based deployments.
    • Built-in features such as deduplication, sharding, and data weighting ensure query efficiency and result quality.

How To

Download SLM Models from GenAI Studio

Genai-studio-1.png
  • Click on icon 1 to open the GenAI Studio Hub page.
  • At icon 2, enter the URL of your GenAI Studio.
  • Click on icon 3 to save your settings.
  • Once the configuration is successful, a notification will appear as shown at icon 4.
  • Icon 5 displays a list of all models supported by GenAI Studio.
  • Click on icon 6 to download the desired model.
Genai-studio-2.png
  • Once the download begins, a loading icon will be displayed.
Genai-studio-3.png
  • After the download is complete, a notification will appear, and the download icon will change to a completion icon.
Genai-studio-4.png
  • After the model has finished downloading, click on icon 1 to create a new conversation.
  • Next, click on icon 2 to select the model you just downloaded and start using it.


Crate a new chatbot assistant with RAG

Rag-1.png
Rag-1.png
Rag-2.png
Rag-3.png
Rag-4.png
Rag-5.png
Rag-6.png
Rag-7.png
Rag-8.png
Rag-9.png
Rag-10.png
Rag-11.png
Rag-12.png
Rag-13.png


Speak

  • Audio to Azure AI Speech API
Stt-tts-1.png
Stt-tts-2.png
Stt-tts-3.png
Stt-tts-4.png


Configuration

- Set Azure TTS Service
Retrieved from "https://ess-wiki.advantech.com.tw/wiki/index.php?title=Edge_AI_SDK/GenAIChatbot&oldid=39167"