klas/t6_mem0
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add streaming support for OpenSource App (#217)

Browse Source
This commit is contained in:
aaishikdutta
2023-07-11 01:38:22 +05:30
committed by GitHub
parent 65414af098
commit ae87dc4a6d
2 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -224,7 +224,7 @@ print(naval_chat_bot.chat("what did the author say about happiness?"))
### Stream Response ### Stream Response
- You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Currently only supports OpenAI model. - You can add config to your query method to stream responses like ChatGPT does. You would require a downstream handler to render the chunk in your desirable format. Supports both OpenAI model and OpenSourceApp.
- To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear. - To use this, instantiate a `QueryConfig` or `ChatConfig` object with `stream=True`. Then pass it to the `.chat()` or `.query()` method. The following example iterates through the chunks and prints them as they appear.

3
embedchain/embedchain.py
View File

@@ -350,7 +350,7 @@ class OpenSourceApp(EmbedChain):
print("Successfully loaded open source embedding model.") print("Successfully loaded open source embedding model.")
super().__init__(config) super().__init__(config)
def get_llm_model_answer(self, prompt): def get_llm_model_answer(self, prompt, config: ChatConfig):
from gpt4all import GPT4All from gpt4all import GPT4All
global gpt4all_model global gpt4all_model
@@ -358,6 +358,7 @@ class OpenSourceApp(EmbedChain):
gpt4all_model = GPT4All("orca-mini-3b.ggmlv3.q4_0.bin") gpt4all_model = GPT4All("orca-mini-3b.ggmlv3.q4_0.bin")
response = gpt4all_model.generate( response = gpt4all_model.generate(
prompt=prompt, prompt=prompt,
streaming=config.stream
) )
return response return response