[Feature] Add support for vllm as llm source (#1149)

This commit is contained in:
Deshraj Yadav
2024-01-09 17:38:53 +05:30
committed by GitHub
parent 5f653e69ae
commit 0373fa231c
9 changed files with 111 additions and 15 deletions

View File

@@ -14,6 +14,7 @@ Embedchain comes with built-in support for various popular large language models
<Card title="Cohere" href="#cohere"></Card>
<Card title="Together" href="#together"></Card>
<Card title="Ollama" href="#ollama"></Card>
<Card title="vLLM" href="#vllm"></Card>
<Card title="GPT4All" href="#gpt4all"></Card>
<Card title="JinaChat" href="#jinachat"></Card>
<Card title="Hugging Face" href="#hugging-face"></Card>
@@ -393,6 +394,34 @@ llm:
</CodeGroup>
## Ollama
Setup vLLM by following instructions given in [their docs](https://docs.vllm.ai/en/latest/getting_started/installation.html).
<CodeGroup>
```python main.py
import os
from embedchain import App
# load llm configuration from config.yaml file
app = App.from_config(config_path="config.yaml")
```
```yaml config.yaml
llm:
provider: vllm
config:
model: 'meta-llama/Llama-2-70b-hf'
temperature: 0.5
top_p: 1
top_k: 10
stream: true
trust_remote_code: true
```
</CodeGroup>
## GPT4ALL
Install related dependencies using the following command:
@@ -515,7 +544,7 @@ app = App.from_config(config_path="config.yaml")
```yaml config.yaml
llm:
provider: huggingface
provider: huggingface
config:
endpoint: https://api-inference.huggingface.co/models/gpt2 # replace with your personal endpoint
```
@@ -525,7 +554,7 @@ If your endpoint requires additional parameters, you can pass them in the `model
```
llm:
provider: huggingface
provider: huggingface
config:
endpoint: <YOUR_ENDPOINT_URL_HERE>
model_kwargs: