[Feature] Add support for vllm as llm source (#1149)
This commit is contained in:
@@ -14,6 +14,7 @@ Embedchain comes with built-in support for various popular large language models
|
||||
<Card title="Cohere" href="#cohere"></Card>
|
||||
<Card title="Together" href="#together"></Card>
|
||||
<Card title="Ollama" href="#ollama"></Card>
|
||||
<Card title="vLLM" href="#vllm"></Card>
|
||||
<Card title="GPT4All" href="#gpt4all"></Card>
|
||||
<Card title="JinaChat" href="#jinachat"></Card>
|
||||
<Card title="Hugging Face" href="#hugging-face"></Card>
|
||||
@@ -393,6 +394,34 @@ llm:
|
||||
|
||||
</CodeGroup>
|
||||
|
||||
## Ollama
|
||||
|
||||
Setup vLLM by following instructions given in [their docs](https://docs.vllm.ai/en/latest/getting_started/installation.html).
|
||||
|
||||
<CodeGroup>
|
||||
|
||||
```python main.py
|
||||
import os
|
||||
from embedchain import App
|
||||
|
||||
# load llm configuration from config.yaml file
|
||||
app = App.from_config(config_path="config.yaml")
|
||||
```
|
||||
|
||||
```yaml config.yaml
|
||||
llm:
|
||||
provider: vllm
|
||||
config:
|
||||
model: 'meta-llama/Llama-2-70b-hf'
|
||||
temperature: 0.5
|
||||
top_p: 1
|
||||
top_k: 10
|
||||
stream: true
|
||||
trust_remote_code: true
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
|
||||
## GPT4ALL
|
||||
|
||||
Install related dependencies using the following command:
|
||||
@@ -515,7 +544,7 @@ app = App.from_config(config_path="config.yaml")
|
||||
|
||||
```yaml config.yaml
|
||||
llm:
|
||||
provider: huggingface
|
||||
provider: huggingface
|
||||
config:
|
||||
endpoint: https://api-inference.huggingface.co/models/gpt2 # replace with your personal endpoint
|
||||
```
|
||||
@@ -525,7 +554,7 @@ If your endpoint requires additional parameters, you can pass them in the `model
|
||||
|
||||
```
|
||||
llm:
|
||||
provider: huggingface
|
||||
provider: huggingface
|
||||
config:
|
||||
endpoint: <YOUR_ENDPOINT_URL_HERE>
|
||||
model_kwargs:
|
||||
|
||||
Reference in New Issue
Block a user