119 lines
3.5 KiB
Plaintext
119 lines
3.5 KiB
Plaintext
---
|
|
title: Multimodal Support
|
|
description: Integrate images into your interactions with Mem0
|
|
icon: "image"
|
|
iconType: "solid"
|
|
---
|
|
|
|
Mem0 extends its capabilities beyond text by supporting multimodal data, including images. With this feature, users can seamlessly integrate images into their interactions—allowing Mem0 to extract relevant information from visual content and enrich the memory system.
|
|
|
|
## How It Works
|
|
|
|
When a user submits an image, Mem0 processes it to extract textual information and other pertinent details. These details are then added to the user's memory, enhancing the system's ability to understand and recall visual inputs.
|
|
|
|
<CodeGroup>
|
|
```python Code
|
|
from mem0 import MemoryClient
|
|
|
|
# Initialize the MemoryClient with your API key
|
|
client = MemoryClient(api_key="your_api_key_here")
|
|
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": "Hi, my name is Alice."
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "Nice to meet you, Alice! What do you like to eat?"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": {
|
|
"type": "image_url",
|
|
"image_url": {
|
|
"url": "https://www.superhealthykids.com/wp-content/uploads/2021/10/best-veggie-pizza-featured-image-square-2.jpg"
|
|
}
|
|
}
|
|
},
|
|
]
|
|
|
|
# Calling the add method to ingest messages into the memory system
|
|
client.add(messages, user_id="alice")
|
|
```
|
|
|
|
```json Output
|
|
{
|
|
"results": [
|
|
{
|
|
"memory": "Name is Alice",
|
|
"event": "ADD",
|
|
"id": "7ae113a3-3cb5-46e9-b6f7-486c36391847"
|
|
},
|
|
{
|
|
"memory": "Likes large pizza with toppings including cherry tomatoes, black olives, green spinach, yellow bell peppers, diced ham, and sliced mushrooms",
|
|
"event": "ADD",
|
|
"id": "56545065-7dee-4acf-8bf2-a5b2535aabb3"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Image Integration Methods
|
|
|
|
Mem0 supports incorporating images into user interactions using two primary methods: by providing an image URL or by using a Base64-encoded image. The examples below demonstrate both approaches.
|
|
|
|
## 1. Using an Image URL (Recommended)
|
|
|
|
You can include an image by providing its direct URL. This method is simple and efficient for online images.
|
|
|
|
```python {2, 5-13}
|
|
# Define the image URL
|
|
image_url = "https://www.superhealthykids.com/wp-content/uploads/2021/10/best-veggie-pizza-featured-image-square-2.jpg"
|
|
|
|
# Create the message dictionary with the image URL
|
|
image_message = {
|
|
"role": "user",
|
|
"content": {
|
|
"type": "image_url",
|
|
"image_url": {
|
|
"url": image_url
|
|
}
|
|
}
|
|
}
|
|
client.add([image_message], user_id="alice")
|
|
```
|
|
|
|
## 2. Using Base64 Image Encoding for Local Files
|
|
|
|
For local images—or when embedding the image directly is preferable—you can use a Base64-encoded string.
|
|
```python
|
|
import base64
|
|
|
|
# Path to the image file
|
|
image_path = "path/to/your/image.jpg"
|
|
|
|
# Encode the image in Base64
|
|
with open(image_path, "rb") as image_file:
|
|
base64_image = base64.b64encode(image_file.read()).decode("utf-8")
|
|
|
|
# Create the message dictionary with the Base64-encoded image
|
|
image_message = {
|
|
"role": "user",
|
|
"content": {
|
|
"type": "image_url",
|
|
"image_url": {
|
|
"url": f"data:image/jpeg;base64,{base64_image}"
|
|
}
|
|
}
|
|
}
|
|
client.add([image_message], user_id="alice")
|
|
```
|
|
|
|
Using these methods, you can seamlessly incorporate images into your interactions, further enhancing Mem0's multimodal capabilities.
|
|
|
|
If you have any questions, please feel free to reach out to us using one of the following methods:
|
|
|
|
<Snippet file="get-help.mdx" />
|