Skip to main content

Files and Images in Agent messages

For AI agents: see llms.txt for the complete documentation index. Markdown versions are available by adding .md to a page URL or requesting Accept: text/markdown.

You can add images and files for the LLM to reference in the messages.

NOTE: Sending URLs to LLMs is much easier with the cloud backend, since it has publicly available storage URLs. To develop locally you can use ngrok or similar to proxy the traffic.

Example code:

Running the example​

git clone https://github.com/convex/agent.git
cd agent
npm run setup
npm run dev

Sending an image by uploading first and generating asynchronously​

The standard approach is to:

  1. Upload the file to the database (uploadFile action). Note: this can be in a regular action or in an httpAction, depending on what's more convenient.
  2. Send a message to the thread (submitFileQuestion action)
  3. Send the file to the LLM to generate / stream text asynchronously (generateResponse action)
  4. Query for the messages from the thread (listThreadMessages query)

Rationale:

It's better to submit a message in a mutation vs. an action because you can use an optimistic update on the client side to show the sent message immediately and have it disappear exactly when the message comes down in the query.

However, you can't save to file storage from a mutation, so the file needs to already exist (hence the fileId).

You can then asynchronously generate the response (with retries / etc) without the client waiting.

1: Saving the file​

import { storeFile } from "@convex-dev/agent";
import { components } from "./_generated/api";

const { file } = await storeFile(
  ctx,
  components.agent,
  new Blob([bytes], { type: mimeType }),
  {
    filename,
    sha256,
  },
);
const { fileId, url, storageId } = file;

2: Sending the message​

// in your mutation
const { filePart, imagePart } = await getFile(ctx, components.agent, fileId);
const { messageId } = await fileAgent.saveMessage(ctx, {
  threadId,
  message: {
    role: "user",
    content: [
      imagePart ?? filePart, // if it's an image, prefer that kind.
      { type: "text", text: "What is this image?" },
    ],
  },
  metadata: { fileIds: [fileId] }, // IMPORTANT: this tracks the file usage.
});

3: Generating the response & querying the responses​

This is done in the same way as text inputs.

// in an action
await thread.generateText({ promptMessageId: messageId });
// in a query
const messages = await agent.listMessages(ctx, { threadId, paginationOpts });

Inline saving approach​

You can also pass in an image / file direction when generating text, if you're in an action. Any image or file passed in the message argument will automatically be saved in file storage if it's larger than 64k, and a fileId will be saved to the message.

Example:

await thread.generateText({
  prompt: [
    {
      role: "user",
      content: [
        { type: "image", image: imageBytes, mediaType: "image/png" },
        { type: "text", text: "What is this image?" },
      ],
    },
  ],
});

Under the hood​

Saving to the files has 3 components:

  1. Saving to file storage (in your app, not in the component's storage). This means you can access it directly with the storageId and generate URLs.
  2. Saving a reference (the storageId) to the file in the component. This will automatically keep track of how many messages are referencing the file, so you can vacuum files that are no longer used (see files/vacuum.ts).
  3. Inserting a URL in place of the data in the message sent to the LLM, along with the media type and other metadata provided. It will be inferred if not provided in guessMimeType.

Can I just store the file myself and pass in a URL?​

Yes! You can always pass a URL in the place of an image or file to the LLM.

const storageId = await ctx.storage.store(blob);
const url = await ctx.storage.getUrl(storageId);

await thread.generateText({
  prompt: [
    {
      role: "user",
      content: [
        { type: "image", image: url, mediaType: blob.type },
        { type: "text", text: "What is this image?" },
      ],
    },
  ],
});

Timeout while downloading <url> from the provider​

When a message carries an image or file as a URL, the component stores the URL and hands it to the AI SDK. Providers that accept URLs for that media type (OpenAI does for image/*) download it themselves; for other providers the AI SDK downloads it inside your action before the request. If a provider error reports Timeout while downloading and names the image URL, the provider could not fetch that URL in time. The AI_APICallError request URL may be the provider's API endpoint; that alone does not identify which fetch timed out. Third-party hosts and CDNs can respond inconsistently, so the same image URL may work on a later attempt.

To avoid relying on that host during generation, fetch the image in an action and pass its bytes in the message. You can set the fetch timeout or add retries. The bytes are sent inline on this call. When the message is saved, images over 64 KiB are moved to Convex file storage, so later turns replay a storage URL. Either way only your own fetch, not the provider request, depends on your image host:

export const ask = internalAction({
  args: { threadId: v.string(), imageUrl: v.string(), prompt: v.string() },
  handler: async (ctx, { threadId, imageUrl, prompt }) => {
    const url = new URL(imageUrl);
    if (
      url.origin !== "https://images.example.com" ||
      url.username ||
      url.password
    ) {
      throw new Error("Untrusted image URL");
    }
    const response = await fetch(url, {
      signal: AbortSignal.timeout(10_000),
      redirect: "error",
    });
    if (!response.ok) throw new Error(`Fetch failed: ${response.status}`);
    const mediaType = response.headers.get("content-type");
    if (!mediaType?.startsWith("image/")) {
      throw new Error(`Not an image: ${mediaType}`);
    }
    const image = new Uint8Array(await response.arrayBuffer());
    const { thread } = await agent.continueThread(ctx, { threadId });
    await thread.generateText({
      prompt: [
        {
          role: "user",
          content: [
            { type: "image", image, mediaType },
            { type: "text", text: prompt },
          ],
        },
      ],
    });
  },
});

Replace images.example.com with a host your application trusts. Do not fetch arbitrary user-supplied URLs: a server that does so can be pointed at internal hosts. Apart from localhost URLs, which it inlines for local development, the component does not download URLs on your behalf.

Generating images​

There's an example in files/generateImage.ts that takes a prompt, generates an image with OpenAI's dall-e 2, then saves the image to a thread.

You can try it out with:

npx convex run files:generateImage:replyWithImage '{prompt: "make a picture of a cat" }'