1. Web
  2. Web APIs
  3. Prompt API
  4. Multimodal prompts

Multimodal prompts

The Prompt API accepts multimodal inputs, including image and audio content. This article looks at how to handle multimodal inputs in your app.

Specifying expected input types

To declare that you want to use image and/or audio inputs in your session, you need to include them in the expectedInputs option of the create() method:

js
return await LanguageModel.create({
  expectedInputs: [
    { type: "text", languages: ["en"] },
    { type: "image" },
    { type: "audio" },
  ],
  expectedOutputs: [{ type: "text", languages: ["en"] }],
});

Providing multimodal input data

When providing multimodal inputs — for example in a prompt(), promptStreaming(), or append() call, or in the initialPrompts option of a create() call — you need to specify the correct data type in your input objects, and point to the data source in your value properties.

The following example passes three user inputs into a prompt() call, one of each type — text, image, and audio.

js
const response = await session.prompt([
  {
    role: "user",
    content: [
      { type: "text", value: "Describe my image and audio:" },
      { type: "image", value: imgElem },
      { type: "audio", value: audioBuffer },
    ],
  },
]);

What data types are accepted?

The Prompt API accepts several different formats for audio and image data:

Complete example

Let's look at a multimodal example, which allows you to select a local image file and have the API describe it for you.

The overall app structure is very similar to examples in previous guides. We won't walk through all the code exhaustively; instead, we'll just explain the most relevant parts. To check out the complete codebase in more detail, press the "Play" button in the rendered live output to open the full code in MDN Playground.

HTML

The file to describe is chosen using an <input type="file"> element. The API's image description is output to a <p> element. We also include an <img> element to display the chosen image.

html
<h1>Prompt API demo</h1>
<p>
  <strong>Focus the demo window, then press a key to start the app</strong>.
  This demo loads an image from your local filesystem, and then uses the Prompt
  API to describe it. First released in Chrome 148.
</p>

<h2>Input</h2>

<section>
  <form>
    <div>
      <label for="url">Choose image from your local files:</label>
      <input type="file" id="inputElem" accept="image/*" />
    </div>
    <button type="submit" id="submit">Submit query</button
    ><button type="button" id="abort">Abort query</button>
  </form>
  <img />
</section>

<h2>Output</h2>

<p class="prompt-output"></p>

JavaScript

We create a session variable to hold our session. Because using the API requires transient activation, we populate session inside a keydown event handler on the demo window. When the user focuses the demo and presses a key, we first check whether the API is supported; if not, we print a non-support message. If support is available, we check whether session already has a value assigned (we don't want to create a new session each time). If not, we run the init() function.

js
let session;
window.addEventListener("keydown", () => {
  if (!("LanguageModel" in window)) {
    promptOutput.innerHTML = `<span class="error">Your browser doesn't support the Prompt API!</span>`;
  } else if (!session) {
    init();
  }
});

The init() function generates a LanguageModel instance using the custom getSession() function.

Provided generation is successful, we assign the resulting LanguageModel instance to the session variable, print a success message to the output <p>, enable the <input> so that images can be chosen, and assign event listeners to update the UI when a new image is chosen in the file picker, and handle submission of a prompt query.

js
async function init() {
  session = await getSession();
  if (!session) return;
  promptOutput.textContent = `Session created.`;
  inputElem.disabled = false;
  inputElem.addEventListener("change", getImage);
  form.addEventListener("submit", handleSubmission);
}

The getSession() function works the same as in other examples (getSession() is explained here), except that we include image in our expectedInputs option as well as text:

js
return await LanguageModel.create({
  expectedInputs: [{ type: "text", languages: ["en"] }, { type: "image" }],
  expectedOutputs: [{ type: "text", languages: ["en"] }],
});

The getImage() function first checks whether a file is chosen in the <input type="file"> picker. If not, we print a suitable error to the output <p> and then return. At the end of the function body, we set the <img> element's src attribute to an object URL created from the file selected in the file picker, so that the image will be displayed in the UI.

Above that, we add two event listeners to the <img>:

  • If an error event fires on the <img>, we print a suitable error to the output <p> and then return.
  • If a load event fires on the <img>, we print a success message to the output <p> to tell the user the app is ready to query the image, and then enable the submit <button> so the query can be submitted.
js
function getImage() {
  const file = inputElem.files[0];
  if (!file) {
    promptOutput.innerHTML = `<span class="error">No file selected!</span>`;
    return;
  }

  imgElem.addEventListener("error", () => {
    promptOutput.innerHTML = `<span class="error">Image not loaded!</span>`;
    return;
  });

  imgElem.addEventListener("load", () => {
    promptOutput.innerHTML = "Image query ready to submit!";
    submitBtn.disabled = false;
  });

  imgElem.src = URL.createObjectURL(file);
}

The handleSubmission() function uses the same flow as previous examples to prompt the language model and retrieve its output (see explanation). The main difference is that in the prompt() call inputs, we first ask the API to describe the image, and then pass it a reference to the <img> element itself.

js
async function handleSubmission(e) {
  e.preventDefault();
  try {
    promptOutput.textContent = "...generating response...";
    submitBtn.disabled = true;
    abortBtn.disabled = false;

    const controller = new AbortController();
    abortBtn.addEventListener("click", () => {
      controller.abort("Query aborted by user.");
      submitBtn.disabled = false;
      abortBtn.disabled = true;
    });

    const response = await session.prompt(
      [
        {
          role: "user",
          content: [
            { type: "text", value: "Please describe the following image:" },
            { type: "image", value: imgElem },
          ],
        },
      ],
      {
        signal: controller.signal,
      },
    );

    promptOutput.textContent = response;

    submitBtn.disabled = false;
    abortBtn.disabled = true;
    console.log(`${session.contextUsage}/${session.contextWindow}`);
  } catch (e) {
    promptOutput.innerHTML = `<span class="error">${e}</span>`;
    submitBtn.disabled = true;
    abortBtn.disabled = false;
  }
}

Result

Focus the embedded demo window and press a key on your keyboard to start the app, then select an image using the file picker. When the image loads, press the "Submit query" button. After a short wait, the API's description of the image should appear in the output <p>.

See also

Help improve MDN

Learn how to contribute

This page was last modified on by MDN contributors.

View this page on GitHub • Report a problem with this content