ChatOpenAI

This will help you getting started with ChatOpenAI chat models. For detailed documentation of all ChatOpenAI features and configurations head to the API reference.

Overview

Integration details

Class	Package	Local	Serializable	PY support	Package downloads	Package latest
ChatOpenAI	@langchain/openai	❌	✅	✅

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Token usage	Logprobs
✅	✅	✅	✅	❌	❌	✅	✅	✅

Setup

• TODO: Update with relevant info.

To access ChatOpenAI models you’ll need to create a ChatOpenAI account, get an API key, and install the @langchain/openai integration package.

Credentials

• TODO: Update with relevant info.

Head to OpenAI’s website to sign up to ChatOpenAI and generate an API key. Once you’ve done this set the OPENAI_API_KEY environment variable:

export OPENAI_API_KEY="your-api-key"

If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:

```{=mdx}

```bash
# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

### Installation

The LangChain ChatOpenAI integration lives in the `@langchain/openai` package:

```{=mdx}

```bash npm2yarn
npm i @langchain/openai

## Instantiation

Now we can instantiate our model object and generate chat completions:

::: {.cell execution_count=19}
``` {.typescript .cell-code}
import { ChatOpenAI } from "@langchain/openai"

const llm = new ChatOpenAI({
    model: "gpt-4o",
    temperature: 0,
    maxTokens: undefined,
    timeout: undefined,
    maxRetries: 2,
    // other params...
})

:::

Invocation

const aiMsg = await llm.invoke([
  [
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
  ],
  ["human", "I love programming."],
]);
aiMsg;

AIMessage {
  "id": "chatcmpl-9qoATPJseCeF0xADWYOeVKgNKPqdJ",
  "content": "J'adore la programmation.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 5,
      "promptTokens": 31,
      "totalTokens": 36
    },
    "finish_reason": "stop",
    "system_fingerprint": "fp_4e2b2da518"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 31,
    "output_tokens": 5,
    "total_tokens": 36
  }
}

console.log(aiMsg.content);

J'adore la programmation.

Chaining

We can chain our model with a prompt template like so:

• TODO: Run cells so output can be seen.

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});

AIMessage {
  "id": "chatcmpl-9qoAUdg1rAZhsYnwirmpUlGRH8033",
  "content": "Ich liebe das Programmieren.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 6,
      "promptTokens": 26,
      "totalTokens": 32
    },
    "finish_reason": "stop",
    "system_fingerprint": "fp_4e2b2da518"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 26,
    "output_tokens": 6,
    "total_tokens": 32
  }
}

Multimodal messages

info

This feature is currently in preview. The message schema may change in future releases.

OpenAI supports interleaving images with text in input messages with their gpt-4-vision-preview. Here’s an example of how this looks:

import * as fs from "node:fs/promises";

import { ChatOpenAI } from "@langchain/openai";
import { HumanMessage } from "@langchain/core/messages";

const imageData = await fs.readFile("../../../../../examples/hotdog.jpg");
const chat = new ChatOpenAI({
  model: "gpt-4-vision-preview",
  maxTokens: 1024,
});
const message = new HumanMessage({
  content: [
    {
      type: "text",
      text: "What's in this image?",
    },
    {
      type: "image_url",
      image_url: {
        url: `data:image/jpeg;base64,${imageData.toString("base64")}`,
      },
    },
  ],
});

const res = await chat.invoke([message]);
console.log(res);

AIMessage {
  "id": "chatcmpl-9qoAVJVZOVYSAvRDpQXVnDF0xIStb",
  "content": "The image shows a classic hot dog, consisting of a grilled or steamed sausage served in the slit of a partially sliced bun. The sausage appears to have grill marks, indicating it may have been cooked on a grill. There are no visible condiments or additional toppings in this image; it's a simple and basic presentation of a hot dog.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 69,
      "promptTokens": 438,
      "totalTokens": 507
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 438,
    "output_tokens": 69,
    "total_tokens": 507
  }
}

const hostedImageMessage = new HumanMessage({
  content: [
    {
      type: "text",
      text: "What does this image say?",
    },
    {
      type: "image_url",
      image_url:
        "https://www.freecodecamp.org/news/content/images/2023/05/Screenshot-2023-05-29-at-5.40.38-PM.png",
    },
  ],
});
const res2 = await chat.invoke([hostedImageMessage]);
console.log(res2);

AIMessage {
  "id": "chatcmpl-9qoAZXqa3VGSRx4pY6frLAo7NbAqq",
  "content": "The image contains the text \"LangChain\" and features a graphic of a parrot to the left and two interlinked rings to the right of the text.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 32,
      "promptTokens": 778,
      "totalTokens": 810
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 778,
    "output_tokens": 32,
    "total_tokens": 810
  }
}

const lowDetailImage = new HumanMessage({
  content: [
    {
      type: "text",
      text: "Summarize the contents of this image.",
    },
    {
      type: "image_url",
      image_url: {
        url: "https://blog.langchain.dev/content/images/size/w1248/format/webp/2023/10/Screenshot-2023-10-03-at-4.55.29-PM.png",
        detail: "low",
      },
    },
  ],
});
const res3 = await chat.invoke([lowDetailImage]);
console.log(res3);

AIMessage {
  "id": "chatcmpl-9qoAb7AIDkgfO5VzJ8lY4h3Rxose7",
  "content": "This image appears to be a screenshot of a user interface for a digital assistant or search tool named \"WebLangChain,\" indicated to be powered by \"Twirly.\" The interface has a dark theme with a prompt that says, \"Ask me anything about anything!\" Below the prompt, there is an input field where users can type their queries. Suggested questions are provided below this field, including \"what is langchain?\", \"history of mesopotamia\", \"how to build a discord bot\", \"leonardo dicaprio girlfriend\", \"fun gift ideas for software engineers\", \"how does a prism separate light\", and \"what beer is best.\" These sample questions suggest the assistant can answer a wide range of topics. On the right side of the input box, there is a send button shaped like a blue speech bubble with a white arrow inside, indicating the action to submit a question.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 180,
      "promptTokens": 101,
      "totalTokens": 281
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 101,
    "output_tokens": 180,
    "total_tokens": 281
  }
}

Tool calling

OpenAI chat models support calling multiple functions to get all required data to answer a question. Here’s an example how a conversation turn with this functionality might look:

import { ChatOpenAI } from "@langchain/openai";
import { ToolMessage } from "@langchain/core/messages";

// Mocked out function, could be a database/API call in production
function getCurrentWeather(location: string, _unit?: string) {
  if (location.toLowerCase().includes("tokyo")) {
    return JSON.stringify({ location, temperature: "10", unit: "celsius" });
  } else if (location.toLowerCase().includes("san francisco")) {
    return JSON.stringify({
      location,
      temperature: "72",
      unit: "fahrenheit",
    });
  } else {
    return JSON.stringify({ location, temperature: "22", unit: "celsius" });
  }
}

// Bind function to the model as a tool
const chat = new ChatOpenAI({
  model: "gpt-3.5-turbo-1106",
  maxTokens: 128,
}).bind({
  tools: [
    {
      type: "function",
      function: {
        name: "get_current_weather",
        description: "Get the current weather in a given location",
        parameters: {
          type: "object",
          properties: {
            location: {
              type: "string",
              description: "The city and state, e.g. San Francisco, CA",
            },
            unit: { type: "string", enum: ["celsius", "fahrenheit"] },
          },
          required: ["location"],
        },
      },
    },
  ],
  tool_choice: "auto",
});

// Ask initial question that requires multiple tool calls
const res = await chat.invoke([
  ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
]);
console.log(res.tool_calls);

[
  {
    name: 'get_current_weather',
    args: { location: 'San Francisco, CA', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_XVerHEcCvaXpdPObWM4rLIkQ'
  },
  {
    name: 'get_current_weather',
    args: { location: 'Tokyo, Japan', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_p0QzmXrnfL9Jiw52WbzESoSs'
  },
  {
    name: 'get_current_weather',
    args: { location: 'Paris, France', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_6ka7piwYmAtvsFnwgXbfHtK6'
  }
]

// Format the results from calling the tool calls back to OpenAI as ToolMessages
const toolMessages = res.additional_kwargs.tool_calls?.map((toolCall) => {
  const toolCallResult = getCurrentWeather(
    JSON.parse(toolCall.function.arguments).location
  );
  return new ToolMessage({
    tool_call_id: toolCall.id,
    name: toolCall.function.name,
    content: toolCallResult,
  });
});

// Send the results back as the next step in the conversation
const finalResponse = await chat.invoke([
  ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
  res,
  ...(toolMessages ?? []),
]);

console.log(finalResponse);

AIMessage {
  "id": "chatcmpl-9qoAjNgsOegV4rkLkL0nYyXJEvwV0",
  "content": "The current weather in:\n- San Francisco, CA is 72°F\n- Tokyo, Japan is 10°C\n- Paris, France is 22°C",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 33,
      "promptTokens": 248,
      "totalTokens": 281
    },
    "finish_reason": "stop",
    "system_fingerprint": "fp_adbef9f124"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 248,
    "output_tokens": 33,
    "total_tokens": 281
  }
}

.withStructuredOutput({ ... })

You can also use the .withStructuredOutput({ ... }) method to coerce ChatOpenAI into returning a structured output.

The method allows for passing in either a Zod object, or a valid JSON schema (like what is returned from zodToJsonSchema).

Using the method is simple. Just define your LLM and call .withStructuredOutput({ ... }) on it, passing the desired schema.

Here is an example using a Zod schema and the functionCalling mode (default mode):

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";

const model = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4-turbo-preview",
});

const calculatorSchema = z.object({
  operation: z.enum(["add", "subtract", "multiply", "divide"]),
  number1: z.number(),
  number2: z.number(),
});

const modelWithStructuredOutput = model.withStructuredOutput(calculatorSchema);

const prompt = ChatPromptTemplate.fromMessages([
  ["system", "You are VERY bad at math and must always use a calculator."],
  ["human", "Please help me!! What is 2 + 2?"],
]);
const chain = prompt.pipe(modelWithStructuredOutput);
const result = await chain.invoke({});
console.log(result);

{ operation: 'add', number1: 2, number2: 2 }

You can also specify includeRaw to return the parsed and raw output in the result.

const includeRawModel = model.withStructuredOutput(calculatorSchema, {
  name: "calculator",
  includeRaw: true,
});

const includeRawChain = prompt.pipe(includeRawModel);
const includeRawResult = await includeRawChain.invoke({});
console.log(includeRawResult);

{
  raw: AIMessage {
    "id": "chatcmpl-9qoAl49McgKC5htc6M8fu4dGkavHn",
    "content": "",
    "additional_kwargs": {
      "tool_calls": [
        {
          "id": "call_zO1pNCvFXBFjXYpR7U3sC1Hq",
          "type": "function",
          "function": "[Object]"
        }
      ]
    },
    "response_metadata": {
      "tokenUsage": {
        "completionTokens": 15,
        "promptTokens": 93,
        "totalTokens": 108
      },
      "finish_reason": "stop"
    },
    "tool_calls": [
      {
        "name": "calculator",
        "args": {
          "operation": "add",
          "number1": 2,
          "number2": 2
        },
        "type": "tool_call",
        "id": "call_zO1pNCvFXBFjXYpR7U3sC1Hq"
      }
    ],
    "invalid_tool_calls": [],
    "usage_metadata": {
      "input_tokens": 93,
      "output_tokens": 15,
      "total_tokens": 108
    }
  },
  parsed: { operation: 'add', number1: 2, number2: 2 }
}

Additionally, you can pass in an OpenAI function definition or JSON schema directly:

info

If using jsonMode as the method you must include context in your prompt about the structured output you want. This must include the keyword: JSON.

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4-turbo-preview",
});

const calculatorSchema = {
  type: "object",
  properties: {
    operation: {
      type: "string",
      enum: ["add", "subtract", "multiply", "divide"],
    },
    number1: { type: "number" },
    number2: { type: "number" },
  },
  required: ["operation", "number1", "number2"],
};

// Default mode is "functionCalling"
const modelWithStructuredOutput = model.withStructuredOutput(calculatorSchema);

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are VERY bad at math and must always use a calculator.
Respond with a JSON object containing three keys:
'operation': the type of operation to execute, either 'add', 'subtract', 'multiply' or 'divide',
'number1': the first number to operate on,
'number2': the second number to operate on.
`,
  ],
  ["human", "Please help me!! What is 2 + 2?"],
]);
const chain = prompt.pipe(modelWithStructuredOutput);
const result = await chain.invoke({});
console.log(result);

{ number1: 2, number2: 2, operation: 'add' }

You can also specify ‘includeRaw’ to return the parsed and raw output in the result, as well as a “name” field to give the LLM additional context as to what you are generating.

const includeRawModel = model.withStructuredOutput(calculatorSchema, {
  name: "calculator",
  includeRaw: true,
  method: "jsonMode",
});

const includeRawChain = prompt.pipe(includeRawModel);
const includeRawResult = await includeRawChain.invoke({});
console.log(includeRawResult);

{
  raw: AIMessage {
    "id": "chatcmpl-9qoAnMHNXbmxETthVRVKR5txyRQ4s",
    "content": "{\n  \"operation\": \"add\",\n  \"number1\": 2,\n  \"number2\": 2\n}",
    "additional_kwargs": {},
    "response_metadata": {
      "tokenUsage": {
        "completionTokens": 25,
        "promptTokens": 91,
        "totalTokens": 116
      },
      "finish_reason": "stop"
    },
    "tool_calls": [],
    "invalid_tool_calls": [],
    "usage_metadata": {
      "input_tokens": 91,
      "output_tokens": 25,
      "total_tokens": 116
    }
  },
  parsed: { operation: 'add', number1: 2, number2: 2 }
}

Disabling parallel tool calls

If you have multiple tools bound to the model, but you’d only like for a single tool to be called at a time, you can pass the parallel_tool_calls call option to enable/disable this behavior. By default, parallel_tool_calls is set to true.

import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const model = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4o",
});

// Define your tools
const calculatorSchema = z
  .object({
    operation: z.enum(["add", "subtract", "multiply", "divide"]),
    number1: z.number(),
    number2: z.number(),
  })
  .describe("A tool to perform basic arithmetic operations");
const weatherSchema = z
  .object({
    city: z.string(),
  })
  .describe("A tool to get the weather in a city");

// Bind tools to the model
const modelWithTools = model.bindTools([
  {
    type: "function",
    function: {
      name: "calculator",
      description: calculatorSchema.description,
      parameters: zodToJsonSchema(calculatorSchema),
    },
  },
  {
    type: "function",
    function: {
      name: "weather",
      description: weatherSchema.description,
      parameters: zodToJsonSchema(weatherSchema),
    },
  },
]);

// Invoke the model with `parallel_tool_calls` set to `true`
const response = await modelWithTools.invoke(
  ["What is the weather in san francisco and what is 23716 times 27342?"],
  {
    parallel_tool_calls: true,
  }
);

We can see it called two tools:

console.log(response.tool_calls);

[
  {
    name: 'weather',
    args: { city: 'san francisco' },
    type: 'tool_call',
    id: 'call_aMhRLnzYEaotQjJODwi2i47B'
  },
  {
    name: 'calculator',
    args: { operation: 'multiply', number1: 23716, number2: 27342 },
    type: 'tool_call',
    id: 'call_DzAjEgoYfXj34groG4ED6W5w'
  }
]

Invoke the model with parallel_tool_calls set to false

const response2 = await modelWithTools.invoke(
  ["What is the weather in san francisco and what is 23716 times 27342?"],
  {
    parallel_tool_calls: false,
  }
);

We can see it called one tool

console.log(response2.tool_calls);

[
  {
    name: 'weather',
    args: { city: 'san francisco' },
    type: 'tool_call',
    id: 'call_U9r7WbJQCDoaJ8nFH7KUcF2n'
  }
]

Custom URLs

You can customize the base URL the SDK sends requests to by passing a configuration parameter like this:

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  temperature: 0.9,
  configuration: {
    baseURL: "https://your_custom_url.com",
  },
});

const message = await model.invoke("Hi there!");

You can also pass other ClientOptions parameters accepted by the official SDK.

If you are hosting on Azure OpenAI, see the dedicated page instead.

Calling fine-tuned models

You can call fine-tuned OpenAI models by passing in your corresponding modelName parameter.

This generally takes the form of ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}. For example:

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  temperature: 0.9,
  model: "ft:gpt-3.5-turbo-0613:{ORG_NAME}::{MODEL_ID}",
});

const message = await model.invoke("Hi there!");

Generation metadata

If you need additional information like logprobs or token usage, these will be returned directly in the .invoke response.

tip

Requires @langchain/core version >=0.1.48.

import { ChatOpenAI } from "@langchain/openai";

// See https://cookbook.openai.com/examples/using_logprobs for details
const model = new ChatOpenAI({
  logprobs: true,
  // topLogprobs: 5,
});

const responseMessage = await model.invoke("Hi there!");
console.dir(responseMessage.response_metadata.logprobs, { depth: null });

{
  content: [
    {
      token: 'Hello',
      logprob: -0.00047559434,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.00004429897,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.00003166338,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.00062582217,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -9.0883464e-7,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.12731916,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -5.5122365e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000049663133,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.000010564331,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

With callbacks

You can also use the callbacks system:

import { ChatOpenAI } from "@langchain/openai";

// See https://cookbook.openai.com/examples/using_logprobs for details
const model = new ChatOpenAI({
  logprobs: true,
  // topLogprobs: 5,
});

const result = await model.invoke("Hi there!", {
  callbacks: [
    {
      handleLLMEnd(output) {
        console.dir(output.generations[0][0].generationInfo.logprobs, {
          depth: null,
        });
      },
    },
  ],
});

{
  content: [
    {
      token: 'Hello',
      logprob: -0.0004585519,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.000049305523,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.000029517714,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.00073185476,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -9.0883464e-7,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.104538105,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -6.704273e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000052643223,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.000012590794,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

console.dir(result.response_metadata.logprobs, { depth: null });

{
  content: [
    {
      token: 'Hello',
      logprob: -0.0004585519,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.000049305523,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.000029517714,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.00073185476,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -9.0883464e-7,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.104538105,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -6.704273e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000052643223,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.000012590794,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

Streaming tokens

OpenAI supports streaming token counts via an opt-in call option. This can be set by passing { stream_options: { include_usage: true } }. Setting this call option will cause the model to return an additional chunk at the end of the stream, containing the token usage.

import type { AIMessageChunk } from "@langchain/core/messages";
import { ChatOpenAI } from "@langchain/openai";
import { concat } from "@langchain/core/utils/stream";

// Instantiate the model
const model = new ChatOpenAI();

const response = await model.stream("Hello, how are you?", {
  // Pass the stream options
  stream_options: {
    include_usage: true,
  },
});

// Iterate over the response, only saving the last chunk
let finalResult: AIMessageChunk | undefined;
for await (const chunk of response) {
  if (finalResult) {
    finalResult = concat(finalResult, chunk);
  } else {
    finalResult = chunk;
  }
}

console.log(finalResult?.usage_metadata);

{ input_tokens: 13, output_tokens: 39, total_tokens: 52 }

API reference

For detailed documentation of all ChatOpenAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html