LLM Config Deep-dive - AG2

LLM Configuration

In AG2, agents use LLMs as key components to understand and react. To configure an agent's access to LLMs, you can specify an llm_config argument in its constructor. For example, the following snippet shows a configuration that uses gpt-4.1:

import os
from autogen import LLMConfig

llm_config = LLMConfig(config_list={"api_type": "openai", "model": "gpt-5", "api_key": os.environ["OPENAI_API_KEY"]})

Warning

It is important to never commit secrets into your code, therefore we read the OpenAI API key from an environment variable.

This llm_config can then be passed to an agent's constructor to enable it to use the LLM.

import autogen

assistant = autogen.AssistantAgent(name="assistant", llm_config=llm_config)

Introduction to config_list

Different tasks may require different models, and the config_list allows specifying the different endpoints and configurations that are to be used. It is a list of dictionaries, each of which contains the following keys depending on the kind of endpoint being used:

Example:

[
    {
      "api_type": "openai",
      "model": "gpt-5",
      "api_key": os.environ['OPENAI_API_KEY']
    }
]

Example:

[
    {
      "model": "my-gpt-4.1-deployment",
      "api_type": "azure",
      "api_key": os.environ['AZURE_OPENAI_API_KEY'],
      "base_url": "https://ENDPOINT.openai.azure.com/",
      "api_version": "2025-01-01"
    }
]

Example:

[
    {
      "api_type": "openai",
      "model": "llama-7B",
      "base_url": "http://localhost:1234"
    }
]

Tip

These will create a model client which assumes an OpenAI API (or compatible) endpoint. To use custom model clients, see here.

OAI_CONFIG_LIST pattern

A common, useful pattern used is to define this config_list via JSON (specified as a file or an environment variable set to a JSON-formatted string) and then use the from_json method to load it:

llm_config = autogen.LLMConfig.from_json(
    env="OAI_CONFIG_LIST",  # Or path="path/to/config.json"
)

# Then, create the assistant agent with the config
assistant = autogen.AssistantAgent(name="assistant", llm_config=llm_config)

This can be helpful as it keeps all the configuration in one place across different projects or notebooks.

Why is it a list?

Being a list allows you to define multiple models that can be used by the agent. This is useful for a few reasons:

How does an agent decide which model to pick out of the list?

An agent uses the very first model available in the "config_list" and makes LLM calls against this model. If the model fails (e.g. API throttling) the agent will retry the request against the 2nd model and so on until prompt completion is received (or throws an error if none of the models successfully completes the request). In general there's no implicit/hidden logic inside agents that is used to pick "the best model for the task". However, some specialized agents may attempt to choose "the best model for the task". It is the developers responsibility to pick the right models and use them with agents.

Config list filtering

As described above, the list can be filtered based on certain criteria. This is defined as a dictionary of key to filter on and values to filter by. For example, if you want to select a config with specific model(s), you can pass either a single string or a list of strings:

# Single model (string)
filter_dict = {"model": "gpt-5-nano"}

# Multiple models (list)
filter_dict = {"model": ["gpt-5-nano", "gpt-4.1"]}

This can then be applied to a config when constructing an LLM_CONFIG with the where method:

llm_config = autogen.LLMConfig(path="OAI_CONFIG_LIST").where(**filter_dict)

Or, directly when loading the config list using from_json:

llm_config = autogen.LLMConfig.from_json(path="OAI_CONFIG_LIST").where(**filter_dict)

Tags

Model names can differ between OpenAI and Azure OpenAI, so tags offer an easy way to smooth over this inconsistency. Tags are a list of strings in the config_list, for example for the following config_list:

config_list = [
    {"api_type": "openai", "model": "my-gpt-4.1-deployment", "api_key": "", "tags": ["gpt4o", "openai"]},
    {"api_type": "openai", "model": "llama-7B", "base_url": "http://127.0.0.1:8080", "tags": ["llama", "local"]},
]

When filtering the config_list, you can pass a single tag or multiple tags. A config is selected if it has at least one of the specified tags:

# Single tag
filter_dict = {"tags": "llama"}
llm_config = autogen.LLMConfig(*config_list).where(**filter_dict)
assert len(llm_config.config_list) == 1

# Multiple tags
filter_dict = {"tags": ["llama", "another_tag"]}
llm_config = autogen.LLMConfig(*config_list).where(**filter_dict)

Adding http client in llm_config for proxy

In AG2, a deepcopy is used on llm_config to ensure that the llm_config passed by user is not modified internally. You may get an error if the llm_config contains objects of a class that do not support deepcopy. To fix this, you need to implement a __deepcopy__ method for the class.

The below example shows how to implement a __deepcopy__ method for http client and add a proxy.

#!pip install httpx
import httpx

from autogen import LLMConfig

class MyHttpClient(httpx.Client):
    def __deepcopy__(self, memo):
        return self

llm_config = LLMConfig(config_list={
    "api_type": "openai",
    "model": "my-gpt-5-deployment",
    "api_key": "",
}, http_client=MyHttpClient(proxy="http://localhost:8030"))

Using Azure Active Directory (AAD) Authentication

Azure Active Directory (AAD) provides secure access to resources and applications. Follow the steps below to configure AAD authentication for AG2.

Prerequisites

For more detailed and up-to-date instructions, please refer to the official Azure OpenAI documentation.

Step 1: Register an Application in AAD

  1. Navigate to the Azure portal.
  2. Go to Azure Active Directory > App registrations.
  3. Click on New registration.
  4. Enter a name for your application.
  5. Set the Redirect URI (optional).
  6. Click Register.

For detailed instructions, refer to the official Azure AD Quickstart documentation.

Step 2: Configure API Permissions

  1. After registration, go to API permissions.
  2. Click Add a permission.
  3. Select Microsoft Graph and then Delegated permissions.
  4. Add the necessary permissions (e.g., User.Read).

For more details, see API permissions in Microsoft Graph

Step 3: Obtain Client ID and Tenant ID

  1. Go to Overview of your registered application.
  2. Note down the Application (client) ID and Directory (tenant) ID.

For more details, visit Register an application with the Microsoft identity platform

Step 4: Configure Your Application

Use the obtained Client ID and Tenant ID in your application configuration. Here’s an example of how to do this in your configuration file:

aad_config = {
    "client_id": "YOUR_CLIENT_ID",
    "tenant_id": "YOUR_TENANT_ID",
    "authority": "https://login.microsoftonline.com/YOUR_TENANT_ID",
    "scope": ["https://graph.microsoft.com/.default"],
}

Step 5: Authenticate and Acquire Tokens

Use the following code to authenticate and acquire tokens:

from msal import ConfidentialClientApplication

app = ConfidentialClientApplication(
    client_id=aad_config["client_id"],
    client_credential="YOUR_CLIENT_SECRET",
    authority=aad_config["authority"]
)

result = app.acquire_token_for_client(scopes=aad_config["scope"])

if "access_token" in result:
    print("Token acquired")
else:
    print("Error acquiring token:", result.get("error"))

For more details, refer to the Authenticate and authorize in Azure OpenAI Service and How to configure Azure OpenAI Service with Microsoft Entra ID authentication.

Step 6: Configure Azure OpenAI with AAD Auth in AG2

To use AAD authentication with Azure OpenAI in AG2, configure the llm_config with the necessary parameters. Here is an example configuration:

from autogen import LLMConfig

llm_config = LLMConfig(config_list={
    "model": "gpt-4",
    "base_url": "YOUR_BASE_URL",
    "api_type": "azure",
    "api_version": "2025-01-01",
    "max_tokens": 1000,
    "azure_ad_token_provider": "DEFAULT"
})

In this configuration:

Example of Initializing an Assistant Agent with AAD Auth

import autogen

# Initialize the assistant agent with the AAD authenticated config
assistant = autogen.AssistantAgent(name="assistant", llm_config=llm_config)

Troubleshooting

If you encounter issues, check the following:

This documentation provides a complete guide to configure and use AAD authentication with Azure OpenAI in AG2.

Other configuration parameters

Besides the config_list, there are other parameters that can be used to configure the LLM. These are split between parameters specifically used by Autogen and those passed into the model client.

AG2 specific parameters

Extra model client parameters

It is also possible to passthrough parameters through to the OpenAI client. Parameters that correspond to the OpenAI client or the OpenAI completions create API can be supplied. This is commonly used for things like temperature, or timeout.

Example

from autogen import LLMConfig

llm_config = LLMConfig(
    {
        "model": "my-gpt-5-deployment",
        "api_key": os.environ.get("AZURE_OPENAI_API_KEY"),
        "api_type": "azure",
        "base_url": os.environ.get("AZURE_OPENAI_API_BASE"),
        "api_version": "2024-02-01",
    },
    {
        "api_type": "openai",
        "model": "llama-7B",
        "base_url": "http://127.0.0.1:8080",
        "api_type": "openai",
    },
    temperature = 0.9,
    timeout = 300,
)

Other helpers for loading a config list

See this notebook for examples of using the above functions.