Amazon Bedrock - AG2

Amazon Bedrock

AG2 allows you to use Amazon's generative AI Bedrock service to run inference with a number of open-weight models and as well as their own models.

Amazon Bedrock supports models from providers such as Meta, Anthropic, Cohere, and Mistral.

In this notebook, we demonstrate how to use Anthropic's Sonnet model for AgentChat in AG2.

Model features / support

Amazon Bedrock supports a wide range of models, not only for text generation but also for image classification and generation. Not all features are supported by AG2 or by the Converse API used. Please see Amazon's documentation on the features supported by the Converse API.

At this point in time AG2 supports text generation and image classification (passing images to the LLM).

Requirements

To use Amazon Bedrock with AG2, first you need to install the ag2[bedrock] package.

Pricing

When we combine the number of models supported and costs being on a per-region basis, it's not feasible to maintain the costs for each model+region combination within the AG2 implementation. Therefore, it's recommended that you add the following to your config with cost per 1,000 input and output tokens, respectively:

{
    ...
    "price": [0.003, 0.015]
    ...
}

Amazon Bedrock pricing is available here.

# If you need to install AG2 with Amazon Bedrock
pip install ag2[bedrock]

Set the config for Amazon Bedrock

Amazon's Bedrock does not use the api_key as per other cloud inference providers for authentication, instead it uses a number of access, token, and profile values. These fields will need to be added to your client configuration. Please check the Amazon Bedrock documentation to determine which ones you will need to add.

The available parameters are:

Beyond the authentication credentials, the only mandatory parameters are api_type and model.

The following parameters are common across all models used:

You can also include parameters specific to the model you are using (see the model detail within Amazon's documentation for more information), the four supported additional parameters are:

An additional parameter can be added that denotes whether the model supports a system prompt (which is where the system messages are not included in the message list, but in a separate parameter). This defaults to True, so set it to False if your model (for example Mistral's Instruct models) doesn't support this feature:

Retry Configuration

AG2 supports configuring exponential backoff and retry behavior for Bedrock API calls. This helps handle transient errors, rate limits, and network issues gracefully. The following retry configuration parameters are available:

Best Practices:

Environment Variable Support: You can also configure retries via the AWS_MAX_ATTEMPTS environment variable, which maps to total_max_attempts.

It is important to add the api_type field and set it to a string that corresponds to the client type used: bedrock.

Example:

[
    {
        "api_type": "bedrock",
        "model": "amazon.titan-text-premier-v1:0",
        "api_key": BEDROCK_API_KEY,
        "aws_region": "us-east-1",
        "aws_access_key": "",
        "aws_secret_key": "",
        "aws_session_token": "",
        "aws_profile_name": "",
    },
    {
        "api_type": "bedrock",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0",
        "api_key": BEDROCK_API_KEY,
        "aws_region": "us-east-1",
        "aws_access_key": "",
        "aws_secret_key": "",
        "aws_session_token": "",
        "aws_profile_name": "",
        "temperature": 0.5,
        "topP": 0.2,
        "maxTokens": 250,
        "total_max_attempts": 5,  # Retry configuration
        "mode": "standard",
    },
    {
        "api_type": "bedrock",
        "model": "mistral.mixtral-8x7b-instruct-v0:1",
        "api_key": BEDROCK_API_KEY,
        "aws_region": "us-east-1",
        "aws_access_key": "",
        "aws_secret_key": "",
        "supports_system_prompts": False, # Mistral Instruct models don't support a separate system prompt
        "total_max_attempts": 8,  # More retries for reliability
        "mode": "adaptive",  # Adaptive mode for rate limit handling
        "price": [0.00045, 0.0007] # Specific pricing for this model/region
    }
]

Advanced: additionalModelRequestFields

Amazon Bedrock models often offer advanced, model-specific features that are not part of general AG2 config. AG2 now supports the additional_model_request_fields parameter, which allows you to send these advanced options directly to Bedrock.

Typical use cases include (but are not limited to):

How to use:

For example, to enable Claude's 'thinking' mode:

llm_config_bedrock = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "anthropic.claude-3-7-sonnet-20250219-v1:0",
        "aws_region": "us-east-1",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "additional_model_request_fields": {
            # Claude thinking configuration example
            "thinking": {
                "type": "enabled",
                "budget_tokens": 1024,  # Must be < max_tokens and above or equal to 1024 tokens
            },
        },
        "max_tokens": 4096,
    },
)

Refer to the Amazon Bedrock model documentation to discover which features are supported by your chosen model. AG2 will pass all fields in additional_model_request_fields through to Bedrock as-is;

Using within an AWS Lambda function

If you are using your AG2 code within an AWS Lambda function, you can utilise the attached role to access the Bedrock service and do not need to provide access, token, or profile values.

Retry Configuration Examples

Basic Retry Configuration

The following example shows a basic configuration with default retry settings (5 total attempts, standard mode):

llm_config_default = autogen.LLMConfig(config_list={
    "api_type": "bedrock",
    "model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "aws_region": "us-east-1",
    "aws_access_key": "[FILL THIS IN]",
    "aws_secret_key": "[FILL THIS IN]",
    # Default: total_max_attempts=5, mode="standard"
})

High-Reliability Configuration

For critical applications that need maximum retry attempts:

llm_config_reliable = autogen.LLMConfig(config_list={
    "api_type": "bedrock",
    "model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "aws_region": "us-east-1",
    "aws_access_key": "[FILL THIS IN]",
    "aws_secret_key": "[FILL THIS IN]",
    "total_max_attempts": 10,  # More retries for reliability
    "mode": "adaptive",  # Best for handling various error types
})

Rate-Limit Optimized Configuration

For handling rate limits and throttling:

llm_config_rate_limit = autogen.LLMConfig(config_list={
    "api_type": "bedrock",
    "model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "aws_region": "us-east-1",
    "aws_access_key": "[FILL THIS IN]",
    "aws_secret_key": "[FILL THIS IN]",
    "total_max_attempts": 8,
    "mode": "adaptive",  # Best for rate limit handling
})

Fast-Fail Configuration

For applications that need quick failure detection:

llm_config_fast_fail = autogen.LLMConfig(config_list={
    "api_type": "bedrock",
    "model": "anthropic.claude-3-5-sonnet-20241022-v2:0",
    "aws_region": "us-east-1",
    "aws_access_key": "[FILL THIS IN]",
    "aws_secret_key": "[FILL THIS IN]",
    "total_max_attempts": 2,  # Minimal retries for fast failure
    "mode": "standard",
})

For more detailed information on retry configuration, see the Exponential Backoff and Retry Configuration notebook.

Two-agent Coding Example

Configuration

Start with our configuration - we'll use Anthropic's Sonnet model and put in recent pricing. Additionally, we'll reduce the temperature to 0.1 so its responses are less varied.

Then we'll construct a simple conversation between a User proxy and an ConversableAgent, which uses the Sonnet model.

from typing_extensions import Annotated

import autogen

llm_config_bedrock = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0",
        "api_key": BEDROCK_API_KEY,
        "aws_region": "us-east-1",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "price": [0.003, 0.015],
    },
    temperature=0.1,
    cache_seed=None,  # turn off caching
)

assistant = autogen.AssistantAgent("assistant", llm_config=llm_config_bedrock)

user_proxy = autogen.UserProxyAgent(
    "user_proxy",
    human_input_mode="NEVER",
    code_execution_config={
        "work_dir": "coding",
        "use_docker": False,
    },
    is_termination_msg=lambda x: x.get("content", "") and "TERMINATE" in x.get("content", ""),
    max_consecutive_auto_reply=1,
)

user_proxy.initiate_chat(
    assistant,
    message="Write a python program to print the first 10 numbers of the Fibonacci sequence. Just output the python code, no additional information.",
)

user_proxy (to assistant):

Write a python program to print the first 10 numbers of the Fibonacci sequence. Just output the python code, no additional information.

--------------------------------------------------------------------------------
assistant (to user_proxy):

'''python
# Define a function to calculate Fibonacci sequence
def fibonacci(n):
    if n <= 0:
        return []
    elif n == 1:
        return [0]
    elif n == 2:
        return [0, 1]
    else:
        sequence = [0, 1]
        for i in range(2, n):
            sequence.append(sequence[i-1] + sequence[i-2])
        return sequence

# Call the function to get the first 10 Fibonacci numbers
fib_sequence = fibonacci(10)
print(fib_sequence)
'''

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...
user_proxy (to assistant):

exitcode: 0 (execution succeeded)
Code output:
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

--------------------------------------------------------------------------------
assistant (to user_proxy):

Great, the code executed successfully and printed the first 10 numbers of the Fibonacci sequence correctly.

TERMINATE

--------------------------------------------------------------------------------

Tool Call Example

In this example, instead of writing code, we will show how we can perform multiple tool calling with Meta's Llama 3.1 70B model, where it recommends calling more than one tool at a time.

We'll use a simple travel agent assistant program where we have a couple of tools for weather and currency conversion.

import json
from typing import Literal

import autogen

llm_config_bedrock = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "meta.llama3-1-70b-instruct-v1:0",
        "aws_region": "us-west-2",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "price": [0.00265, 0.0035],
    },
    cache_seed=None,  # turn off caching
)

# Create the agent and include examples of the function calling JSON in the prompt
# to help guide the model
chatbot = autogen.AssistantAgent(
    name="chatbot",
    llm_config=llm_config_bedrock,
    system_message="""For currency exchange and weather forecasting tasks,
        only use the functions you have been provided with.
        Output only the word 'TERMINATE' when an answer has been provided.
        Use both tools together if you can.""",
)

user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    is_termination_msg=lambda x: x.get("content", "") and "TERMINATE" in x.get("content", ""),
    human_input_mode="NEVER",
    max_consecutive_auto_reply=2,
)

# Create the two functions, annotating them so that those descriptions can be passed through to the LLM.
# With Meta's Llama 3.1 models, they are more likely to pass a numeric parameter as a string, e.g. "123.45" instead of 123.45, so we'll convert numeric parameters from strings to floats if necessary.
# We associate them with the agents using `register_for_execution` for the user_proxy so it can execute the function and `register_for_llm` for the chatbot (powered by the LLM) so it can pass the function definitions to the LLM.

# Currency Exchange function

CurrencySymbol = Literal["USD", "EUR"]

# Define our function that we expect to call

def exchange_rate(base_currency: CurrencySymbol, quote_currency: CurrencySymbol) -> float:
    if base_currency == quote_currency:
        return 1.0
    elif base_currency == "USD" and quote_currency == "EUR":
        return 1 / 1.1
    elif base_currency == "EUR" and quote_currency == "USD":
        return 1.1
    else:
        raise ValueError(f"Unknown currencies {base_currency}, {quote_currency}")

# Register the function with the agent

@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Currency exchange calculator.")
def currency_calculator(
    base_amount: Annotated[float, "Amount of currency in base_currency, float values (no strings), e.g. 987.82"],
    base_currency: Annotated[CurrencySymbol, "Base currency"] = "USD",
    quote_currency: Annotated[CurrencySymbol, "Quote currency"] = "EUR",
) -> str:
    # If the amount is passed in as a string, e.g. "123.45", attempt to convert to a float
    if isinstance(base_amount, str):
        base_amount = float(base_amount)

quote_amount = exchange_rate(base_currency, quote_currency) * base_amount
    return f"{format(quote_amount, '.2f')} {quote_currency}"

# Weather function

# Example function to make available to model
def get_current_weather(location, unit="fahrenheit"):
    """Get the weather for some location"""
    if "chicago" in location.lower():
        return json.dumps({"location": "Chicago", "temperature": "13", "unit": unit})
    elif "san francisco" in location.lower():
        return json.dumps({"location": "San Francisco", "temperature": "55", "unit": unit})
    elif "new york" in location.lower():
        return json.dumps({"location": "New York", "temperature": "11", "unit": unit})
    else:
        return json.dumps({"location": location, "temperature": "unknown"})

# Register the function with the agent

@user_proxy.register_for_execution()
@chatbot.register_for_llm(description="Weather forecast for US cities.")
def weather_forecast(
    location: Annotated[str, "City name"],
) -> str:
    weather_details = get_current_weather(location=location)
    weather = json.loads(weather_details)
    return f"{weather['location']} will be {weather['temperature']} degrees {weather['unit']}"

# start the conversation
res = user_proxy.initiate_chat(
    chatbot,
    message="What's the weather in New York and can you tell me how much is 123.45 EUR in USD so I can spend it on my holiday?",
    summary_method="reflection_with_llm",
)

print(res.summary["content"])

Group Chat Example with Anthropic's Claude 3 Sonnet, Mistral's Large 2, and Meta's Llama 3.1 70B

The flexibility of using LLMs from the industry's leading providers, particularly larger models, with Amazon Bedrock allows you to use multiple of them in a single workflow.

Here we have a conversation that has two models (Anthropic's Claude 3 Sonnet and Mistral's Large 2) debate each other with another as the judge (Meta's Llama 3.1 70B). Additionally, a tool call is made to pull through some mock news that they will debate on.

from typing import Annotated, Literal

import autogen
from autogen import AssistantAgent, GroupChat, GroupChatManager, UserProxyAgent

llm_config_sonnet = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0",
        "aws_region": "us-east-1",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "price": [0.003, 0.015],
    },
    temperature=0.1,
    cache_seed=None,  # turn off caching
)

llm_config_mistral = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "mistral.mistral-large-2407-v1:0",
        "aws_region": "us-west-2",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "price": [0.003, 0.009],
    },
    temperature=0.1,
    cache_seed=None,  # turn off caching
)

llm_config_llama31_70b = autogen.LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "meta.llama3-1-70b-instruct-v1:0",
        "aws_region": "us-west-2",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
        "price": [0.00265, 0.0035],
    },
    cache_seed=None,  # turn off caching
)

alice = AssistantAgent(
    "sonnet_agent",
    system_message="You are from Anthropic, an AI company that created the Sonnet large language model. You make arguments to support your company's position. You analyse given text. You are not a programmer and don't use Python. Pass to mistral_agent when you have finished. Start your response with 'I am sonnet_agent'.",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
    llm_config=llm_config_sonnet,
)

bob = autogen.AssistantAgent(
    "mistral_agent",
    system_message="You are from Mistral, an AI company that created the Large v2 large language model. You make arguments to support your company's position. You analyse given text. You are not a programmer and don't use Python. Pass to the judge if you have finished. Start your response with 'I am mistral_agent'.",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
    llm_config=llm_config_mistral,
)

charlie = AssistantAgent(
    "research_assistant",
    system_message="You are a helpful assistant to research the latest news and headlines. You have access to call functions to get the latest news articles for research through 'code_interpreter'.",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
    llm_config=llm_config_llama31_70b,
)

# Add judge agent

dan = AssistantAgent(
    "judge",
    system_message="You are a judge. You will evaluate the arguments and make a decision on which one is more convincing. End your decision with the word 'TERMINATE' to conclude the debate.",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
    llm_config=llm_config_llama31_70b,
)

code_interpreter = UserProxyAgent(
    "code_interpreter",
    human_input_mode="NEVER",
    code_execution_config={
        "work_dir": "coding",
        "use_docker": False,
    },
    default_auto_reply="",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
)

@code_interpreter.register_for_execution()  # Decorator factory for registering a function to be executed by an agent
@charlie.register_for_llm(
    name="get_headlines", description="Get the headline of a particular day."
)  # Decorator factory for registering a function to be used by an agent
def get_headlines(headline_date: Annotated[str, "Date in MMDDYY format, e.g., 06192024"]) -> str:
    mock_news = {
        "06202024": "Epic Duel of the Titans: Anthropic and Mistral Usher in a New Era of Text Generation Excellence.\n        In a groundbreaking revelation that has sent shockwaves through the AI industry, Anthropic has unveiled\n        their state-of-the-art text generation model, Sonnet, hailed as a monumental leap in artificial intelligence.\n        Almost simultaneously, Mistral countered with their equally formidable creation, Large 2, showcasing\n        unparalleled prowess in generating coherent and contextually rich text. This scintillating rivalry\n        between two AI behemoths promises to revolutionize the landscape of machine learning, heralding an\n        era of unprecedented creativity and sophistication in text generation that will reshape industries,\n        ignite innovation, and captivate minds worldwide.",
        "06192024": "OpenAI founder Sutskever sets up new AI company devoted to safe superintelligence.",
    }
    return mock_news.get(headline_date, "No news available for today.")

user_proxy = UserProxyAgent(
    "user_proxy",
    human_input_mode="NEVER",
    code_execution_config=False,
    default_auto_reply="",
    is_termination_msg=lambda x: x.get("content", "").find("TERMINATE") >= 0,
)

groupchat = GroupChat(
    agents=[alice, bob, charlie, dan, code_interpreter],
    messages=[],
    allow_repeat_speaker=False,
    max_round=10,
)

manager = GroupChatManager(
    groupchat=groupchat,
    llm_config=llm_config_llama31_70b,
)

task = "Analyze the potential of Anthropic and Mistral to revolutionize the field of AI based on today's headlines. Today is 06202024. Start by selecting 'research_assistant' to get relevant news articles and then ask sonnet_agent and mistral_agent to respond before the judge evaluates the conversation."

user_proxy.initiate_chat(manager, message=task)

Image classification with Anthropic's Claude 3 Sonnet

AG2's Amazon Bedrock client class supports inputting images for the LLM to respond to.

In this simple example, we'll use an image on the Internet and send it to Anthropic's Claude 3 Sonnet model to describe.

Here's the image we'll use:

llm_config_sonnet = LLMConfig(config_list={
        "api_type": "bedrock",
        "model": "anthropic.claude-3-sonnet-20240229-v1:0",
        "aws_region": "us-east-1",
        "aws_access_key": "[FILL THIS IN]",
        "aws_secret_key": "[FILL THIS IN]",
    },
    cache_seed=None,  # turn off caching
)

# We'll use a Multimodal agent to handle the image

import autogen
from autogen import AssistantAgent, UserProxyAgent
from autogen.agentchat.contrib.multimodal_conversable_agent import MultimodalConversableAgent

image_agent = MultimodalConversableAgent(
    name="image-explainer",
    max_consecutive_auto_reply=10,
    llm_config=llm_config_sonnet,
)

user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    system_message="A human admin.",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=0,
    code_execution_config={
        "use_docker": False
    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.
)

# Ask the image_agent to describe the image
result = user_proxy.initiate_chat(
    image_agent,
    message="""What's happening in this image?
<img https://microsoft.github.io/autogen/assets/images/love-ec54b2666729d3e9d93f91773d1a77cf.png>.
""",
)

User_proxy (to image-explainer):

What's happening in this image?
<image>.

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
image-explainer (to User_proxy):

This image appears to be an advertisement or promotional material for a company called Autogen. The central figure is a stylized robot or android holding up a signboard with the company's name on it. The signboard also features a colorful heart design made up of many smaller hearts, suggesting themes related to love, care, or affection. The robot has a friendly, cartoonish expression with a large blue eye or lens. The overall style and color scheme give it a vibrant, eye-catching look that likely aims to portray Autogen as an innovative, approachable technology brand focused on connecting with people.

--------------------------------------------------------------------------------