Non-OpenAI Models - AG2

RealtimeAgent with Gemini API

Deprecated

RealtimeAgent is deprecated as of v0.12 and will be removed in v0.14. It relies on deprecated realtime API endpoints. This blog post and associated notebooks will also be removed in v0.14.

TL;DR:

• RealtimeAgent now supports Gemini Multimodal Live API

Why is this important?

We previously supported a Realtime Agent powered by OpenAI. In December 2024, Google rolled out Gemini 2.0, which includes the multi-modal live APIs. These APIs enable advanced capabilities such as real-time processing of audio inputs in live conversational settings. To ensure developers can fully leverage the capabilities of the latest LLMs, we now also support a RealtimeAgent powered by Gemini.

Cross-Framework LLM Tool Integration with AG2

Tool Interoperability: Use LangChain, CrewAI & PydanticAI Tools in AG2 | v0.6 - YouTube

TL;DR AG2 lets you bring in Tools from different frameworks like LangChain, CrewAI, and PydanticAI.

• LangChain Tools: Useful for tasks like API querying and web scraping.
• CrewAI Tools: Offers a variety of tools for web scraping, search, and more.
• PydanticAI Tools: Adds context-driven tools and structured data processing.

Enhanced Support for Non-OpenAI Models

TL;DR

• AutoGen has expanded integrations with a variety of cloud-based model providers beyond OpenAI.
• Leverage models and platforms from Gemini, Anthropic, Mistral AI, Together.AI, and Groq for your AutoGen agents.
• Utilise models specifically for chat, language, image, and coding.
• LLM provider diversification can provide cost and resilience benefits.

In addition to the recently released AutoGen Google Gemini client, new client classes for Mistral AI, Anthropic, Together.AI, and Groq enable you to utilize over 75 different large language models in your AutoGen agent workflow.

These new client classes tailor AutoGen's underlying messages to each provider's unique requirements and remove that complexity from the developer, who can then focus on building their AutoGen workflow.

Using them is as simple as installing the client-specific library and updating your LLM config with the relevant api_type and model. We'll demonstrate how to use them below.

The community is continuing to enhance and build new client classes as cloud-based inference providers arrive. So, watch this space, and feel free to discuss or develop another one.