## RealtimeAgent with Gemini API

**TL;DR:**

- RealtimeAgent now supports [Gemini Multimodal Live API](https://ai.google.dev/api/multimodal-live)

### Why is this important?

We previously supported a Realtime Agent powered by OpenAI. In December 2024, Google rolled out [Gemini 2.0](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/), which includes the multi-modal live APIs. These APIs enable advanced capabilities such as real-time processing of audio inputs in live conversational settings. To ensure developers can fully leverage the capabilities of the latest LLMs, we now also support a RealtimeAgent powered by Gemini.

## Real-Time Voice Interactions over WebRTC

**TL;DR:** 
- Build a real-time voice application using [WebRTC](https://webrtc.org/) and connect it with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent). [Demo implementation](https://github.com/ag2ai/realtime-agent-over-webrtc). 
- **Optimized for Real-Time Interactions**: Experience seamless voice communication with minimal latency and enhanced reliability.

## Real-Time Voice Interactions with the WebSocket Audio Adapter

**TL;DR:** 
- **Demo implementation**: Implement a website using websockets and communicate using voice with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) 
- **Introducing [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter.websocketaudioadapter)**: Stream audio directly from your browser using [WebSockets](https://fastapi.tiangolo.com/advanced/websockets/). 
- **Simplified Development**: Connect to real-time agents quickly and effortlessly with minimal setup.

## **Realtime over WebSockets**

In our [previous blog post](https://docs.ag2.ai/0.10.3/docs/blog/2024/12/20/RealtimeAgent/index), we introduced a way to interact with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) using [**`TwilioAudioAdapter`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/TwilioAudioAdapter#autogen.agentchat.realtime.experimental.TwilioAudioAdapter.twilioaudioadapter). While effective, this approach required a setup-intensive process involving [Twilio](https://www.twilio.com/) integration, account configuration, number forwarding, and other complexities. Today, we're excited to introduce the [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter.websocketaudioadapter), a streamlined approach to real-time audio streaming directly via a web browser.

This post explores the features, benefits, and implementation of the [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter.websocketaudioadapter), showing how it transforms the way we connect with real-time agents.

## Introducing RealtimeAgent Capabilities in AG2

AG2 RealtimeAgent: Live Demo of Voice-Enabled Multi-Agent Conversations

**TL;DR:** 
- [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) is coming in the AG2 0.6 release, enabling real-time conversational AI. - Features include real-time voice interactions, seamless task delegation to Swarm teams, and Twilio-based telephony integration. - Learn how to integrate [Twilio](https://www.twilio.com/) and [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent) into your swarm in this blogpost.

#### [**Realtime API Support: What's New?**](https://docs.ag2.ai/0.10.3/docs/blog/2024/12/20/RealtimeAgent/#realtime-api-support-whats-new)

We're thrilled to announce the release of [**`RealtimeAgent`**](https://docs.ag2.ai/0.10.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent), extending AG2's capabilities to support **real-time conversational AI tasks**. This new experimental feature makes it possible for developers to build agents capable of handling voice-based interactions with minimal latency, integrating [OpenAI’s Realtime API](https://openai.com/index/introducing-the-realtime-api/), [Twilio](https://www.twilio.com/) for telephony, and [AG2’s Swarm orchestration](https://docs.ag2.ai/0.10.3/docs/user-guide/advanced-concepts/orchestration/swarm/deprecation).