## RealtimeAgent with Gemini API

Deprecated

`RealtimeAgent` is deprecated as of v0.12 and will be removed in v0.14. It relies on deprecated realtime API endpoints. This blog post and associated notebooks will also be removed in v0.14.

**TL;DR:**

- RealtimeAgent now supports [Gemini Multimodal Live API](https://ai.google.dev/api/multimodal-live)

### Why is this important?

We previously supported a Realtime Agent powered by OpenAI. In December 2024, Google rolled out [Gemini 2.0](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/), which includes the multi-modal live APIs. These APIs enable advanced capabilities such as real-time processing of audio inputs in live conversational settings. To ensure developers can fully leverage the capabilities of the latest LLMs, we now also support a RealtimeAgent powered by Gemini.

## Real-Time Voice Interactions over WebRTC

Deprecated

**TL;DR:**  
- Build a real-time voice application using [WebRTC](https://webrtc.org/) and connect it with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/). [Demo implementation](https://github.com/ag2ai/realtime-agent-over-webrtc).  
- **Optimized for Real-Time Interactions**: Experience seamless voice communication with minimal latency and enhanced reliability.

## Real-Time Voice Interactions with the WebSocket Audio Adapter

Deprecated

**TL;DR:**  
- **Demo implementation**: Implement a website using websockets and communicate using voice with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/)  
- **Introducing [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter/#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter)**: Stream audio directly from your browser using [WebSockets](https://fastapi.tiangolo.com/advanced/websockets/).  
- **Simplified Development**: Connect to real-time agents quickly and effortlessly with minimal setup.

## **Realtime over WebSockets**

In our [previous blog post](https://docs.ag2.ai/docs/blog/2024/12/20/RealtimeAgent/), we introduced a way to interact with the [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/) using [**`TwilioAudioAdapter`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/TwilioAudioAdapter/#autogen.agentchat.realtime.experimental.TwilioAudioAdapter). While effective, this approach required a setup-intensive process involving [Twilio](https://www.twilio.com/) integration, account configuration, number forwarding, and other complexities. Today, we're excited to introduce the [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter/#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter), a streamlined approach to real-time audio streaming directly via a web browser.

This post explores the features, benefits, and implementation of the [**`WebSocketAudioAdapter`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/WebSocketAudioAdapter/#autogen.agentchat.realtime.experimental.WebSocketAudioAdapter), showing how it transforms the way we connect with real-time agents.

## Introducing RealtimeAgent Capabilities in AG2

Deprecated

AG2 RealtimeAgent: Live Demo of Voice-Enabled Multi-Agent Conversations | v0.6 - YouTube

Tap to unmute

[AG2 RealtimeAgent: Live Demo of Voice-Enabled Multi-Agent Conversations | v0.6](https://www.youtube.com/watch?v=NbZW2goinKM) AG2

AG2993 subscribers

**TL;DR:**  
- [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/) is coming in the AG2 0.6 release, enabling real-time conversational AI. 
- Features include real-time voice interactions, seamless task delegation to Swarm teams, and Twilio-based telephony integration. 
- Learn how to integrate [Twilio](https://www.twilio.com/) and [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/) into your swarm in this blogpost.

#### **Realtime API Support: What's New?**

We're thrilled to announce the release of [**`RealtimeAgent`**](https://docs.ag2.ai/0.13.3/docs/api-reference/autogen/agentchat/realtime/experimental/RealtimeAgent/), extending AG2's capabilities to support **real-time conversational AI tasks**. This new experimental feature makes it possible for developers to build agents capable of handling voice-based interactions with minimal latency, integrating [OpenAI’s Realtime API](https://openai.com/index/introducing-the-realtime-api/), [Twilio](https://www.twilio.com/) for telephony, and [AG2’s Swarm orchestration](https://docs.ag2.ai/0.13.3/docs/user-guide/advanced-concepts/orchestration/swarm/deprecation/).