# WebSurferAgent with Firecrawl Integration

This notebook demonstrates how to use the WebSurferAgent with the Firecrawl tool for web scraping and crawling.

## Setup

First, import the necessary modules and set up the WebSurferAgent with Firecrawl.

```python
import os

from autogen.agents.experimental.websurfer import WebSurferAgent

# Set up your API keys
FIRECRAWL_API_KEY = os.getenv("FIRECRAWL_API_KEY", "your_firecrawl_api_key_here")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "your_openai_api_key_here")

# LLM configuration
llm_config = {
    "model": "gpt-4",
    "api_key": OPENAI_API_KEY,
    "temperature": 0.1,
}

print("✓ Setup complete")
```

## Creating WebSurferAgent with Firecrawl

Create a WebSurferAgent that uses Firecrawl as its web tool.

```python
# Create WebSurferAgent with Firecrawl
websurfer_agent = WebSurferAgent(
    name="WebSurfer",
    llm_config=llm_config,
    web_tool="firecrawl",
    web_tool_kwargs={
        "firecrawl_api_key": FIRECRAWL_API_KEY,
        # Optional: specify custom Firecrawl API URL for self-hosted instances
        # "firecrawl_api_url": "https://your-firecrawl-instance.com",
    },
    system_message="You are a helpful web researcher. Use Firecrawl to scrape, crawl, search, and research websites to answer user questions.",
)

print(f"✓ WebSurferAgent created with {type(websurfer_agent.tool).__name__}")
print(
    f"✓ Available tool methods: {[method for method in dir(websurfer_agent.tool) if not method.startswith('_') and callable(getattr(websurfer_agent.tool, method))]}"
)
```

## Available Firecrawl Methods

The WebSurferAgent with Firecrawl tool provides access to all Firecrawl capabilities:

1. **Scrape**: Extract content from a single URL
2. **Crawl**: Recursively crawl a website starting from a URL
3. **Map**: Discover URLs from a website
4. **Search**: Search the web for content
5. **Deep Research**: Perform comprehensive research on a topic with analysis

## Example Usage

Here are some example tasks you can perform with the WebSurferAgent using Firecrawl:

```python
# Example 1: Simple web scraping
scrape_message = "Scrape the homepage of https://example.com and summarize the main content."

# Example 2: Website crawling
crawl_message = "Crawl https://example.com and find all the pages related to documentation."

# Example 3: Website mapping
map_message = "Map the structure of https://example.com and list all the main sections."

# Example 4: Web search
search_message = "Search for recent articles about artificial intelligence and summarize the top 3 results."

# Example 5: Deep research
research_message = "Perform deep research on 'sustainable energy trends 2024' and provide a comprehensive analysis."

print("Example messages prepared. Run them individually with the websurfer_agent.")
```

## Running Tasks

To run any of the above tasks, you would use the agent like this:

```python
# Create a user proxy agent to interact with the websurfer
from autogen import UserProxyAgent

user_proxy = UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
    code_execution_config=False,
)

# Start a conversation
user_proxy.initiate_chat(
    websurfer_agent,
    message=scrape_message,
    max_turns=1,
)
```

## Key Features

The WebSurferAgent with Firecrawl provides:

- **Multiple web interaction methods**: scrape, crawl, map, search, and deep research
- **Self-hosted support**: Connect to your own Firecrawl instance via `firecrawl_api_url`
- **Flexible configuration**: Pass any Firecrawl parameters through `web_tool_kwargs`
- **Consistent API**: Same interface as other WebSurferAgent tools (Tavily, DuckDuckGo, etc.)
- **Rich content extraction**: Supports multiple output formats (markdown, HTML)
- **Advanced filtering**: Include/exclude patterns, custom headers, timeouts

This makes the WebSurferAgent a powerful tool for web research and content extraction tasks.