# DocAgent

In the realm of AI and automation, handling documents and extracting information efficiently is of utmost importance. [`DocAgent`](https://docs.ag2.ai/0.9.8/docs/api-reference/autogen/agents/experimental/DocAgent) introduces an agentic solution to this problem. It handles document ingestion and query tasks seamlessly, and with natural language instructions, by leveraging an internal swarm of agents to streamline document processing and information retrieval.

Tip

DocAgent is new and keen to support your RAG needs. In the first iterations it has been kept simple and uses either local vector Chroma database (default) or an in-memory option (InMemoryQueryEngine).

DocAgent will continue to be developed to a production-ready standard and your feedback and [contributions](https://docs.ag2.ai/latest/docs/contributor-guide/contributing) are most welcome!

## Installation

Install AG2 with the `rag` extra to install the necessary packages for the DocAgent.

```
pip install ag2[openai,rag]
```

## Capabilities

The document agent can perform the following tasks: 1. Ingest documents from a local file or URL. Supported formats:

```
- PDF
- DOCX (DOCX, DOTX, DOCM, DOTM)
- XLSX
- PPTX (PPTX, POTX, PPSX, PPTM, POTM, PPSM)
- HTML
- ASCIIDOC (ADOC, ASCIIDOC, ASC)
- MD (MD, MARKDOWN)
- XML (XML, NXML)
- TXT
- JSON
- CSV
- IMAGE (BMP, JPG, JPEG, PNG, TIFF, TIF)
```

1. Answer questions with RAG capability

Tip

DocAgent answers questions only related to ingested documents, even if it using an LLM it won't answer general knowledge questions.

## Internal Swarm

[`DocAgent`](https://docs.ag2.ai/0.9.8/docs/api-reference/autogen/agents/experimental/DocAgent) leverages a [`Swarm`](https://docs.ag2.ai/0.9.8/docs/user-guide/advanced-concepts/orchestration/swarm/deprecation) of internal agents to handle the complex document processing tasks fluidly and efficiently.

Here's a breakdown of how the internal swarm chat is used within the [`DocAgent`](https://docs.ag2.ai/0.9.8/docs/api-reference/autogen/agents/experimental/DocAgent):

### Swarm Agents

[`DocAgent`](https://docs.ag2.ai/0.9.8/docs/api-reference/autogen/agents/experimental/DocAgent) orchestrates the following swarm agents:

- **Triage Agent**: Decides what type of task to perform from user requests.
- **Task Manager Agent**: Manages the tasks and initiates actions.
- **Data Ingestion Agent**: Ingests the documents.
- **Query Agent**: Answers user questions based on ingested documents.
- **Error Agent**: If anything fails, the error agent will report the problem back.
- **Summary Agent**: Generates a summary of the completed tasks.

### Workflow

1. **Initialization**: The `DocAgent` initializes the swarm agents and sets up the context variables.
2. **Triage User Requests**: The `Triage Agent` categorizes the tasks into ingestions and queries.
3. **Task Management**: The `Task Manager Agent` manages the tasks and ensures they are executed in the correct sequence.
4. **Data Ingestion**: The `Data Ingestion Agent` processes the documents.
5. **Query Execution**: The `Query Agent` answers the user's questions.
6. **Summary Generation**: The `Summary Agent` generates a summary of the completed tasks.

## Parsing documents and web pages

Internally, DocAgent uses [Docling](https://github.com/DS4SD/docling) to convert documents into Markdown files, which are then using for ingestion into a data store (which is then used for querying).

For files (whether local files or URLs that point to files), DocAgent will download those files for Docling to convert. If it's a web page, DocAgent will use [Selenium](https://github.com/SeleniumHQ/Selenium) to browse to the page and extract the content.

## Data Store and Query Engine

DocAgent can currently use two different data stores and query engines.

### Vector Store: Chroma database

[Chroma](https://www.trychroma.com/) is a popular, open-source, vector database. DocAgent will take the Markdown and embed and store the data into the vector database.

This requires the `OPENAI_API_KEY` environment variable as OpenAI's GPT 4o model is used to create the embeddings.

DocAgent will use [LlamaIndex](https://github.com/run-llama/llama_index)'s vector store library to embed and query the Chroma database.

This is the default data store and query engine, so you do not need to create it and pass it to the DocAgent.

### In-memory

DocAgent can also store the Docling Markdown in memory when digesting documents.

When it comes time to query, DocAgent will inject the Markdown into the system message of an internal query agent and use an LLM to respond to the question.

Import notes about using the in-memory query engine:

- The full Markdown content, from the ingested documents, is put into the context window, so be cautious of putting too much content in as it may be too much for the LLM.
- The Markdown is put into the start of the system message to maximize the chance of utilizing the LLM's caching ability and reduce cost. However, adding more documents in subsequent messages will change the system message and the cache will not be effective, so try to ingest all the documents before querying.
- As it's in-memory, ingestion content will be lost when the agent is destroyed.
- LLMs can be better at answering some queries than the vector store as they are processing all of the context to answer the query. However, this does come at the cost of more token usage.

To use the in-memory query engine, you will need to create it and pass it to the DocAgent, see the following example:

```
from autogen.agents.experimental import DocAgent, InMemoryQueryEngine

# Example LLM Config
llm_config_list = {"config_list": {"api_type": "openai", "model": "gpt-4o-mini", "cache_seed": None}}

# Create our In-Memory Query Engine
inmemory_qe = InMemoryQueryEngine(llm_config=llm_config_list)

# Include the Query Engine when creating your DocAgent
with llm_config
    doc_agent = DocAgent(
        name="doc_agent",
        query_engine=inmemory_qe,
    )

...
```

## Example

Tip

The internal ingestation of documents requires an LLM and it will use OpenAI's GPT-4o. Please ensure you have an `OPENAI_API_KEY` environment variable set.

Warning

This agent is currently in our `experimental` namespace, indicating that we have tested the functionality but the agent's interface may change. Please use it with that in mind.

If you do find any bugs please [log an issue](https://github.com/ag2ai/ag2/issues) in the AG2 repository.

In the following simple example we ask the [`DocAgent`](https://docs.ag2.ai/0.9.8/docs/api-reference/autogen/agents/experimental/DocAgent) to ingest a document and then provide a financial summary.

Note that the request is handled in natural language and the output will show the internal agents working together to understand, classify, ingest, and query.

```
from autogen import LLMConfig
from autogen.agents.experimental import DocAgent

llm_config = LLMConfig(api_type="openai", model="gpt-4o")

# Create our DocAgent
with llm_config:
    document_agent = DocAgent()

# Update this path to suit your environment
response = document_agent.run(
    "Can you ingest ../test/agentchat/contrib/graph_rag/Toast_financial_report.pdf and tell me the fiscal year 2024 financial summary?",
    max_turns=1
)

# Iterate through the chat automatically with console output
response.process()
```

The fiscal year 2024 financial summary for Toast, Inc. is as follows:

- Total assets increased to $2,227 million by September 30, 2024, from $1,958 million at the end of 2023.
- The company's total liabilities stood at $807 million.
- Stockholders' equity was reported at $1,420 million.
- Toast, Inc. generated total revenue of $3,622 million for the nine months ending September 30, 2024.
- The company recorded a gross profit of $857 million.
- Operating expenses amounted to $873 million, resulting in an operating loss of $16 million.
- There was a net loss of $13 million for the period, with a basic and diluted loss per share of $0.02.

```