Featured image of post Microsoft Releases Magentic UI, a Hands-On Review of the Interactive AI Agent Assistant

Microsoft Releases Magentic UI, a Hands-On Review of the Interactive AI Agent Assistant

What Is Magentic UI?

Magentic UI is an open-source Agent assistant that Microsoft released with much fanfare on May 19, and it just updated to its second version, 0.04, on the 24th

Judging from the official introduction, the goal is to build a human-centered intelligent assistant, powered at its core by several different agent systems, capable of web browsing and operation, code generation and execution, and file generation and analysis.

What mainly sets it apart from other AI browser tools is its transparent and controllable interactive interface, which supports efficient human-AI collaborative participation. The system is built on Microsoft’s agent-building framework AutoGen and is currently still at the prototype research stage.

Core features include:

🧑‍🤝‍🧑 Collaborative planning: users and the AI work together through chat and a plan editor to create and revise step-by-step execution plans.
🤝 Collaborative tasks: users can intervene in real time and guide task execution directly through the web browser or the chat interface. Magentic-UI can proactively ask for confirmation or request assistance when needed.
🛡️ Action guard: sensitive operations require explicit user approval before they can be executed.
🧠 Plan learning and retrieval: it learns from past tasks to optimize future automation workflows and saves good solutions into a plan library. Stored plans can be invoked automatically or manually in subsequent tasks.
🔀 Parallel task execution: it supports running multiple tasks in parallel, and a session status indicator shows in real time when Magentic-UI needs your input or has completed a task.

Design Architecture

The underlying system of Magentic-UI consists of a group of specialized agents adapted from AutoGen’s Magentic-One system (shown in the figure above), with each agent implementing different functions through modular collaboration:

Agent Functionality

Orchestrator

  • Functional role: the main LLM-based controlling agent, responsible for collaboratively making plans with the user, deciding when to request user feedback, and assigning subtasks to other agents.
  • Collaboration flow:
    1. Jointly create a step-by-step execution plan with the user;
    2. Dynamically determine the points where user intervention is needed;
    3. Dispatch subtasks to other agents for execution.

WebSurfer

  • Functional role: an LLM agent equipped with a web browser, supporting multi-round web interaction operations.
  • Core capabilities:
    • Following the Orchestrator’s instructions, perform operations such as clicking, typing, scrolling, and page navigation;
    • Beyond AutoGen’s multimodal web browser: adds tab management, option selection, file upload, and multimodal query support.

Coder

  • Functional role: an LLM agent equipped with a Docker code execution container.
  • Core capabilities:
    • Write and execute Python and Shell commands;
    • Feed the execution results back to the Orchestrator.

**FileSurfer

**

  • Functional role: an LLM agent integrating a Docker container with the MarkItDown package file-conversion tool.
  • Core capabilities:
    • Locate files in Magentic-UI’s controlled directory;
    • Convert files to Markdown format and support question answering over their content.

UserProxy

  • Functional role: the agent role that interacts with the system on behalf of the user.
  • Collaboration mode: the Orchestrator can delegate tasks directly to the user (rather than to other agents) to complete.

User Interaction Flow

The interaction flow of a user using Magentic UI can be illustrated by the following diagram:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
graph TD

    A[用户发起请求] --> B[输入文本/附加图片];

    B --> C{Magentic-UI 生成分步计划};

    C -- 生成成功 --> D[用户进入计划编辑界面];

    C -- 生成失败 --> C1[提示用户重新输入];

    C1 --> B;

  

    D -- 用户编辑计划 --> E{编辑操作};

    E -- 添加/删除/修改步骤 --> D;

    E -- 确认完成 --> F[总控代理存储计划];

  

    F --> G[启动计划执行引擎];

    G --> H{按顺序执行步骤};

  

    H --> I{当前步骤类型?};

    subgraph 具体步骤执行

        direction LR

        I -- WebSurfer --> J[执行网页操作];

        I -- Coder --> K[生成并执行代码];

        I -- FileSurfer --> L[处理文件];

        I -- UserProxy --> M[提示用户手动操作];

    end

  

    J --> N{步骤是否成功完成?};

    K --> N;

    L --> N;

    M --> N;

  

    N -- 是 --> O{所有步骤均已完成?};

    N -- 否 --> P[记录错误原因];

  

    P --> Q{是否需要重新规划?};

    Q -- 是 --> R[请求用户许可重新规划];

    R -- 同意 --> C;

    R -- 拒绝 --> S[终止流程];

    Q -- 否 --> S;

  

    O -- 是 --> T[生成最终结果];

    O -- 否 --> H;

  

    T --> U[向用户呈现结果];

    U --> V[结束];

  1. Input and plan generation

    • The user triggers the flow through text/images, and the system generates an initial plan. If it fails, the user is prompted to re-enter.
    • The user iteratively refines the plan through the editing interface, and after confirmation, moves to the execution phase.
  2. Plan execution logic

    • The Orchestrator assigns tasks step by step to the corresponding agents (WebSurfer/Coder/FileSurfer) or to the user.
    • After each step is completed, the status is verified; if it fails, a decision is made on whether replanning is needed (requiring user authorization).
  3. Dynamic control and termination

    • The user can intervene at any time: pause execution, modify the plan, or terminate the flow.
    • Once all steps are completed, the final result is delivered, with transparent error feedback provided in exceptional cases.

Tool Deployment

Magentic AI is mainly developed in Python, with the web part based on FastAPI. Its AI control primarily uses Playwright to control a virtual browser inside a Docker container, which can be displayed to the user in real time. The large model behind it is mainly used through the API access methods of various open large models.

Therefore, to deploy Magentic UI, you first need to confirm the relevant prerequisites:

  • Docker is installed; if it is a Windows system, install Docker Desktop and integrate it with WSL
  • Provide the API KEY for accessing the large model; the default is OPENAI’s API KEY
  • A Python runtime environment with a version >3.10

The tool was deployed in WSL. First create a Python virtual environment, then install Magentic AI and its related dependencies

1
2
3
python3 -m venv .venv
source .venv/bin/activate
pip3 install magentic-ui

Make sure Docker is started, and before running Magentic UI there should be no related containers running

1
docker ps

For the large model, you can use OPENAI on Microsoft Azure AI; after completing the configuration, set the corresponding api key as the environment variable OPENAI_API_KEY

1
export OPENAI_API_KEY="YOUR_API_KEY"

In addition, Magentic UI itself needs to be configured for the large model; the Azure AI configuration template is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
model_config: &client

  provider: AzureOpenAIChatCompletionClient

  config:

    model: gpt-4o

    azure_endpoint: "<YOUR ENDPOINT>"

    azure_deployment: "<YOUR DEPLOYMENT>"

    api_version: "2024-10-21"

    azure_ad_token_provider:

      provider: autogen_ext.auth.azure.AzureTokenProvider

      config:

        provider_kind: DefaultAzureCredential

        scopes:

          - https://cognitiveservices.azure.com/.default

    max_retries: 10

  

orchestrator_client: *client

coder_client: *client

web_surfer_client: *client

file_surfer_client: *client

action_guard_client: *client

Running

Then run Magentic ui, which uses port 8081 by default. The first run will download multiple different docker images and related dependencies, which takes a long time, so wait for the installation to complete

1
magentic ui --port 8081

Once the following information appears in the running log, startup has succeeded

1
| INFO     | magentic_ui.backend.web.app:lifespan:59 - Application startup complete. Navigate to http://127.0.0.1:8081

Access http://localhost:8081 in the browser

You will see the following interface

After starting a session, for example, ask the AI to help us complete a test

1
Please test whether the login functionality for different users on the saucedemo.com website works correctly, and summarize the issues in markdown format

If the installation is correct and the large model calls work normally, Magentic UI will plan the relevant task plan and ask the user to confirm

The user can also adjust the plan based on the feedback, and during plan execution can interact with the AI to change its execution path. For a concrete execution demonstration, see the video below.

At this point, checking the docker runtime information in WSL, you can see that 3 containers are running, two of which are mangenti UI’s coder and file surfer agents, and the other is a remotely viewable vnc_browser container

After the series of steps is completed, Magentic UI obtains the following markdown-format result summary based on the initial instruction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
以下是对saucedemo.com网站不同用户登录功能测试的结果汇总:

1. **标准用户** (`standard_user`):
    
    - 登录成功。
    - 无问题,用户可以正常访问库存页面并浏览产品。
2. **被锁定用户** (`locked_out_user`):
    
    - 登录失败。
    - 显示错误消息 "Epic sadface: Sorry, this user has been locked out." 账户已被锁定,无法访问。
3. **问题用户** (`problem_user`):
    
    - 登录成功。
    - 产品图像存在不匹配的问题,图像错误是已知问题。
4. **性能故障用户** (`performance_glitch_user`):
    
    - 登录成功。
    - 无明显系统减慢或异常,用户可以正常导航。

这些结果表明网站能够正确处理不同账户类型的登录场景。如果您想亲自测试,可以访问 [saucedemo.com](https://www.saucedemo.com/)。

Hands-On Impressions

Overall, after using it, repositioning AI as an assistant role is clearly more in line with the reality of current AI applications, and displaying the AI’s operation process in real time through the container makes AI task execution no longer a black box, which is indeed much friendlier.

However, in use I still feel the practicality is not very good. Although the official statement says it is still research-oriented at this stage, there is still considerable room for improvement.

  1. Installation and configuration are relatively complex, with numerous dependency packages, and it takes a long time
  2. The capability of the large model itself has a large impact on the execution process, including whether interactions can be completed normally. There are many call failures and exceptions.
  3. The token consumption of the large model is considerable; to complete one task, it actually breaks down into dozens of different requests, and the token consumption of each request is also in the thousands, so with a paid model the cost is high

The figure below shows the token consumption during my trial as viewed in the Azure backend, which is quite considerable.

In short, the development direction of Magentic UI is worth looking forward to, but for now we still need to wait for its further refinement.


Built with Hugo
Theme Stack designed by Jimmy