“We automated 150 tasks with AI Agents, just copy us” - Microsoft AI

---

---

AI Summary

Podcast Summary

• Topic: Automation of real tasks with AI agents, impact of reasoning models like OAN, and AI agents used at work.
• Guests: AI agent experts from Microsoft.

Windows Agent Arena

• Description: A benchmark for testing AI agents on real-world tasks in computing environments.
• Agents: Focus on desktop or PC controlling agents with access to common Windows OS applications.
• Importance: Allows measurement of agent performance on practical tasks beyond memorization benchmarks.
• Future: Excitement about agents handling less common applications with fewer users.

Agent Performance and Benchmarks

• Current Benchmarks: Include MLU, Big Bench, etc., which are specific in their measurements.
• New Benchmarks: Mind Web, Web Arena, Visual Web Arena, UFO, OS World, Android World.
• Human Performance: Casual Windows users have about a 74% success rate on the benchmark tasks.

AI Agent Development and Use

• Agent Capabilities: Reasoning, planning, and executing with minimal human intervention.
• Agent Construction: Use of internal proprietary perception models and planners, not relying on Autogen.
• Mainstream Adoption: Requires a safe and robust way for humans to intervene and work with agents.

AI Agents in the Workplace

• Usage: GitHub Copilot for coding, GPT for reasoning and idea generation.
• Impact of Advanced Models: OAN and reasoning models can handle complex tasks but raise concerns about inference costs and accessibility.

Windows Agent Arena Demonstration

• Tasks: Installing extensions, enabling privacy features, changing profile names, etc.
• Process: Agents plan, execute actions, and receive feedback through screenshots and state information.

AI Agent Future and Open Source

• Near-Term Future: Humans in the loop with AI agents, balancing strengths of LLMs and human capabilities.
• Open Source Debate: A balance between open source and closed source models drives innovation.

AI Agent Interaction and Accessibility

• Modalities: Text, voice, and potentially other forms of interaction depending on user needs and accessibility.
• Contextual Understanding: Agents need to understand and learn from human input and context.

Windows Agent Arena Access

• Setup: Requires setup of a “golden image” and running commands as detailed on the GitHub page.
• Contribution: Open to new tasks and agents from users, with documentation provided for extending default agents.

Final Thoughts

• Economic Factors: Pre-built agents that learn user preferences over time are likely to become mainstream.
• Personal Use: Preferences for modalities vary, with voice being useful for hands-free tasks.
• AI Agent Development: Encourages community involvement in expanding tasks and testing agents.

Detailed Instructions and URLs

• GitHub Page: Detailed setup instructions and documentation for Windows Agent Arena are available on the GitHub page (URL not provided in the transcript).
• Commands: Specific commands for running benchmarks and extending agents are mentioned, but exact commands are not provided in the transcript.
• Contribution: Users can contribute by adding new tasks or agents, with guidance available in the documentation (URL not provided in the transcript).

Tips

• Running Benchmarks: It is recommended to run benchmarks in a cloud environment for resource-intensive tasks.
• Developing Agents: Users are encouraged to develop and test their own agents using the provided framework.

(Note: The summary is based on the provided transcript text and does not include any additional information or URLs not present in the transcript.)