Download as PDF, PPTX
![● Get Service by function
○ Web search documentation
○ Direct link
● Get Service endpoint
● Generate code to integrate the Service
● Do
○ Compile/Interpret
○ Run
○ Detect errors
○ Improve code
○ Map output to JSON
○ Validate output
While [hasErrors]
● Enrich the current context from Service](https://image.slidesharecdn.com/multiagents-250515211155-16549b4d/75/Multi-Agent-Era-will-Define-the-Future-of-Software-11-2048.jpg)
![● AI Agent
○ Brain – an AI Model (usually LLM), performing reasoning and planning
○ Thoughts – internal action-observation loop
○ Body – capabilities and tools the agent is equipped with
○ Action – interaction with the environment using tools
● Thought Types (LLMs can only input/output text. Multimodal use tools for images)
○ Planning, Analysis, Decision making, Optimization
○ Self-reflection, Prioritization, Goal Setting
● Tool Types
○ Search, Image generation, Information retrieval, API call
○ Attributes: [Description], [Endpoint], [Typed Arguments], [Output]
● Observation Types
○ Feedback from environment, API/Query responses, Sensor readings, Events
Describe the
Tool to Agent
Prompt the
Agent
Agent
recognizes
the Tool
Agent
generates
code for Tool
Tool outputs
to Agent
Agent
generates
response](https://image.slidesharecdn.com/multiagents-250515211155-16549b4d/75/Multi-Agent-Era-will-Define-the-Future-of-Software-13-2048.jpg)
Uploaded byIvo Andreev
Multi-Agent Era will Define the Future of Software
The potential of LLMs is severely underutilized as they are much more capable than generating completions or summarizing content. LLMs demonstrate remarkable capabilities in reaching a level of reasoning and planning comparable to human abilities. Satya Nadella revealed his vision of traditional software being replaced by AI layer based on multi-agents. In this session we introduce agents, multi-agents, the agent stack with Azure AI Foundry Semantic Kernel, A2A protocol, MCP protocol and more. We will make first steps into the concept with a practical implementation.
More Related Content
Similar to Multi-Agent Era will Define the Future of Software
![[DSC Europe 24] Lana Malic - AI Agents: The Future of Autonomous Decision-Mak...](https://cdn.slidesharecdn.com/ss_thumbnails/lanamalic-aiagents-250212223710-84219c4c-thumbnail.jpg?width=640&height=640&fit=bounds)
Multi-Agent Era will Define the Future of Software
- 1. 10 May |Saturday | Sofia
- 4. Microsoft MVP (AI& IoT) [email protected] www.linkedin.com/in/ivelin www.slideshare.net/ivoandreev Horizon Europe Eurostars -Eureka RIF Cyprus Innovation Fund Denmark Solution Architect External Expert • External Expert
- 5. ● Satya NadellaBG2 Podcast: Agents Will Replace ALL Software ● https://youtu.be/9NtsnzRFJ_o?t=2808 ● Transcript - https://app.podscribe.ai/episode/118164535 ● Large Language Model based Multi-Agents: A Survey of Progress and Challenges ● https://arxiv.org/html/2402.01680v2 ● Hugging Face AI Agents Course ● https://huggingface.co/learn/agents-course/en/unit0/introduction ● Azure AI Agent Service ● https://learn.microsoft.com/en-us/azure/ai-services/agents/overview ● Addy Osmani: Hard Truths about AI-assisted Coding ● https://addyo.substack.com/p/the-70-problem-hard-truths-about ● Multi-agent Sample with Semantic Kernel ● https://github.com/microsoft/ai-developer/blob/main/Python/challenges/Challenge-08.md ● Multi-agent Samples with Azure Assistant API ● https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/Assistants/multi-agent/ ● https://github.com/Azure-Samples/contoso-creative-writer/
- 7. The AI tierwill become the place where all the logic is, people will start replacing backends. (Dec 2024) IT Business Challenges… ● Traditional tools - uncapable of maintaining context and handling complex, continuous interactions well. ● Manual repetitive tasks handling reduces productivity and efficiency. ● Inefficient fragmentation of data and functionality across systems ● Users require increasingly personalized experiences and better interaction ● Processing vast amounts of data manually is challenging ● Organizations need to quickly adopt innovative solutions
- 8. Agents Early PhaseAdoption ● Customers service – early area of agent adoption for cost and efficiency ● Agents will orchestrate business apps ● Agents will take over logic, interact with backends, replace traditional apps ● Agents will help automation of repetitive tasks ● Microsoft is integrating AI agents via connectors (Adobe, SAP, Dynamics) Open Issues ● Business model is uncertain and to be decided ● Demand for compute resources will grow exponentially
- 9. AI Agents … •Are Autonomous • Have Objectives • Interact with Environment & Act
- 10. LLMs can domuch more than prompt-completion: 1. Text understanding 2. Logical reasoning 3. Code generation and debugging 4. Data extraction 5. Multimodal (images, video, audio) – generate/understand 6. Chatbots, recommenders 7. Autonomous multi-agent collaboration • Orchestrate multiple AI systems to collaborate • Automate workflows • Planning and resource allocation
- 11. ● Get Serviceby function ○ Web search documentation ○ Direct link ● Get Service endpoint ● Generate code to integrate the Service ● Do ○ Compile/Interpret ○ Run ○ Detect errors ○ Improve code ○ Map output to JSON ○ Validate output While [hasErrors] ● Enrich the current context from Service
- 12. What is thegeo location of a vessel with IMO number 9644342? Note: Licensing issues may apply when reproducing data from other sources
- 13. ● AI Agent ○Brain – an AI Model (usually LLM), performing reasoning and planning ○ Thoughts – internal action-observation loop ○ Body – capabilities and tools the agent is equipped with ○ Action – interaction with the environment using tools ● Thought Types (LLMs can only input/output text. Multimodal use tools for images) ○ Planning, Analysis, Decision making, Optimization ○ Self-reflection, Prioritization, Goal Setting ● Tool Types ○ Search, Image generation, Information retrieval, API call ○ Attributes: [Description], [Endpoint], [Typed Arguments], [Output] ● Observation Types ○ Feedback from environment, API/Query responses, Sensor readings, Events Describe the Tool to Agent Prompt the Agent Agent recognizes the Tool Agent generates code for Tool Tool outputs to Agent Agent generates response
- 14. Chatbot • Conversation withlimited knowledge of context from window AI Agent • AI does not just talk, it takes actions on behalf of users • LLMs that combine state, tools, autonomous execution. • Combines strategic capabilities with autonomy • Highest level safety, security and responsibility Agent Stack • Involves retaining message history & multiple LLM calls.
- 15. LLM Agents &Coding • AI Agents for Coding • First Impressions and Second Thoughts • The Knowledge Paradox
- 16. The Landscape ○ Salesforcewitness 30% productivity increase and paused hires ○ Stripe laid off personnel reflecting increased efficiency ○ Microsoft expect that 95% of code will be AI-generated AI Coding Assistants (Cursor, Claude Code, Cline, Copilot, Colab, CodeWhisperer) ○ Understand tasks and iteratively solve problems ○ Identify potential issues and fix ○ Write and run tests, inspect UI after rendering
- 17. ● Engineering leader,GoogleChrome ● 16th Rank (44.07K) ● 350K followers ● Integrates Gemini 1.0 & 2.0 AI in Chrome DevTools ● Contributions ○ React ○ Angular ○ Wordpress ○ Next.js
- 18. ● Engineers reportdramatic productivity increase with AI, though software is not noticeably better ● Customers demand more intelligent solutions ● Software quality is not defined by coding speed ● Two types of usage Bootstraping Iterating Purpose • Zero to MVP • Daily Development Usage • Design a concept • Generate initial codebase • Rapid validation • Code completion and suggestions • Complex refactoring tasks • Generate tests and documentation • Troubleshooting
- 19. MVP-Quality Software ● Goodmain-success scenario ● Error messages are confusing ● Edge cases lead to crashes ● Low UX, confusing UI ● Performance & Security issues Hidden Truths ● AI cannot fully understand the customer problem ● AI will not create new algorithms but reuse patterns ● Propose non-existing functions and libraries ● “Looks good” to non-experienced developers
- 20. ● AI toolshelp more to seniors rather than beginners ● When code appears, this does not allow forming of analytical skills ● Senior devs identify the reason because of years of pattern recognition Senior Devs Junior Devs YES • Refactoring AI suggestions • Handle edge cases AI omitted • Add comprehensive error handling • Improve maintainability • Understand the code • Build-up foundation and knowledge • Use AI as a learning tool, not as generator BUT… • Rapidly prototype • Explore alternative approaches • Generate basic implementation to refine • Automate routine tasks • Do not fully understand the generated code • Accept incorrect or outdated solutions • Miss critical security and performance issues • Build fragile systems they do not understand
- 21. Effective Teams ● Setboundaries and establish patterns for AI agents ● Use Human in the loop on the way to AI autonomy AI Coding Tasks ● Automating routines to save energy for complex problems ● Exploration and testing concepts ● Accelerating what we understand and know Agentic Software Engineering ● English as a programing language is a paradigm shift ● Better requirement specification ● Stronger design thinking and focus on validation
- 22. Agent Global Ecosystem •MCP Protocol • Google Agent-2-Agent Protocol • AI Foundry Semantic Kernel
- 23. ● Open HTTPprotocol enabling agent collaboration across domains ● Standardize agents to share tasks, discover capabilities and act ● Collaboration of 50+ technology partners Business Value ● Complements Model Context Protocol (MCP) by Anthropic ○ MCP allows AI to access data and extension its capabilities ● Redundancy – avoid repetitive code implementation ● Compatibility – avoid differences in implementation across tech stacks Key Concepts ● Agent Card - JSON metadata, describes agent capabilities ● Task – the central unit of work to complete end-user request ● A2A Client, A2A Server – Send/receive request for task execution ● Artifact – output, generated by the agent during task
- 24. What is it ●Abstraction layer on tools and services to build AI Agent application. ● Open-source SDK by Microsoft Key Features ● Dev kits for .NET/Python/Java ● Unified connect of sources, models and functions ● Supports A2A protocol to integrate A2A agents ● Supports MCP protocol for access to extensions ● Simplifies multiagent systems development Future Evolution ● Tool ecosystem growth ● Optimized deployment workflows ● Improved tool execution & memory management
- 25. AI Service Connectors– connect AI services from different providers Vector Store Connectors – connect stores(memory) from different providers Functions – arranged in plugin containers Prompt Templates – combine context, instructions and plugin calls Filters – provide security and control over function execution
- 26. From Agents toMulti-Agents • When multi-agents are better • Typical Challenges
- 27. ● Involves multipleintelligent agents interacting and collaborating. ● Decisions and action in typically human domains Benefits o Complex problem-solving o Coordination o Scalability Communication Structures Azure Assistants API (with Agents Support) ● Easy to use framework for multi-agent systems ● Persistent MA systems, virtually indefinite context ● Access and process files, code and functions
- 28. Collective Intelligence ● TraditionalMA Systems use reinforcement learning to learn from offline data ● LLM-MA learn from instant feedback Scalability ● Each LLM-based agent is based on an LLM (i.e. GPT4) and requires huge resources ● Coordination complexity raises significantly with number of agents Multi-Modal Environment ● LLM agents are focused on text-based environments ● Integration in multimodal environment (sensors, video, audio) is a huge challenge Hallucinations Challenge ● Generating a factually incorrect output of an LLM is a huge problem ● The problem could be multiplied in MA scenario ● Solved by Ground Truth (RAG) or Human in the Loop
- 29. Azure AI Foundry– Agent Service • Assistants vs. AI Agents • Service Highlights • Agentic Tools
- 30. ● A fullymanaged service to build, deploy and scale AI agents (Preview Dec 2024) ● Other Frameworks: smolagents, LlamaIndex, Langgraph Assistant AI Agent Definition AI model assisting end-users Smart autonomous microservice w/o interactive UI Function Accurate response on context Planning and reasoning Use Cases Routine tasks • Chatbot, guidance Complex problems • Automation, workflows Trigger Reactive, on request Proactive, works autonomously Interaction Text-based, user-driven Action-oriented Memory Retain within a session Maintains indefinite context in time Complexity One-step: Request – Response Multi-step: Decision - Action
- 31. ● Your dataare: • NOT available to OpenAI and not used to improve OpenAI models • NOT used to train or improve OpenAI Service models • NOT used to improve Microsoft or 3rd party products • NOT available to other customers ● Deployment Model • Deployment location – Global ($), Data Zone ($+10%), Regional ($+100%) • Usage - Standard, Provisioned, Batch API ● Billing • Charged by the usage of the base model of each agent • File search is billed by vector storage • • Agent Service ● Limits - 2’000’000 tokens/min
- 32. ● Knowledge Tools •Grounding with Bing Search – real-time web data ($35/1000 requests) • File Search – augment the model with data from files • • • • Azure AI Search – AI-powered information retrieval • • Action Tools • Function Calling – describe and run external functions • Code Interpreter – interpret and rune Python code in isolation • OpenAPI – connect to external API for interoperability • !!! Azure Functions – triggers and bindings, allow serverless scalability
- 34. ● Clone therepo ○ In PowerShell navigate to target root folder ● Check Python requirements (3.8+) ● Create a virtual environment for dependency isolation ● Activate (switch Python to) the virtual environment ● Install requirements (55 Python packages) ● Install Jupyter Lab in the environment ● Create Azure OpenAI service ● Create deployments of required OpenAI models (Azure AI Foundry > Deployments) git clone https://github.com/Azure-Samples/azureai-samples.git cd azureai-samples/scenarios/Assistants/multi-agent python --version python -m venv env envScriptsActivate pip install -r requirements.txt pip install jupyterlab
- 35. ● Configure theenvironment variables in the .env file ● Get Endpoint, Key, API Version, Deployment ○ GPT4_DEPLOYMENT_NAME (gpt-4o model) ○ DALLE3_DEPLOYMENT_NAME (dall-e-3 model) ○ GPT4VISION_DEPLOYMENT_NAME (gpt-4o model) ● Start Jupyter Lab from the project folder and environment ● Load the environment explicitly jupyter lab load_dotenv(dotenv_path="sample.env")