From Prompts to Protocols: Benchmarking Multi-Agent AI Frameworks

-

As large language models (LLMs) become more powerful and versatile, the focus is rapidly shifting from single-model prompting to cooperative multi-agent workflows. These frameworks enable agents to communicate, delegate tasks, and complete complex objectives in a structured and scalable manner.

In this post, I benchmark three leading orchestration frameworks: CrewAI, MetaGPT, and LangGraph. Each is tested with three open-source models from Ollama: LLaMA 3, Mistral, and Phi-3. The goal is to evaluate how well each framework handles real-world task orchestration using a consistent and meaningful use case.

Whether you're a developer building intelligent assistants or an architect evaluating agent frameworks for production, this post will help you compare the tradeoffs clearly and practically.

Benchmark Use Case: Report Generation via Agents

To keep the evaluation realistic and reproducible, I used a single well-defined task across all frameworks:

Produce a list of sections for the report titled “The Future of Work in an AI World,” covering:

  • Impact of AI on white‑collar jobs
  • Future of remote work with AI
  • AI and automation in blue‑collar sectors
  • Policy and ethical challenges

This task was modeled using three collaborating agents:

  • Planner: Defines the report structure
  • Researcher: Generates content for each section
  • Composer: Compiles all outputs into the final report

Each framework implemented the same Planner → Researcher → Composer flow.

Benchmark Setup

  • Hardware: RTX 3060 12GB GPU, 64GB RAM, Ubuntu
  • Models via Ollama: LLaMA 3, Mistral, Phi-3
  • Task type: Multi-agent collaboration
  • Evaluation Criteria: Structure, insight, flow, collaboration, creativity

Benchmark Summary Table

Scored on a scale of 1.0–5.0 per dimension using decimal precision for nuance.

Key Findings

  • MetaGPT + Phi-3: Highest overall score. Strong structure, original insights, and smooth collaboration across agents. Excellent planning and execution flow.
  • CrewAI + Mistral: Very readable and structured. Slightly less creative but performed well across all criteria.
  • LangGraph + Mistral: Consistent and accurate. Slightly rigid in voice, but high marks for flow and factual accuracy.
  • CrewAI + LLaMA 3: Underperformed due to weak handoff between agents and placeholder-like output. Reflects either prompt alignment issues or framework limitations in this run.

Conclusion

All three frameworks can support multi-agent orchestration, but your ideal choice depends on the use case:

  • CrewAI: Best for rapid prototyping and simple chains
  • MetaGPT: Excels at clean logic and modular workflows
  • LangGraph: Offers flow control for more complex or stateful tasks

Model choice also matters:

  • Phi-3: Most balanced across all runs
  • Mistral: Fast and sharp
  • LLaMA 3: Good reasoning when agents were aligned properly

GitHub Repository: https://github.com/Algocrat/multi-agent-benchmark

Popular posts from this blog

Setup Pivot4J Analytics to Consume SSAS Multidimensional Models

-
Image
Pivot4J Analytics is a recent replacement of JPivot OLAP Browser. Pivot4J Analytics can be used to consume SSAS Multidimensional Models. Setting up Pivot4J Analytics is very simple and should take less than 40 minutes. These are the steps: A. Configure HTTP access for your SSAS server Follow this TechNet article:  http://technet.microsoft.com/en-us/library/gg492140.aspx B. Install Recent JDK Download recent JDK from  https://jdk8.java.net/download.html And perform default install. C. Setup Environment Variable JAVA_HOME Open File Explorer Right Click on Computer and click on properties Click on Advanced Systems Setting Click on Advanced tab Click on Environment Variables Create a new system variable Enter the Variable name as JAVA_HOME and the value to your jdk bin path (In my case it is C:\Program Files\Java\jdk1.8.0\bin) NOTE Make sure u start with  ;  in the Value so that it doesn’t corrupt the other environment variables which is alread...
Read more

How to write MDX or DMX stored procedures

-
Learn how to create and execute MDX (Multidimensional Expressions) and DMX (Data Mining Extensions) stored procedures in SQL Server to perform advanced data analysis and business intelligence operations. Introduction to MDX and DMX SQL Server provides rich support for data analysis through its support of MDX and DMX languages. MDX is primarily used for querying and manipulating data in multidimensional databases, typically OLAP cubes. On the other hand, DMX is used for managing and querying data mining models, making it essential for advanced analytics and predictive modeling. In this post, we’ll explore how to write stored procedures that utilize these languages within SQL Server. A stored procedure allows you to automate your queries and can be used to encapsulate MDX or DMX scripts for reusable and efficient data analysis. Creating a Basic MDX Stored Procedure MDX queries are often used for complex rep...
Read more

Mastering Quartz.NET: From Basics to a Full-Fledged Job Scheduling Service

-
Quartz.NET is a powerful, open-source job scheduling framework for .NET applications. It allows developers to schedule jobs—pieces of code that run on a schedule—like recurring tasks, background processing, or delayed operations. In this blog post, we'll walk through everything you need to build a full-service job scheduling system using Quartz.NET, including job setup, trigger configuration, persistence, and dynamic API-based management. 1. Getting Started To begin, you'll need to add Quartz.NET to your .NET Core project. We also recommend using its hosted service integration so it runs as a background service in your application. dotnet add package Quartz dotnet add package Quartz.Extensions.Hosting Now, register Quartz in your application's dependency injection system: builder.Services.AddQuartz(q => { q.UseMicrosoftDependencyInjectionJobFactory(); }); builder.Services.AddQuartzHostedService(q => { q.WaitForJobsToComplete...
Read more