AutoGen isn’t Practical for Real-world Applications, Yet

Why AutoGen is Impractical with the Current Generation of LLMs

Image by author — generated with DALL.E 3

Intro

I am steering clear of using AutoGen in any customer-facing apps. I’ve come to this conclusion after building several AutoGen prototypes. I’m not writing merely to demonize the framework. I have enjoyed using it, and it is already an excellent tool for research, learning, and hobby projects. Instead, I’m raising concerns about using AutoGen and similar multi-agent frameworks with the current generation of LLMs.

I’ll demonstrate with an experiment; let’s get into the details.

The Experiment

I set up a multi-agent framework with AutoGen for multi-hop question answering. The idea behind this is simple: multi-hop questions appear in many domains, such as law, finance, and research, making it an excellent and relevant use case. Basic retrieval augmented generation (RAG) is often an inadequate solution; however, a multi-agent approach might give more flexibility.

multi-hop question answering involves answering complex questions that require gathering and synthesizing information from multiple data sources or pieces…