Beyond Agile: How to Develop as a Swarm.
The last two decades were a triumphal march of the agile development. The agile revolution's backbone is the Agile Manifesto from 2001 (https://agilemanifesto.org). It was cited more than 2000 times in scientific papers. Yet, this concept is not the solution for every development problem. Such frameworks (e.g. Scrum) have limitations regarding development of physical products. Overcoming those challenges would result in increased development in aerospace and defense industries . Thus, the benefits affect potentially a 200 billion euro market in revenue (in Europe alone). Firstly, this blogpost identifies two critical issues of agile product development. Then, it proposes a two solution inspired by swarm theory.
What limits agile product development?
The first challenge concerns scaling agile teams. Exponentially increasing social complexity sets a „natural limit“ of teams in general. In agile frameworks the lack of organic growth mechanisms determines this limit. The second challenge is about documentation. This issue finds its roots in the agile principles. I want to present those factors in Scrum, the most known agile framework for teams.
How agile fails to scale
The Scrum Guide proposes a perfect size for a team between three and nine members. The reason is the coordination effort. The framework does not adapt with the size of the team (no organic growth). So, it has a natural limit. If you have a very good Scrum Master, you could probably work with a large team of 15 to 20 members. The Scrum Master could help coordinate this large team. However, this would oppose the principle of self-organization from the Agile Manifesto.
Scrum presents a workaround to enable larger teams or enterprises. It introduces the "Scrum of Scrums" in such cases. Then you can have up to 9 teams with 9 members each, resulting in 81 engineers and 9 scrum masters. Yet, those teams are separated agents and do not work on the same backlog (task list). Thus, agile teams have a critical point of weakness when you need a large number of different specialists to work together.
How agile fails to document
The second issue with agile development is about the documentation. The Agile Manifesto explicitly says „working software over comprehensive documentation“. Unfortunately, product documentation is a crucial ingredient for certification of certain products. This is especially true in the aerospace and defense tech. A working product without a clean documentation does not sell - it simply does not get certification from authorities.
Every framework claiming to be agile focuses on the working product. Thus, the team reaches a point of tough choices: do we make it work or do we document it right? This questions is a direct result of the agile mindset. Also, it implements a "single point of failure" on certification-dependent products.
How could the swarm solve the issues?
In nature, we identify different types of swarms (e.g. birds or fish). The spectrum of cooperative behavior goes from pairs, to groups, to packs, to herds and finally swarms. It is broad and the definitions are not clear. So, by using the term "swarm" I will refer only to swarms of social insects. No other animals show so well how to solve complex tasks far beyond the cognitive capabilities of the individual. The focus will lie on solving the limits presented before by using swarm theory.
Scaling in swarm development
The size of a bee colony varies during the year. At the lowest point (autumn, winter) the minimal number of bees is around 5,000 for one nest. During spring and summer it easily reaches 20,000 bees and big swarms have as many as 40,000 bees. Scaling from 5,000 to 40,000 team members seems rather unrealistic in human teams. The goal should be to scale from 50 to 400 engineers for specific projects. This is possible by implementing key aspects of swarm behavior. The development process must adopt work distribution, decentralized decision making and particle optimization. This would enable organic growth of development teams and solve the first issue.
Documenting in swarm development
Swarm intelligence, specifically swarm communication, provides the solution of the second issue. Obviously, social insects do not document their work for certification. However, they have sophisticated communication within the swarm and exchange constantly useful communication. Actually, bees can take decisions as a group. This includes choosing where to build the new nest. They succeed without having complex linguistic capabilities like humans.
To use this, we must reverse engineer the communication methods of swarms. Then we must replicate them for humans and add one specific ingredient: recording. Humans have limited information input channels and need recorded information. The goal is to create the documentation by communicating like a swarm and recording it.
Conclusion: Objectives of further research
The agile product development struggles with two issues: scaling teams and documenting products. In this blogpost I outlined two possible solutions inspired by swarm theory. However, using swarm theory-based cooperation methods and swarm intelligence is too vague. I conclude that those solutions are merely hypotheses. Consequently, I will focus my future research on this matter. At the same time, I will make this my PhD thesis "Beyond Agile: How to Develop as a Swarm".