A few weeks ago, I wrote about how we should design any system from one logical level up, i.e. considering the environment of our system. A Redditor offered an interesting comment that this approach was contrary to how anything natural evolved. Natural evolution is always bottoms up and that seems a lot more flexible.
This is certainly a valid observation and it got me thinking. Natural evolution happens bottom-up with everything co-evolving at the same time. Why should we not define and build the lowest levels first?
The purpose of a system
I have come to the conclusion that the difference is one of “purpose”. Purpose, in complex systems, is a decentralized thing, its shape differing from actor to actor. But the goal is always the same – an actor acts a certain way or pursues a certain goal because that, according to their limited mental model of the system, will allow them to perform better in their environment.
The existence of a purpose implies the existence of one or more actors. For nature to have a purpose requires something to exist outside of nature. This supernatural entity would be contending with a supernatural environment. Without getting onto divine turf, a simpler conclusion is that nature has no definite purpose.
Natural systems evolve without a sense of purpose or definite objective. Every element of the natural system tries to perpetuate itself in a changing environment by adapting. The system, on the whole, tries to attain a stable equilibrium regardless of what the equilibrium looks like. Any stable state will do – nature has no opinion on the quality of the outcome. It is acceptable in the natural world for entire evolutionary hierarchies to collapse if they no longer fit in with the environment.
In man-made systems like large organizations or software architectures, we are not quite as generous. These systems exist to fulfill a certain role, and we intervene in them with clear intent. This is the essence of strategy – defining goals and the action needed to achieve these goals. Evolution in man-made systems doesn’t run rampant, it is constrained to proceed in the directions which potentially lead to the outcomes we want – at least as far as we can tell at the moment. We want to minimize large failures that would result from completely uninformed trial and error.
This is why system design should be done from one level up. First, we visualize the effect we want to create (our purpose), and then we take an action that is likely to attain that objective. In man ade systems, this is evolution.
This is what makes building shared context in teams so critical. A shared understanding leads to shared motivation and intent of action. The more actors share the same mental model, the more likely it is that an action can be pulled off successfully. Building the shared context is akin to extinction in the natural world, we are pruning those paths of evolution which might lead to an unsatisfactory equilibrium for us as a team. This is the “purpose” of the team. Hence the modern insistence on aligning engineering teams with top-level business objectives. If the teams are aligned to the organization’s global “purpose”, they are less likely to be focussed on local maxima when they operate.
Everything is bottom up
This explanation, though practically useful, is philosophically misleading. Human and natural systems are not disconnected. All in all, the Redditor was right. Everything happens bottom-up. Even that is incorrect. Better to say that everything happens all at once. Every component of a system, whether above or below, reacts to changes around it. The confusion only exists because we subconsciously draw a logical boundary that defines up or down.
It seems like we are building top-down because we are looking at only a part of the system to make a tactical decision. As an architect, I design from one level up because that allows me to impress my intent upon that specific neighborhood of the software system. I first define a boundary (consciously or subconsciously) inside which I want to take top-down action. Now I can be system-minded and look just outside the boundary to visualize what might be going on “outside”. The act of defining that scope gives the illusion of control, top-down action, and purpose.
But seen from the outside, I am just another actor responding to my environment and motivations. This is the essence of Conway’s Law – regardless of my intentions, there are things I do not know and hence alternative actions that I cannot take.
On the grand cosmic scale, we are all just doing the best we can to improve our situation in an un-opinionated universe. And that is good.
Read Next: Control and chaos in platform systems
If you liked this, subscribe to my weekly newsletter It Depends to read about software engineering and technical leadership
