The Role of an Agile Architect

In his great book “Building Microservices,” Sam Newman makes the following observation about IT Architects in his chapter on “The Evolutionary Architect”: “Architects are responsible for a lot of things. They need to ensure there is a set of principles that guide development, and that those principles match the organization’s strategy. They need to make sure as well that these principles don’t require working practices that make developers miserable.”

There’s an enormous amount of wisdom in those two sentences. They define an architect as someone who has to think about development principles, but also about implementing those principles. Finally, it also requires that the architect work within the guidelines of the organization’s strategy — an architecture must serve the purpose of helping teams build systems that provide business value.

It’s that particular role of Architect as boundary setter that I’ve found to be the most important. In many ways, it’s like the job that the farmer or rancher used to have when they first settled a new piece of land. They had to first survey the terrain and see what’s there. They then had to measure and determine where the boundaries should lie. They had to build the fences. The fences did two things — they kept the livestock in, but they also kept wild animals out. It gave the herd enough space to grow, but not so much that they would fall into ravines or other dangerous areas. Likewise, an agile team should be able to push all the way up against the fences of an architectural boundary, but not cross over them — it’s always the balancing act that Newman in being able to set guidelines while not making your developers miserable. When the herd grows and requires more pasture to graze, the farmer can push the fences further out but that growth is ultimately limited by the property line. Similarly, when the developers need to go beyond an architectural boundary and have good reason to, that can be worked out with the architect as long as it does not conflict with the organization’s strategy and the overall goals of the farm (i.e., the property line).

The first boundary that the architect has to set up is what patterns and idioms the team will use in building the application. To a large extent, this requires that an architect not only understand the importance of patterns but have internalized a catalog of patterns that they can recognize and put into practice when needed. The solutions should not be new insomuch as the issues we solve are not new. So, for instance, if we are building a web application to show a runner all of their different runs that have been tracked by a GPS-enabled “smart shoe,” then that is a fairly well-known and understood type of application. There is no need to create new patterns for building these when basic patterns such as Single Page Applications (SPA’s), microservices and Backends For Frontends (BFF’s) exist.

However, when the problem isn’t building a web application, but building a system that has to provide suggestions for improvement to that same runner’s gait based on changes in pace over terrain or how hard their shoes are striking the pavement, then an entirely different set of patterns apply. There, you have to think about how to ingest all of the data coming from the smart shoe — that brings in patterns around building IOT apps. Likewise, the processing of gigabytes of raw data into actionable recommendations may require the application of the scatter-gather pattern as implemented in Apache Spark, together with the implementation of Machine Learning patterns implemented in Tensorflow.

The second boundary that an architect has to set together with the team is the choice of languages and libraries that the project will be written in. It’s so closely tied to the first boundary that it’s sometimes hard to separate the two, but a disciplined look at this question can often result in better results for the team.

The major question to address here is — why would a team, especially an established team, even want to change the language and tools that they’ve been using for years? Perhaps bad habits should be broken. For instance, in one team I worked with they had been using Java and Spring for a long time and had picked up poor practices that needed to be abandoned. This particular team coded a lot of GUI logic and HTML directly in Spring MVC and never really developed an aptitude for building good APIs. They benefitted from having to learn new habits while learning a new language and libraries (JavaScript and node.js). Polyglot programming is another consideration. More and more teams want the flexibility to choose the right language for the job. For example, a common consideration is using node.js for I/O heavy operations and Java for CPU-intensive operations.

The third boundary that the architect needs to set is what are the services and platforms that will be used by the team. This should be closely tied to the patterns and libraries that you’re choosing for the project — you don’t want to find out you have an impedance mismatch between what you want to accomplish and what your chosen platform allows you to do. As an example — you will probably have an easier time implementing a microservices architecture if you do so in a Container-as-a-Service (CaaS) or Platform-as-a-Service (PaaS) environment than you would by working with native VM’s in a public or private cloud environment, even with the help of an automation library. I’ve seen too many teams get caught up in building script after script for VM-level deployment automation that would have time better spent in learning and using Kubernetes or Cloud Foundry.

The final boundary ties closely into the previous one — and that is what the choice of development, testing, and monitoring tools will be, and how you will set up your CI/CD pipelines. This set of decisions is wide-ranging because it touches on many different areas of both sides of DevOps. I’ve seen far too many teams become very caught up in optimizing things for developers while ignoring decisions that would make operations easier — such as getting developers to agree on operations decisions such as a common logging platforms, common logging formats, a consistent approach for correlation IDs through microservices, synthetic transactions, Application Performance Management (APM), etc.

As an organization grows, it is possible, in fact likely, for the need for a level of specialization to arise. For instance, as the number of teams increases, you may find that it becomes advantageous to have a number of devops architects who specialize on the operations side that are tightly aligned with the DevOps and SRE (Site Reliability Engineering) guilds within your organization. In fact, these specialized architects are often good guild leads for different guilds aligned with each of the different boundary areas.

Probably the most common specialization is that of an application architect; someone who knows how to build different types of applications and knows the patterns for that application type deeply; that’s often connected with a deep knowledge of a particular language and library set for implementing those patterns. A platform architect is one who specializes in Platform or Services; this may be someone who is a deep expert in Virtualization technologies like VMWare, or in Container technologies like Docker and Kubernetes. Likewise, even specialization for specific services, such as Database services are specializations that a larger organization will usually find a need for. An Enterprise Architect is someone who doesn’t really have a specialization but can work effectively with teams on any of the different areas by establishing boundaries and enablers to ensure that overall organizational goals are met by their holistic “system of systems”. Any organization should have a mix of the more specialized architects and the more generalized enterprise architects in order to make sure that teams can be given the appropriate level of guidance at the right time.

Depending on their specialization, an architect’s involvement during the full lifecycle of the project may vary. An application architect, for instance, is typically engaged for the entire duration and often stepping in to write code to demonstrate a concept or illustrate an architectural boundary. The DevOps/SRE architect is front-loaded to ensure the DevOps toolchain, and delivery pipeline are set up for the build squad. The architect continues on the project with a focus on SRE tasks. Enterprise architects are in a consultative role and provide the other specialized architect’s guidance when necessary.

Finally, there are some specific traits that all architects, regardless of their specialization or lack of specialization need to have.

Architects need to be visionaries. A key job of any architect is to outline the future. That means they have to be able to inspire the members of their teams to push out towards the boundaries that they’ve set, and also to challenge those boundaries when they believe they’ve found a better way of doing things. Most importantly, we believe the role of any architect is to teach. This is one reason why a simple notation, for instance, one that allows just a few simple diagrams such as (Simon Brown’s C4 model, our own Services, and Container diagrams) are important when conveying complex information in a simple way.

Architects need to act as mediators. Key to this teaching role is helping to resolve technical conflicts among team members and across teams within a tribe. Part of this is having an understanding of where the disagreements have arisen in the past, giving the architect a better sense of where they will arise in the future. A key part of outlining the future is to make sure that you’ve anticipated the biggest possible issues and make sure that you have a plan for settling disagreements about them as much as possible before they start.

Architects need to have strong communications skills. The architects need to be able to communicate their architecture to the developers. They must also be able to present it to an Architecture Review Board/Committee and clearly justify why they made certain architectural decisions. Being able to communicate complex technical concepts at a level that can be understood by individuals that are not experts in the particular technology is a key skill required by all architects.

Architects must be hands-on. There is nothing that is so stereotyped as a “paper” architect who is out of touch with the realities of development. Architects in the modern agile custom software development world cannot succumb to this stereotype — they have to be able to code and contribute real running code to projects. However, they have to remember that this code is meant to be only part of the overall solution, not all of it. Generally speaking, an architect should strive to create just code to inform and explain the important patterns and concepts, and to set up the boundaries, and no more. It is also important that there must be words around the code; while code is explanatory in and of itself, it’s not enough — code explains the “how” of a solution, but doesn’t explain the “why” or what the alternate choices were that were examined and rejected. The best place that we have seen for this kind of information is in project documentation that is part of the same GitHub repository as the source code. When looked at this way the words that explain the “whys” of an architecture often just as, if not more important than the code examples themselves. An Agile Architect (like a good farmer) is able to “get their hands dirty” by going deep where necessary — which can include coding, setting up a CI/CD pipeline, etc. The Agile Architect gets hands-on in areas of key risk and importance in order to ensure smooth delivery of a sound and safe system.

This article was written together with IBM Distinguished Engineers Darcy Lalor and Shahir Daya. A shorter version of this is available on the IBM Garage Method site.