Architectural Decision Making: How Senior Engineers Approach Design and Scalability
Brent DallingPublished: April 12, 2024
In software development we are faced with a plethora of options for getting the job done. You can give a room of 10 developers the same set of simple requirements and get 10 different solutions with wildly different technologies. But, how do we know any of those solutions are better than another? Could the wrong solution impact your business and cost tens of thousands of dollars in remediation costs. Could the wrong solution cause a security incident because the underlying technologies failed to account for secure coding practices or have mechanisms in place to detect supply chain software attacks? Furthermore, could the solution you do choose to move forward with scale to meet the demands of the business? If it can’t, how do you scale? Will it costs tens of thousands to rewrite the software to support scaling to meet demands?
These are all questions that make sense to ask ourselves as we receive those initial requirements. We should be looking for reasons to rule out technologies based on known unknowns and known requirements. What do I mean by known unknowns? Simple. These can include the number of active daily users, the size of files being uploaded, etc. Knowing these known unkowns and keeping these in mind while designing a software solution is paramount to project success. Additionally, knowing our requirements allows us to determine if a technology will meet the requirements. For example, if your project requires database replication and encryption of data in transit and at rest, then you might look into Sequel Server (MSSQL) over MySQL. But, if you just need to store some simple data then it might make sense to use MySQL over MSSQL due to costs and over engineering concerns.
In this article, we’re going to take a deep dive and exploration into the role of senior or staff engineers within the software design and architecture phases. I think you’ll be surprised how lacking most companies are in this department. And how much it can really cost further on in the development and support process.
Understanding Software Architecture
The software architecture of a system represents the design decisions related to overall system structure and behavior. Architecture helps stakeholders understand and analyze how the system will achieve essential qualities such as modifiability, availability, and security.
-- Software Engineering Institute, Carnegie Mellon University
It’s important to have a common definition when we approach understanding the architectural design process, it’s uses, and it’s drawbacks. The quote above is by the Software Engineering Institute at Carnegie Mellon University. The institution has a reputable standing and we will be using their definition of this process as we move forward.
When we consider the definition and begin decomposing it’s meaning we first see the words “design decisions”. In software this usually refers to the programming languages, the databases, the frameworks, and other integral parts of the software system. Specifically, design decisions are a sum of all engineering decisions in a given software system.
So, when designing a solution we are making decisions on what database to use, what language to use, what frameworks to use, and even what data structures and design patterns to use. The rationale behind these decisions are important. Did we choose a database because the senior engineer just likes it? Or, did we choose the database because it best reflects the requirements of the project? Did we choose React because the team likes it? Or, did we choose React because it was the best fit for solving the problems at hand? Did you choose an adapter pattern approach to supporting payment providers because it made sense? Or, did you choose the adapter pattern approach because you read about it recently?
Engineers should never be choosing technology based on preference alone. That being said, it can be hard to discern the motivations behind a senior member of the team and their design decisions. Does the senior engineer like MSSQL because it has worked for them for numerous use-cases of the years and they have deep and rich experience in it? And it just happens to solve the core data-storage requirements? Or, because they like MSSQL and have used it exclusively over the years and refuses to use anything else? We can typically better control these decisions through peer reviews within the architectural design process.
The process typically looks a lot like the project management process. You start by gathering the core requirements and asking question to reduce chances of scope creep or missed deadlines. During this process you’re uncovering what kinds of data will be stored and processed by the software system, how do the users expect to interact with the system, what technical level do the users have, what problems are we aiming to solve, etc. This requirements discovery process allows a chance to document the requirements and potential solutions the team crowdsourced. Then, the approach I prefer, you can have multiple engineers go out and and research how a given solution to a requirement can be fulfilled while keeping in mind other solutions. At the end you compile the results and have a architectural design meeting where the team discusses the findings. Ultimately you place control over the decisions on a select few engineers to reduce the changes of deadlock.
Essentially, the process should be data-informed rather than preference informed. The motivations behind selecting specific solutions should be peer reviewed to eliminate bias and ensure correct understanding of the requirements and to validate whether or not they do meet them. It is dangerous to the health of the project and it’s stakeholders to progress soley on feelings and preferences as it can introduce unneccessary complexity and risk.
The Role of Senior Engineers
The job title “Senior Software Engineer” isn’t actually well defined at an industry leel. Every company has different expecations and requirements of their senior engineers. For example, one company might consider a senior engineer to be someone with a decade of experience, deep domain knowledge, excellent technical expertise, managerial skills, and more. Another company might just look at productivity, self reliance, ability to learn, etc. Some companies do not even have this job title. Everyone on the team is just a software engineer with varying levels of influence. For this reason, we can’t effectively define a software engineer. To do so would be a lie. And, I’m not too fond of those. Especially when it can hurt other people, damage expectations, or lead to a collapse in communications.
But, for the purposes of this article, we’re going to lead with someone who can make experienced backed decisions, has deep technical capabilities, mentorship expectations, and at least mediocre domain knowledge. Essentially, what you’d liekly find in a senior level candidate in most non-FAANG companies.
Let’s define experience-based decision making and theoretical knowledge based decision making.
Experience-based decisions can be defined as decisions emanating from direct or vicarious reinforcements that were received in the past. Typically, in experience-based decision tasks an agent repeatedly makes choices and receives outcomes from the available alternatives, so that choices are based on past experiences, with no explicit description of the payoff distributions from which the outcomes are drawn.
-- Eldad Yechiam1, and Itzhak Aharon
Knowledge-Based Decision-Making (KBDM) in management is a decision-making process that uses predetermined criteria to measure and ensure the optimal outcome for a specific topic.
KBDM is used to make decisions by establishing a thought process and reasoning behind a decision. It gathers vital background essentials to collectively increase understanding about a topic or agreed criteria
-- Wikipedia
Okay, so we can see that Knowledge-Based Decision-Making is actually better in most cases based on my previous arguments in this article. But, if we stop to think for a moment, we understand that experience does lead to knowledge. For example, a freshly minted college grad might know that Postgres has more features and is faster than MSSQL. But, an engineer with experience might know that when you factor in the encryption, logging, and replication requirements that MSSQL is easier to setup and manage. Both use knowledge gained from some form of education.
However, one is actually theoretical in this instance and another is based on past data. We actually leverage both forms of decision making when we consider things we’re an expert at. For example, you choose to rinse dishes before putting them in the dishwasher. You know the dishwasher will wash it. But you also know that sometimes food can get stuck and slightly baked onto your dishes. So while the dishwasher will do most of the work you still need to rinse. Someone new to owning a dishwasher might only consider that the dishwasher is built to wash the dishes and will only later learn that they still need to rinse their dishes for the optimal outcome. This can be applied to our architectural decision making. Knowing a good solution in theory is different that knowing a good solution and having historical data to back it up.
This is where senior engineers shine. They don’t just know several data structures and design patterns. They’ve used them. They’ve made mistakes. They know how to more easily find and correct problems. They know how to do more with what they have. They’re not necessaritely smarter than newer devs. They’re just experienced.
This extends itself to domain knowledge that senior engineers tend to have that junior developers don’t. They know the software system, clients, needs, etc better. This doesn’t always apply though. So take this with a bit of salt. A senior engineer on an existing project might know that adding a new payment provider would prove difficult because the original development process only accounted for one provider. And that to include the new provider, and support future providers, they’ll need to refactor to an adapter pattern. The junior developer could probably know this after a little investigation. But the senior developer with domain knowledge would know this as it’s brought up. Additionally, the senior engineer likely knows how much of an impact the change will have to the software system.
So, in short, senior developers bring years of experience, domain knowledge, and systems understanding to the table. When architecting a new solution they’re invaluable to the process. As they’ve been around the block a few times and will have valuable insights. However, years of experience doesn’t equarte to seniority. You can have a mediocre developer for 10 years and another for 3. And the 3 year experienced developer will outperform the 10 year developer at every metric.
