The Three Pillars of Agile Quality and Testing

By Robert Galen


A few years ago, I entered an organization to do some agile-focused coaching and training. From the outside looking in, it was a fairly mature agile organization. They had internal coaches in place, had implemented Scrum while also leveraging Extreme Programming technical practices for a couple of years, and seemed to be fairly committed and rigorous in their application of the methods.

It was a global financial firm that delivered their IT projects via highly distributed teams. There were a wide variety of tools in place for application lifecycle management (ALM), software development, and software testing support. In my first few days, everything I encountered, while albeit in superficial detail, just felt like a mature agile organization and I was pleasantly surprised. Heck, I was impressed!

For the purposes of this article, my observations will shift to being quality, testing, and tester centric.

pillars of agile testing
The three pillars of agile testing © Eimantas Buzas /

Narrow a Focus

One of the things I noticed is that the firm had gone “all in” on Behavior-Driven Development (BDD) leveraging Cucumber. They had invited in several consultants to teach courses to many of their Scrum teams and, as a result, everyone was excited and “test infected”. Teams were literally creating thousands of BDD-level automated tests in conjunction with delivery of their software. From their teams’ perspective, there was incredible energy and enthusiasm. Literally everyone contributed tests and they measured the number of increasing tests daily.

These tests were executed as part of their continuous integration environment, so there were visible indicators of coverage and counts. It was truly visual and very inspirational. And I could tell that everyone was focused on increasing the number of automated tests – so there was a unity in goals and in each team’s focus.

However, a few days into my coaching, I was invited to a product backlog refinement session where a team was writing and developing their user stories. What I expected was to simply be an observer. What actually happened is that I soon realized that the team did not know how to write a solid user story. They could barely write one at all, which shocked me. After this gap became clear, they asked me to deliver an ad-hoc user story writing class for them. Afterwards, the team was incredibly appreciative and everyone seemed to ‘get’ the place that stories held in developing their BDD-based acceptance tests.

Over the next several days, I started to realize something very important. The organization was at two levels when it came to its agile quality and testing practices. People were either “all in” or they were “unaware or under-practicing” specific techniques. For example, they were “all in” on BDD and writing automated BDD (Cucumber) tests and on continuous integration. However, they struggled mightily with writing basic user stories and literally had no clear or consistent Definition-of-Done.

I also realized that this “seesaw effect” of focusing on a handful of the more technical practices was doing them a disservice. Why? Because it is actually the interplay across practices that influences the effectiveness of your agile testing and the product impact of your quality practices. I prepared a model for them to illustrate the balance that I think is critical in agile quality practices.

I called it The Three Pillars of Agile Quality and Testing, and I began to coach a much more nuanced, broad, and deep approach for their teams.

While this is more of a strategic play and a longer-term focus, the discussions and the changes they drove had an immediate impact on the organization. People started to connect the dots between the technical and softer skill practices. They became much more aware of the interplay across practices and how this drove an improvement in quality results.

I want to share the “Pillars” in this article in the hope that it will help your agile quality strategy development as well.

A Bit More on Strategy

A really important sub-text to the development of the Three Pillars is my ongoing observation that agile organizations at best might have a strategy for their overall transformation. But rarely do they develop or are able to articulate their overall strategy for agile quality and testing.

I guess the assumption is that these aspects are “along for the ride” as part of the development-driven strategies in their agile adoption. But I could not disagree more with that point of view. I believe that calling out, making transparent, and focusing on your quality and testing agile strategies strongly aligns with the Agile Manifesto and makes for a much more rigorous and successful agile transition.

The ultimate value of the Three Pillars is to help organizations think about, articulate, and realize their agile strategies in these areas and, most importantly, to hopefully achieve evolutionary balance.

The Three Pillars of Agile Quality and Testing

As I have said, the driving force behind my creating the Three Pillars was organizational quality imbalance. As I observed what was happening at my client, I clearly recognized the imbalance. However, it was unclear to me how to create a model that would help them. I eventually came upon the following three critical areas, or ‘Pillars’, where I tried to categorize crucial tactics, strategies, and techniques that I have found helpful to agile teams as they create a broad and deep supportive structure for their product quality and testing activities. Here are the pillars at a high level:

1. Development and Test Automation: This pillar is the technology side of quality and testing, and it is not simply focused on testing and testers. It includes tooling, execution of the automation test pyramid, continuous integration, XP technical practices, and support for ALM-distributed collaboration tools.

Often it is the place towards which organizations gravitate first – probably because of our generic affinity for tools solving all of our challenges. An important way to think about this pillar is that it is foundational, in that the other two pillars are built on top of the tooling. And organizations often underestimate the importance, initial cost, and ongoing costs of maintaining foundational agility in this pillar. Continuous investment is an ongoing challenge here.

Finally, this pillar is not centric to the testing function or group. While it includes testing, tooling, and automation, it inherently includes ALL tooling related to product development across the entire agile organization. It provides much of the ‘glue’ in cross-connecting tools and automation towards efficiency and quality.

2. Software Testing: This pillar is focused on the profession of testing. On solid testing practices, not simply agile testing practices, but leveraging the teams’ past testing experience, skills, techniques, and tools. This is the place where agile teams move from a trivial view of agile software testing (which only looks at TDD, ATDD, and developer-based testing) towards a more holistic view of quality.

It is a pillar where the breadth and depth of functional and non-functional testing is embraced. Where exploratory testing is understood and practiced as a viable testing technique. It is where the breadth of non-functional testing is understood and applied to meet business and domain needs, including performance, load, security, and customer usability testing.

By definition, this is where testing strategy resides, where planning and governance sit, and where broad reporting is performed. I am NOT talking about traditional testing with all of its process focus and typical lack of value. But I AM talking about effective professional testing, broadly and deeply applied within agile contexts.

3. Cross-Functional Team Practices: Finally, this pillar is focused on cross-team collaboration, team-based standards, quality attitudes, and, importantly, on building things properly. Consider this the soft-skills area of the three pillars, where we provide direction for how each team will operate – consider them the “rules of engagement”.

For example, this is the place where good old-fashioned reviews and inspections are valued. This would include pairing (across ALL team members), but also slightly more formal reviews of architecture, design, code, and test cases. It is a place where inspection is performed rigorously, as established in the teams’ Definition-of-Done. Where refactoring of the code base and keeping it “well kept” is also of primary importance.

Speaking of Definition-of-Done, this is the pillar where cross-team physical constraints, conventions, and agreements are established. But, more importantly than creating them, it is where the team makes commitments to consistency and actually “holding to” their agreements. Another important focus is on group integrity in conducting powerful retrospectives and fostering continuous improvement in the long term.

agile quality testing

Foundational Practices

But beneath the Three Pillars are some foundational principles and practices that glue everything together. For example, taking a whole-team view of quality and testing, where it is not just the job of the “testers”, but of everyone on the team. I still find far too many agile teams that relegate the ownership of quality and testing only to testers. Continuously challenging this position and coaching the teams toward whole-team ownership is an ongoing focus.

Another foundational area is metrics. We all know that what you measure drives the behavior of the team. As we move towards agility, we need to begin measuring the entire team results rather than functional results, and even showing care when measuring the team, such as understanding that velocity is probably not the best metric to measure a healthy team.

One of the core components of the Three Pillars foundation is a thread that permeates through the pillars and the foundation. It embraces the core idea that each agile, self-directed team has a basic responsibility to build the right things (customer value) and to build them properly (design, construction integrity, and quality).

Figure 1 is an overview of the types of activity and focus within each pillar. This is a high-level view and there are important nuances that are missing – mostly due to a lack of space.

Figure 1. High-level View of the Three Pillars

Cross-cutting Strategy

Beyond the individual pillars, the value resides in cross-cutting concerns. I will go back to my original story to help make this point. My client was advanced in BDD practices, but struggling with user story writing, or even understanding “the point” of a user story.

The results would have been better if they had made the following cross-Pillar connections:

In Pillar #1 – Behavior-Driven Development (BDD) and Acceptance Test-Driven Development (ATDD) are mostly technical practices. They focus on articulating user story acceptance testing in such a way as to make them automatable via a variety of open source tooling. Unfortunately they have an underlying assumption that you understand how to write a solid story.

In Pillar #2 – One thing I did not mention in the story was that every team had a different view of what a story should look like and the ‘rules’ for writing effective stories. There were no norms, consistency rules, templates, or even solid examples. A focus on the software testing aspects of pillar two would have established these practices, which would have significantly helped their teams.

In Pillar #3 – An important aspect of the user story that my client failed to realize was the “conversation part” of the story. If you reference the 3-Cs of effective story writing as a model, one of the Cs is having a conversation about or collaborating on the story. It is the most important ‘C’ if you ask me. It is where the “3 Amigos” of the story (the developer(s), the tester(s), and the product owner(s)) get together; leveraging the story to create conversations that surround the customer problem they are trying to solve.

Do you see the pattern in this case?

You cannot effectively manage to deliver on agile quality practices without cross-cutting the concerns and focus. In this case, effective use of user stories and standards, plus BDD and automation, plus the conversations needed to cross all three pillars. It requires a strategic balance in order to implement any one of the practices properly.

I hope this example illustrates the cross-cutting nature for effective use of the Three Pillars, and that you start doing that on your own as well.

Wrapping Up

This initial article is intended to introduce The Three Pillars of Agile Quality and Testing as a framework or model for solidifying your agile quality strategies and initiatives.

I hope it has increased your thinking around the importance of developing a balanced quality and testing strategy as part of your overall agile transformation. As I have observed and learned, it does not simply happen as a result of going agile, and most of the teams I encounter are largely out of balance in their adoption.

I hope you find the model useful and please send me feedback. If there is interest, I may explore more specific dynamics within each pillar in subsequent articles.

Stay agile my friends,

Agile Testing Trainings

ISTQB ® Certified Tester – Foundation Level Extension, Agile Tester

iSQI ® CAT – Certified Agile Tester

Leave a Reply

Your email address will not be published. Required fields are marked *