thoughts from the test eye

Preparation

The hardware setup of the testlab was 1 server and 4 laptops brought by Compare TestLab, 5 laptops by Adecco IT Konsult and finally 2 laptops brought by James Lyndsay. Many participants brought their own laptops. Before the conference I worked some on setting up a wiki, bug system etc. I tried to find a balance in not over-planning, but noticed later on that I had written down too little information making it hard for the participants in getting started and in knowing the overall missions.

By having a base of 11 laptops we were able to bring at least a few testers into the lab. Many thanks to Compare TestLab for sponsoring with both hardware and someone assisting us with the testlab environment, namely Torbjörn Wiger. Big thanks to Adecco IT Konsult for letting me utilize their laptops in the final minutes of preparation. Big thanks to James Lyndsay for being part of this confused setup and being patient with acting in the unknown and unplanned.

We had selected three applications as part of this setup, namely Sikuli, CiviCRM and Red Notebook. They were chosen because they were fairly buggy, open source and would make a good subject for interesting testing.

The planned agenda for the testlab had been published here with a bit more detail on what I had in mind here. During the conference the agenda changed and its content became something different, still within the planned concepts.

When we arrived to Let’s Test we started to setup things up almost immediately. I got help from Steve Öberg with some installation and by Torbjörn Wiger to set the server up. Doing everything on your own is hard so the help was greatly appreciated. I did not feel that we were fully ready when we stopped on Sunday night, but are you ever fully ready?

Day 1

During the day I talked to more speakers and participants to be part of the events in the test lab. I was very dodgy in details, since I only had rough ideas on what would happen. At 20:00 the door open to the test lab. Quickly afterwards Rikard Edgren, Simon Morley and Christin Wiedemann arrived and agreed to take a team and an application each as their focus area. The first event was collaboration test planning using Corkboard.me to plan in.

Participants started to arrive and we assigned them to the teams. Each of the team leaders did things differently based on what the team members needed. I noticed that some created missions/charters to be tested later while some wrote down their sessions directly. The content of each corkboard.me was different for each team depending on application/system and the team members. At the end of the collaboration session we held a debriefing where each team talked about what they had learned and experienced. Rikard, Simon and Christin held the groups together in an excellent way.

Start of the next session “Group Test Experiment”.

I held a short presentation about the various dimensions of group testing. I selected not to direct them in this, but left them to choose themselves what they wanted to try out. After a while I interrupted the teams and unfolded a scene/scenario that was about to happen. It went something like this “I, the test lead, am going to the release meeting for our products in 15 minutes. I want you all to give me the three most important bugs, where each team focuses on their application”. After roughly 15 minutes, I asked each of the teams to give me a 30 second report. Being able to give the essence of what is important is good to practice and often. Each team delivered some important facts all with a different pitch and undertone. The time was roughly 23:00 and it was time for a last beer before bedtime, at least for me.

Day 2

Discussion about having coaching session with Ilari Henrik Aegerter and Anne-Marie Charrett. Talked to more participants to get them into the lab. I let Rikard test some of the concepts for the evening event in the lab, and got some good feedback which made me change the agenda somewhat. Then I started setting up the lab so that it was possible for more teams instead of the initial idea of just two big teams, which must have been a good decision consider we were at the most close to 60 people if I read the tweets right. I selected Sikuli as the application/system for the evening event as the only system under test. A few people started to arrive at 20:00, but it was not that many to start with… so I considered if I had made the right move. All of a sudden the tables started to fill and the test lab was almost full.

I did a short presentation about the focus for the evening event, at least what I thought at that time. The idea was to have coaching of the teams in the first hour and later on to try out some group testing experiments. But I had to change that since the expectations of the participants were so diverse. Instead I said we would do a little bit of both during the whole evening, but with the major mission of creating a compelling status report.

Even more people arrived, who we filled up in the existing teams and by now we had 8 teams of 4-7 members (if I remember correctly). The context of the test lab had changed a bit from what my initial thoughts, I knew that the plan never survives when reality hits it. We had not put that much focus in describing in detail the full content of the test lab and how to get started in a good way. We thought that it would be easy to fill in that information as we went along, but that never became true.

Ilari and Anne-Marie came up with an idea for how they could coach the teams, which we tried almost directly. I told the teams that “HR (Human Resources) had decided that each team need to appoint someone to be tested.”. This act caused some confusion. Just prior to that I had told the teams that the focus was on creating a status report. So we had conflicting missions. Ilari and Anne-Marie did an excellent job even though it was a tough situation. Some teams were very focused on the testing mission and did not feel they had time for coaching, while others took it in and used the coaching to affect how the testing was done in the team.

Ben Kelly had joined in the test lab and I explained the current context and I explained roughly what I wanted him to do. I general I gave very thin outliers in how they could act in the test lab scene. Instead letting each player expand what was possible. Ben talked to the teams and coached them about what our mission was and about the creating of the status report.

During all this me and James were interacting with the teams. James gave some teams missions while I sometimes instead tried to confuse them. All-in-all we tried to simulate a situation where confusion and the unknown is our arena, but where me and James were roughly sure of what the other was doing in order to act on it.

After roughly 30 minutes of coaching by Ilari and Anne-Marie, I addressed the teams in the role of a project manager stating that “We don’t have time for coaching! I want those status reports!” The confusion was somewhat diminished and a renewed focus was visible. We decided that we would round up and hold debrief in about 40 minutes time, which was at 22:10. Some teams worked in X-Mind, another in the Corkboard.me UI, another in Post-It notes and yet another in Notepad. Each team focused on different areas, but some had found the same obvious issues. Some had found many issues and some found less.

It was my impression that Sikuli was quite buggy and that it really was not able to solve the automation in a good way without many workarounds. Many in the teams did not trust the application.

Even if we tried to keep the teams confused, that the team members where new to each other and new to the applications as well as to the test lab environment, they managed to produce quite good results. The traditional role of a test lead/manager could be questioned based on this or at least investigated further.

A few times when I confronted the teams and asked for specific results or information, where time was a factor, I noticed that some of the seemingly experienced context-driven testers returned questions, poking for more information. This is most often a good trait, but when time and timing is important you need to consider which questions you really want to ask that you believe bring value. Is this a common mistake that context-driven testers do? Still, a lot of the traps and confusion that was thrown on the teams was deflected in an excellent way.

Ben moderated the debriefing by asking for clarification then asked me in my role as a project manager if I was satisfied with the result of each time. It was a difficult question. From a test lab facilitator point of view, I wanted the teams to have a good time but also learn new things. I believe that objective was met. As a project manager I would probably had wanted to see a more consistent reporting style even though the information was on different areas. I was a bit split in what I actually thought here.

By setting up a scene it was easier to let the teams and their team members manoever in a changing context. It was very hard to balance between giving detailed instructions and giving a general outline in which they could act. Some teams embraced the scene quickly, while others found it harder. I believe it is important to be able to roleplay like this as a tester, to be able to act in a scene painted/described by someone else. It lets you explore so much more, with different mindsets.

During the Keynote of Julian Harty, he refered to “the testlab” but at this conference it was “a testlab” namely The Let’s TestLab, which was focused on learning, collaboration and experimentation. I am not sure we were fully doing open source testing nor any scientific method, is that still ok? I think we should embrace that many other testers run testlabs the way they want it to be. Otherwise we limit ourselves in what we can learn.

On previous test labs, I believe, the emphasis has partly been on the systems and applications. On this test lab it has been on the testers themselves and their collaboration. They have been able to explore some dimensions of group testing, note-taking techniques, collaborative test planning/preparation and finally seen different styles of status reporting. As a tester we know things could be different, but also that there is no one true way of performing a task. Hopefully this year Let’s TestLab showed that.

A big thanks to those who wanted to be part of the scene and acted in it. Another thanks to the speakers who helped facilitate with me and James. Finally, thanks everyone who participated that made it great!

On 7-9 May the Let’s Test Conference will take place. During the day there will be lots of interesting tutorials, keynotes and sessions. During the evening the events will continue. One of these activities is the Testlab, that we call Let’s TestLab. Initially I misunderstood Henrik Emilsson when we started to organise the lab. I thought the evening event was the testlab. At the time I did not consider anything wrong with having 150 people or more in the testlab. As I saw the evening event program I considered how could I compete with such a fine setup of activities. Well, this is a conference with many context-driven people. We will have different interests, expectations and focus area. So instead of admiting defeat, I considered what elements that we could add to the testlab to bring a great crowd, but perhaps not all of them.

Here is the line up of events in the testlab:

• Collaborative test planning
• Group Test Experiment
• Test Competition
• Making a compelling status report

Collaborative test planning

We will create possible charters, missions for the coming testing in the lab so that those who wish can practise different testing techniques. Everyone is invited to share his or her idea on how to plan testing in through collaboration.

Part of the line up in this event is:

• Rikard Edgren
• Christin Wiedemann

Group Test Experiment

The context of the testlab will be single testers or groups of testers going in and out of the testlab in different time intervals. Each tester will be unique in the sense that they bring different level of experience, skills and approaches to testing. Based on this, we will start experimenting with group testing. I think we have limited ourselves too long with the setup of pair testing. Going back to the early recommendations and experiences from Brian Marick, James Bach, Cem Kaner, Jonathan Kohl and many others, the setup is nearly the same. With new tools and techniques appearing over the years, some assumptions could be questioned.

Let us assume that there are different aspects and combinations of group testing that serves different purposes, we can say that there are different dimensions that could be explored.

Here are a few dimensions that we will experimented with:

• How many testers (2 or more)
• What role you play as a tester
• User types, User Scenarios or storytelling
• Mission of group test
• Note taking techniques
• Partner combination
• Lateral thinking aspects
• Personality types
• Debriefing techniques
• Accountability
• Focus areas or Characteristic focus
• Test Environment
• Basic Configuration Matrix
• … and new ones that we find along the way

We will explore and experiment with different tools that we use when group testing and share experience on what works in what context. We also experiment with a few pre-defined group test setups such as:

• Cinema testing
• 15-min test run
• Coaching a group of testers
• Wolf pack concept
• Testing Dojo

Part of the line up in the test lab is:

• Ann-Marie Charrett
• Markus Gärtner and Meke Mertsch
• Johan Åtting

Test Competition

Can you really compete in testing? Can you compete between two or more teams?  Can you really estimate the value of one piece of information against another? Well, it depends.

I have been an Ultimate Frisbee player for 34 years. I’ve not played for some time, but once a frisbee fan, always a fan. I think the same goes for testing. There are many things that I feel is similar. Craftmanship/sportmanship and passion is major part. Here is one description I like:

Ultimate has traditionally relied upon a spirit of sportsmanship which places the responsibility for fair play on the player. Highly competitive play is encouraged, but never at the expense of the bond of mutual respect between players, adherence to the agreed upon rules of the game, or the basic joy of play. Protection of these vital elements serves to eliminate adverse conduct from the Ultimate field. Such actions as taunting of opposing players, dangerous aggression, intentional fouling, or other “win-at-all-costs” behavior are contrary to the spirit of the game and must be avoided by all players.

One important element in Ultimate Frisbee is the spirit of the game, which you can see more in detail here [1]. Passion and humility as a tester are important traits, the spirit of the game concept might help us here.

So, my idea for a tester competition will be based on some of these ideas. Two teams compete against each other in form of best bugs and session notes. The two teams go through the opposite teams material and conclude who they think should be the winner with a good reason why.

Make a compelling status report

As the last event in the testlab we want to investigate how we can make a compelling status report for our stakeholders. Having many different testers, session notes spread all over, half-finished bug reports and test ideas half-finished… can we create something that is still valuable to someone? I guess this is a common situation at any lab at any company, still we will dig deep into how to go about this.

Part of the line up in the test lab is:

•  Ben Kelly

References

[1] Ultimate Frisbee Rules –  http://www.cs.rochester.edu/u/ferguson/ultimate/ultimate-simple.html

Many of us has a title that is connected to what we do at work. Every now and then I come across titles that makes me wonder what it really means. This time it is one that has been around for some time now: Thought Lead, what does this mean? I would not be suprised if there is a good explanation of how this title first came into place, but I do not think this is well known and I have not researched it.

You can learn a lot by testing something very different from your normal job. I’d love to professionally test a suggested law, or a chainsaw. For now, I give you an opportunity to test a bread recipe, in English or Swedish.

FAVORITE SOURDOUGH BREAD

FAVORITBRÖDET

It should be possible to bake from it if you know, or can learn, how to breed a sourdough, and bake any bread. I’m interested in correctness, ease of use, inspiration and taste (the last one only for the bread, not the documentation) and other things you think might be worth knowing about.

If you only have a little time and interest, the challenge is to come up with the different test strategies you would use.

Test results in comments or in mail (About page) are more than welcome!

Scripted testing vs exploratory testing approach

I agree with the idea of a polarization between the scripted test approach and exploratory test approach. These approaches include how you perceive testing and a tester. Almost in the same sentence, some say that you do a bit of both scripted and exploratory testing. The perception on testing and how you conduct testing are two different things. I believe that by discussing them together and especially with the polarization example, it more than often confuse the listener.

Instead of saying that you do both scripted and exploratory testing, I think it is more fruitful to talk about testing and checking. M. Bolton and James Bach states that “A check is a component of a confirmatory approach to testing.” in the blog article Elements of Testing and Checking [1]. We might even be so bold to say that it is the main component, but then we might say that another part is smaller and that might not really be true.

The Checklist

One aspect of checks could be something similar to what the co-pilot does when assisting the pilot. If we see the pilot as the explorer, reacting to input and performing based on the current context. In certain situations the co-pilot brings up checklists to help go through a situation that is too complex to remember all the details about. It is up to the pilot and co-pilot to know when the checks are relevant or not. The same situation can be found at the operating table where you have several types assisting personnel that help perform a complex surgery. The use of a checklist in those cases would help avoid the easiest mistakes or misunderstandings. See The Checklist Manifesto [2] for more ideas for testing.

How does this apply to testing then? When we do pair testing there is usually one driver while the other is a bit more passive and documents. What if the driver is the pilot/explorer and the other is the co-tester handling documentation, checklists and checks? When doing collaborative test planning, group exploratory testing or other collaborative test activities, do we then split up in roles such as tester/explorer, co-tester, checker etc to help us define what we do and focus on? Does it provide value to do so?

Types of Checks

Here is a definition of a check [1]:

a check itself has three elements:
1) It involves an observation.
2) The observation is linked to a decision rule.
3) Both the observation and the decision rule can be performed without sapience (that is, without a human brain).

But could we split checks in different parts? What types of checks are there? M. Bolton describes questions that has to do with confirmation, verification, etc. Could we also break down checks into other categories such as checklists, test/quality patterns, guided checks and one-liner checks.

A checklist is a list of items that help you through a complex situation. The items in the checklist could be checks that need to be confirmed or verified to a certain extent.
A test/quality pattern is something that repeatedly gives suspicion that it might be broken so that you need to check it.
A guided check is what we previously called a scripted test. A set of steps that end with something that you wish to verify or confirm, something with a binary answer.
A one-liner check or a check idea is a brief statement of something that should be checked in a certain context that would give a binary answer, such as true/false, ok/nok or yes/no.

Checking clarifies an aspect of what testing is or is not, do these sub-categories help us clarify what we do with the checks? Could we find more types of checks?

Test Management Tools

Based on the assumption that there are at least two types of questions and at least two different types of answers, how do we structure and manage these today? The traditional management tools for testing can often handle all the above categories for checks, but they are unable to handle the answers from testing.

But what do we mean when we say handle in this case? Well, you can structure the checks, plan when you do them and report the result of them. You can then make reports on the overall progress and the overall quality of the system. You can also state what build, system or setup you used in the test. But since the tools cannot really handle the testing part, the progress and the picture on quality really becomes obsolete.

If we look at tools and techniques that are more aligned with testing such as SBTM, they are good at handling the testing story. They are not as good at handling checks, or rather the tools I have seen so far.

Can we and do we want to handle the result from our testing and checking in the same system? When working in small teams where there are fewer stakeholders to work for, the need for information sharing in a big system can be less important. But if you are working in an organization where you have teams in several countries and where the overall development organization can be thousands of people, it can be more important to share information in a bigger system. Still, are you actually able to share information to that many people in an efficient way just because you use an advanced management system? When a lot of people are involved in creating and sharing information there is also a bigger chance that the actual meaning is lost. You then oversimplify the situation that information sharing is easy. If the organisation is big, there is a big chance that the context changes over the organisation and that the information changes meaning over the organisation.

I’ve seen so many test management tools that try to solve the whole test process. By focusing on all aspects they also become quite crappy at all aspects. Picking the tools that are excellent at solving one problem can then be a lot more efficient even if you get to work with several tools. When working with test coverage, communicating your test ideas, reporting status, reporting progress and showing details on what you tested there is a big chance that one tool really cannot solve this for you. We know that we find new techniques from other disciplines that can solve a problem for us when testing. So why do many limit themselves to test case management systems?

Summary

Michael Bolton and many other great minds have explored and delved into this concept, I think we can find more pieces that we can shed some light on. I’ve identified a few areas and there are more to come.

References

[1] Elements of Testing and Checking – http://www.developsense.com/blog/2009/09/elements-of-testing-and-checking/
[2] The Checklist Manifesto – How to Get Things Right - http://www.amazon.com/The-Checklist-Manifesto-Things-Right/dp/0805091742

If performance is crucial for product success, you probably need pretty advanced tools to measure various aspects of your product, to find all bottlenecks and time thiefs. For all other software, performance is just very important, and you might get by with lightweight test methods. You may, or may not have quantified performance requirements, but you should test performance to some degree anyway; for the whole, but also for each detail (when appropriate.)

In TheTestEye’s classification, performance consists of:

Performance: Is the product fast enough?

- Capacity: the many limits of the product, for different circumstances (e.g. slow network.)
- Resource Utilization: appropriate usage of memory, storage and other resources.
- Responsiveness: the speed of which an action is (perceived as) performed.
- Availability: the system is available for use when it should be.
- Throughput: the products ability to process many, many things.
- Endurance: can the product handle load for a long time?
- Feedback: is the feedback from the system on user actions appropriate?
- Scalability: how well does the product scale up, out or down?

Be aware of different definitions of performance testing, e.g. some include reliability, stress handling, robustness, and what stakeholders believe is most important might differ (even when using the same words…)

Ongoing Violation Awareness

The number one lightweight method starts by finding out which of these characteristics that are relevant for your product. Then keep them in the back of your head, and whenever you see something fishy, investigate further and communicate. Often the OK zone is easy to reach, but testers should notice when violations occur. When appropriate, apply the destructive principle: Increase the amount of everything that can be increased.

No Tools

Perceived performance is what matters for end users (but maybe not for a product comparison check list) so think about how it feels, and try using a stop watch. You might get pretty far by load testing with colleagues with several instances each.

Tools

There exists limiters for CPU, RAM, bandwidth etc. and many of them are free (and some of them become obsolete.) A task manager/resource utilization tool can give you hints on memory, CPU, disk, network et.al. Scripting your product to run over weekend is good for endurance and stability testing. JMeter is free and often quick to get running.

Summarizing

Summarizing performance test results is difficult. Aggregations of measurements don’t tell the full story, and the whole story takes a long time to tell. Communicate what is important, which is easier if you have asked stakeholders beforehand.

Warning: For some products, users aren’t as interested in Performance as the developers…

I´m sure you have heard it before. Everyone can test or Everyone does testing. Is that so? Is that really the case? Do you test just because you use a product? Do you test just because you stumble upon a bug? Do you test just because you can write some detailed step into a test management tool?

The Tester’s Pocketbook by Paul Gerrard is not a great book, but it is very good.
It covers fundamentals of software testing, and contains a ton of good ideas that will help you in your testing effort.
I also like it because it is one of few books on testing theory that focus on the human element of software, and its creation.

However, I disagree with some things, and will focus on the test design part.
These are Gerrard’s test design axioms with my verdict:

Test Model – Test Design is based on models
Yes, use many, and also look outside them. Build your own quality characteristics model.

Test Basis – Testers need sources of knowledge to select things to test
Yes, use multiple information sources to understand what is important, the list is bigger though (Purpose, Capabilities, Failure Modes, Quality Characteristics, Usage Scenarios, Creative Ideas, Models, Data, Surroundings, White-box, Public Collections, Internal Collections, Business Objectives, Information Objectives, Product Image, Product Fears, Project Risks, Rumors, Product History, Project Background, Test Artifacts, Debt, Business Knowledge, Field Information, Users, Conversations, Actual Software, Technologies, Standards, References, Competitors, Tools, Context Analysis, Legal Aspects, Many Deliverables, Searching, You.)

Oracle – Testers need sources of knowledge to evaluate actual outcomes or behaviors
Very useful, but not necessary; we can sometimes suspend judgment and communicate noteworthy information.

Coverage – Testing needs a test coverage model or models
Not needed, unless as a tool to get more test ideas, or a way to report what has been tested. Gerrard’s question “how many tests remain?” is not good. A better one is “how much test time do you guess remain?”

Prioritisation – Testing needs a mechanism for ordering tests by value
Don’t waste too much time on this; when necessary use a fast, frugal test triage.
Page 38 states “we must invite stakeholders to take an utilitarian view.” This is not true. We could equally well use a value-based system, e.g. “bought software should not crash”, or just go by feelings about what is right.

Fallibility – Our sources of knowledge are fallible and incomplete
True, but text gets a bit too negative towards the human mind. An engaged mind can make mistakes, but also discover what is important. We can separate right from wrong, we can handle the unknown, we can make up for mistakes done.

So which axiom is missing?
Sampling & Serendipity – testing is inevitably a sampling activity, where serendipity is to our help.

There is also an overall problem when Gerrard states that these axioms are needed to do testing. It should be: needed to do good testing.
Nonetheless, one of the better testing books out there!

If you want to seriously improve your bug reporting skills, read up, or take, the BBST Bug Advocacy course.
If you want to start by improving bug report title/subject/summary; read Lessons Learned in Software Testing, no, 83, or this blog post.

Many people will only read the title, so it is important to make it possible to
* understand how the problem appears
* understand limitations and dependencies
* understand the consequences of the bug

Some people will make their first decision on fix/don’t fix, based solely on the title.
(And for those that look carefully at the report, the title will guide their thinking.)
The goal is to make it possible to understand the bug, and how important it is, just by reading the title.

A few tips

* As short as possible, but no shorter than that.
* Start the sentence with the most important, to capture the reader’s interest.
* Don’t overdo “externalization” to capture interest, rather describe the dry facts, and let the readers draw conclusions
* If it’s difficult, try many times, or write the title after everything else is done (you might find the right words on the way.)
* Include a brief description of what happens.
* Include where the problem happens.
* Describe observations rather than (presumed) facts.
* Use a fair description, don’t exaggerate or understate the consequence of the problem

I also have a tiny, controversial one; start the title with lower case for the first word.
Most testers think they are writing a sentence in a story, and start with upper case (and end with a redundant full stop.) The problem I see with this is that you lose the ability to use upper case for names and terminology. “Exit” refers to an element in the software, “exit” refers to the action, which can be done in several ways. This is nitpicking, yeah, but it’s what I think.

You can’t include everything in the title, so use what’s most important.
The best way to learn this comes as no surprise: practice a lot.

While sifting and sorting the ISTQB Glossary 2.1 I finally found a couple of terms which definitions were both correct and useful:

1. deliverable – Any (work) product that must be delivered to someone other than the (work) product’s author.
Good, because it puts focus on the fact that you are creating the deliverable so it can be useful for someone else.

2. user-based quality – A view of quality, wherein quality is the capacity to satisfy needs, wants and desires of the user(s). A product or service that does not fulfill user needs is unlikely to find any users. This is a context dependent, contingent approach to quality since different business characteristics require different qualities of a product.
This is one of five ISTQB definitions of quality that are worth knowing about. This one is also correctly described.

3. walkthrough – A step-by-step presentation by the author of a document in order to gather information and to establish a common understanding of its content.
A sometimes useful practice, and a definition including the magic “common understanding”.

So if someone says that all ISTQB definitions are wrong, I can confidently say I disagree.