thoughts from the test eye

When you come up with a new test idea, you are using your knowledge and experience, but there is also some sort of stimuli that triggers the idea. Something you see, hear, understand or think about.

You seldom think in totally new ways, you rather combine things in a new way.

These are my favorite test idea triggers:

* think about each feature (and connect with testing or technical knowledge)

* create or understand a model of the software

* think about interaction between items in the system

* thinking about quality attributes (CRUSSPIC STMPL + Accessibility)

* read bug report titles for similar functionality

* read test idea lists

* play around with the software, or a prototype, or a previous version

* change perspective, see your software as a little box among many others

What triggers you?

Do you get help from Session Tester Primers?
(http://sessiontester.openqa.org/download.html)

A cheer to those of you who were able to attend this conferance: http://www.waterfall2006.com/

My favorites:

http://www.waterfall2006.com/crispin.html

http://www.waterfall2006.com/jeffries.html

Thank god everyone is not a believer of the hype around the Agile Movement. Process is king!

Well, “secretly” as in that he does not know that I am in love with him… Yet!

If you haven’t discovered the amazing Cem Kaner yet, I can give you the following advices and hoping that you too might fall in love some day:

• Visit http://www.kaner.com/articles.html and read ANY article from his large publication-section.
• Buy and read any of his fine books (I would especially recommend Lessons Learned in Software Testing)
• Follow his blog on http://www.satisfice.com/kaner/ (though he updates too rarely)!
• If you haven’t got the possibility to see him lecture in real life, take the BBST online course! http://www.testingeducation.org/BBST/index.html
  And yes, it is for free!
• Read his words about Context-Driven Testing and what it mean; and what it doesn’t mean: http://www.context-driven-testing.com/
• Join the context-driven testing forum and read his stunningly thoughtful comments: http://groups.yahoo.com/group/software-testing/

Are you still not convinced about his greatness, I urge you to talk to any test expert in our industry and see what they have to say about him.
Note: A lot of work above is collaborations with other great people in our industry e.g., James Bach, but they don’t really touch me as much as Cem do.

This post was triggered by blog post Resolved as Not Repro - http://thetesteye.com/blog/2009/06/resolved-as-not-repro/
I believe that bug systems too often are used with onlý a this-project-right-now approach, where you care most about just getting all items dealt with.
This is perfectly fine for one-off type of projects, but does not work fully for software where the bug system is (one of) the most important sources regarding quality information.

Since I use a holistic-subjective approach to testing, I need to consider both now-and-then, and also me-and-everyone-else.
Testers, developers et.al will treat bugs differently if they are resolved As Designed, or Fixed, or External, or Worksforme, or Not Repro, or Invalid, or Postponed, or Won’t Fix or Duplicate. 
At the end of this project, or next version, I might look again at all bugs set as Not Repro, or re-verify the most important Fixed bugs.
If there are many By Design issues, some usability folks might look into the details, and consider new approaches for the area.
Support people might search for information, and act differently depending on the solution of the related bug.
And also people that think that metrics matter, might be even more deceived, if the resolutions aren’t the correct ones.

My point is, you can’t really know how the information may be used, so do your best.

Lets say that you have a bug system; and for each bug you have the two fields “State” and “Resolution” where the following values are valid:
State: New, Assigned, Resolved, Closed.
Resolution: Fixed, Invalid, Won’t fix, Duplicate, Not Repro.

Further, you have a field where a product version number should be entered; i.e., the earliest version number of the product where the bug can be reproduced in.

Now, lets say that you report a serious crash bug that is reproduced in the latest build of the product e.g., version 1.1.2. (Oh, I forgot to mention that you work as a software tester at a software company and this happens during development of the upcoming release of your product.)
Two weeks later the bug is resolved by the assigned developer; the State and Resolution is set to “Resolved – Not Repro” and the following comment is added by the developer in the bug:
“I cannot reproduce this anymore (code is same as in build 1.1.4). This might have been fixed when other bugs in the same area were fixed. Resolving as Not Repro…”

So what do you think?

Do you think that this bug should have been resolved as Not Repro?
If not, what resolution would you have chosen?
Would it make any difference if the resolution “Works for me” had been a valid resolution instead of “Not Repro” (as in Bugzilla)?
Any other thoughts on this?

At EuroSTAR 2009 I will present “More and Better Test Ideas“; the main idea being that testers could generate many different types of test ideas, and communicate them in a condensed one-liner format.
If you have great tips on how to come up with really good test ideas, or want to review the paper I’m about to write, let me know.

/Rikard

Recently in Sweden there was a tragic death to a young child that could have been rescued if only the child had come to a hospital in time for a full exam. The one that was blamed for this death was the medical care hotline company that did not understand the severity of the illness and did not send this kid to the hospital. (Read more, in Swedish: http://www.dn.se/sthlm/pojke-dog-efter-rad-att-inte-aka-till-sjukhuset-1.859096)

After this tragic accident, it was discovered that this private medical care hotline company pays out a monthly bonus to those nurses that keep their phone calls short. (Read more, in Swedish: http://www.dn.se/sthlm/skoterskor-far-bonus-for-snabba-rad-1.864534 )
I.e. if they keep the call below 3.48 minutes and during that time complete the medical record, they receive a bonus of 1000 Swedish kronor (approx. € 100). In order to receive the bonus, there are some quality goals as well. E.g., you don’t get the bonus if you unnecessarily send someone to the emergency ward; or if you give a faulty medical advice.
Do I need to tell you that the county council paid the private company by the number of calls they handled.

This is what happens when you use simplified and dangerous metrics as a foundation for incentive pay… And these metrics are easy to abuse because they are based on simplified models of how the real world looks like.
When dealing with people, you are dealing with “complex systems” (read more in An Introduction to General Systems Thinking, by Gerald M. Weinberg ) and you cannot treat every person like they would be the same. I.e., the people calling in (and indeed children that cannot speak for themselves) are treated as a neutral “* 1” or “+ 0” in the metrics equation.
This happens if you include simplified metrics to measure your efficiency when dealing with people; metrics that leaves out the most important and complex parts of the equation: humans and human interaction.
Nurses know how to work with people, they know that people are unique; they know that their job is hard and requires skill and years of experience. They know that some patients require 20 minutes before they are calm or they need such time to explain everything important; they also know that some people just need 25 seconds before they are satisfied.
It is a shame that nurses are measured by how fast they finish a phone call.

It is the same thing that happens again and again in software industry; or rather the peopleware industry. People that work with developing software are measured by metrics that are dangerous and wrong; and in many cases it can have the same tragic outcome as with the young boy that did not reach the hospital in time…

Read more about (dangerous) metrics in the Software Industry:
Software Engineering Metrics: What Do They Measure and How Do We Know?, by Cem Kaner.
Metrics, Schmetrics, by Matthew Heusser.
Meaningful Metrics, by Michael Bolton
Measurements/Metrics/Analysis/Judgment, by Rikard Edgren

With subproject I mean when larger project have a need to split the project into minor parts. The essence of a chicken race is “Each player prefers not to yield to the other, the outcome where neither player yields is the worst possible one for both players.”

The goal for the larger project should be to release in time, with expected quality and so on. The goal for each subproject is usually to do their part and then integrate with the rest. The big problem is when something goes wrong or when one of the subprojects are delayed. In an unhealthy situation the subproject will not tell anyone. They will do all in their power to wait until someone else speaks out that they have been delayed or have problems of some sort. At the project meeting where all subproject managers are asked about delays, quality etc they will hold up a illusion of success. This can go on for a long time. Everyone is waiting for the first one to speak out, letting the rest get some extra breathing room and perhaps some extra time. The whole situation has turned into a chicken race.

The subproject managers will go about each other trying to see if someone is infact late and perhaps bring it to the surface. It is very apparent that everyone might be working towards their subgoals, but very much against the major goal. Naturally this is not always the case, but I have seen it happen many times. It is probably most common when there are many subcontractors and when the company culture promotes this behavior.

My only advice is stay to the truth, as you see it. If you are delayed do tell the others so that they can plan accordingly. In many cases you can change from making the short term decisions which might be costly, to more long term decisions that will be better for everyone. Keeping an openness amonst the subproject managers and über project managers will enable you to move more swiftly. Still, in some cases you might get kicked out as a subcontractor if you are delayed, in those cases you had better have a good plan B.

The front door

The bug that a co-worker found was that you were able to pull open the door a quarter of an inch, just enough to be able to put in a little stick and then wave it infront of the radar. It actually goes quicker to use the stick instead of the card.

I confronted the installer of the security radar and told him about the bug. He answered me “No, that is not possible.”. I told him again, it is so and that one of our consultants use it since he has no pass card. He answered me again “No, that is not possible.”. I did not have time to argue so I left.

I am now going to notify the owner of the house so that they understand the problem. Apparently there are more people using this trick when they have forgotten their card, so the stick is just below the pass card holder for easy access. A few years ago there was something called “Not my job award!”, I guess this would fit into this category.

What is essence of the discussion on release team/showstopper meetings?

I am assuming that there is a meeting. I’ve been to many different kinds of showstopper meetings and most companies handle them differently. One important item on the agenda for the meeting is usually the bugs that are found late in the project, thus at the end phase where they might be considered showstoppers.

When these bugs are discussed if they are to be fixed or not, I’ve seen cases when the argument is a bit vague. The actual topic is sometimes not brought to the surface, instead it is someone who “feels” that this should be fixed or it should not. What is it that they feel? Articulate those feelings and try to be clear on what it is you see. Why should it be fixed or why shouldn’t it?

In these cases I think it is best to be clear that we talk about cost, at least in most cases, in one sense or another, thus the cost of not fixing it or the cost of fixing it. Some costs will affect support, others will affect marketing and sales and some will affect development itself. I can recommend trying to get everyone attending the meeting to focus their thoughts on what the bugs will cost you. Is a few days slip perhaps worth it? In some cases the date for the release is extremely important, this is most often the case for bigger organisations. Instead it is easier to make some changes after the release decision and make a few patches afterwards. The actual release will not reach the customer the same day anyhow. Either way it is cost you discuss and you most often decide based upon it.

Cem Kaner has written an article on this that I’ve used many times when setting up these meetings. You can find the article here.