thoughts from the test eye

When looking for personel in general it is common that we want passionate people who love their work. Most passionate testers that I read about are usually part of the context-driven movement. Can there be testers out there that are really passionate about how they work in the heavy scripted test environment, where someone else does the thinking, writes the script and then hands it over for execution? What personal goals do you have when doing these tasks? Is your main focus in moving out of that zone so you also can do the thinking, write the test script and pass it on?

In the ET heavy environment I tend to see fewer roles and career paths, is this true? There are test leads, test managers and testers. Everyone want to test since that is their passion, no? You have fewer administrators and more doers.

In the scripted test environment I see more different roles and career paths. There are experts in certain domains, writing test cases. There are many test leads who spend their time trying to lead what is to be tested and exactly how. There are test managers allocating the very few testers in executing these tests. There are more administrators and less doers. A few times I’ve seen almost 90% administrators and 10% testers. With administrator I mean someone who has little focus on actually doing any tests themselves, and in some cases do not want to. One common management idea is that anyone can assist with executing a test script. This means that anyone can be a testers, thus you are not special in being a tester and you are easily replaced. There is no skill in being a tester since you just follow that script, no?

Are you then passionate about being a script tester?

Analogies are helpful, not because they come with truths, but because they can help you highlight and think in different ways about the phenomena you are comparing with.
I think you can pick any subject you know a lot about, and after some thinking, interesting things will emerge.

The important moments
If two chess players are at about the same strength, the winner often is the player that realizes at which stage it is necessary to re-think the strategy very carefully. Some moves are more important than others.
For software projects that don’t go perfectly well, and when unknown things happen, the better activities might be straightforward, or require a lot of condideration and creativity.

Technique
There are many typical metods that every good chess player must know about (forks, rook endings etc.) in order to see oppurtunities and apply them in the right situations.
Software projects differ much, much more than chess games, but there are many available quicktests, tricks and test design techniques that you can make good use of, if you know them well.

Theory
In chess there has been an enomous amount of analysis of different opening moves, and a player has a great advantage if she knows a lot about how to start games, and the typical positions and strategies that are likely.
Since each project have a new starting position, we can’t have this in testing.

Understand what is important
In chess there are some key elements the player needs to think about (material, development, centre, king’s safety, pawn structure etc.), but in each game these different aspects have different importance; it doesn’t matter if you have a great advantage on the Queen’s side if your opponenting is mating in three.
It is the same in testing, we might know beforehand which areas and attributes that matter, but since testing can’t be complete, and unknown things always happen, we need to adjust and focus on all those things that are most important.

Time trouble
If you spend too much time on your moves, you will end up in time trouble, and once there, it is a much bigger risk of making big mistakes.
We can get in the same situation in testing (e.g. a fixed release date, even though everything else has been pushed), with the main difference that the test team have little chance of avoiding this by their own means.

“it is better with a bad plan, than no plan at all”
When you learn chess, you are often told that you must have a plan, that having no plan is a bigger mistake. Later, I have realized that a bad plan probably is worse, but by always creating a plan, you get better at it.
So we have the same in testing: it is essential to have a plan, and a bad plan is better than no plan from a learning perspective.

Practicing
There are many ways to learn chess, but a key element is to play a lot; and I think it is the same for testing.

Analysis of games
After a competition game where you have played for a couple of hours, it is common that the two opponents sit together and go through the game, move by move. They talk about their thinking, and examine what would have happened if better moves had been chosen.
The exact same thing is difficult to do for testing, but detailed retrospectives of important decisions or bugs can be a great learning exercise, and we should do more of this. Maybe directly after a pair session?

History
When learning chess you study the history, look at classic games that you learn from and get inspired by.
We don’t have this in software testing, and it is a pity.

Diversity
The people playing chess are of all different types, and like in testing with all types of backgrounds. Maybe the concentration of peculiar people is higher, both in chess and testing; this is a good thing.

Fun
You learn more when enjoying yourself, both in chess and in software testing.

When making comparisons it can be fair to list the most important differences:
* Chess is a game, testing isn’t.
* Team work and collaboration is very different.
* Chess has a defined play area, and a specific set of pieces; and this is certainly not the case for any two software projects.

This is an idea you see and hear now and then. It comes in different shapes, ranging from testers needing to have an independent manager, to testers being best if physically separated from developers, or even outsourced, or crowdsourced.

Cem Kaner writes in The Ongoing Revolution in Software Testing that this notion primarily is a “fear of bias”, and he is right, as always.
The rationale for independent testing boils downs to testing not being influenced by the development. Of course the influence can sometimes be negative, but the positive influence is more common, and more important.

There are three important reasons why I wouldn’t want to work separated from development:
* More and better information (both ways)
* More ownership and motivation
* Working together with different knowledge and focus gives a holistic view, and a better end result.

But since this separation can take so many forms, there is a need for some clarifications.
* If the testing team belong to a separate group hierarchically, but in all positive ways can influence and be influenced by other groups, I see no problem with it.
* If outsourcing includes developers, testers, documenters, and maybe even product management, I see no problem.
* it is OK if parts of the test effort are performed elsewhere, in order to get different views and approaches; Beta testing is an excellent example.
* if the ambition isn’t higher than doing exactly what is specified in detailed requirements, separation might even make that easier.
* Regardless of setup, if there is a mentality of minding your own business only, I don’t think it is a good setup.

I’m not sure if it is the physical or the mental separation that undermines the ability to get involved in each other’s work, which I see as a good thing, not because of lack of trust, but because we wanna make sure that the end result is a great product.

I’ve recently finished James Bach’s book Secrets of a Buccaneer Scholar. I liked it, but I don’t agree with all of it. As a tester, I feel that it inspires me and gives me new ideas in my way of thinking and how I perceive learning, especially self-education. I fully agree with James on that you should follow your passion. If you are able to assist those around you to that end, it will make you grow even more. In march James will be in Sweden and hold a series of courses, one of them is Self-Education for testers. It think it would be very fun and educational to attend, I will see if I can make time.

One thing that I consider after reading the book is how I can inspire my daughter to learn, test new things and to follow her passion. When she receives a new toy, I want her to explore how it works. For instance, she got a little toy puppy that execute somersaults. It had a lot of mechanic inside it and sounded like it was very fragile, if you actually played with it. The purpose of the toy was probably just to watch it. I dislike such toys. I asked my daughter how she thought it worked. She did a somewhat exploratory, destructive test by enabling the puppy to do the somersault, then directly afterwards hugged it tightly. There was a popping sound in the mechanic as the puppy tried to execute the somersault, while being held secure. After that test it was not possible for the puppy to somersault, rather it performed a half one and landed on its nose. I applauded my daughters destructive sense of testing of the toy. Translating that into testing terms, she prioritized which test to start with and considered what was the biggest chance a user would do then executed that. One test to render the system useless. Wonderful!

I think we have a lot to learn (or perhaps relearn) from our children.

Now and then I read that testers don’t know enough about Usability, that there is a need for a Performance Testing expert, that a Security consultant should be called in, or that a master of the used technology would make Installation and Compatibility testing possible.
This might be true in the general case, but there are many testers that are working with the same product suite for years, and over time you become an expert of most aspects of quality for your specific product.

Let’s look at some examples:

One sub-category of Usability is Operability; that an experienced user can perform functions in a fast and efficient way. As a system tester, you have done most operations many times, and you know what you can expect from features in the sense of operability. You can point at places where for instance the ability to delete multiple items with a few clicks would make a difference.
Learnability is another sub-category of Usability. You only learn the basics once, but maybe you have heard customer stories of confusing things, or you could let a new member of the test team think in this direction.
Regarding Accessibility it doesn’t cost too much to use High DPI, speakers turned on, Code Blind or Magnifier sometimes.

Large-scale Performance might require an expensive tool, but I bet you’re doing some simulations without them; maybe just by telling the whole development team to hit the same server at the same time.
There is also a small-scale, low-level Performance aspect that shouldn’t be understimated. If a dialog takes more than a second to display, it might be something that put the user’s confidence in doubt.
If you have tested your product for quite some time, you will immediately notice when something takes just a bit longer than it could take. There of course might be valid reasons for this, but talking to the developer about it might be beneficial to both the producers and consumers of your software.

Security testing seems more difficult, but as a product tester you know at which moments authentication takes place, you know if there are passwords stored, and that they should be encryted; you might not know how to exploit a crash, but you are an expert at provoking the crashes.

For Hardware/OS/Application Compatibility testing you will become more of an expert the longer you work with them, at least if you have the curiousity to learn more things when you get the chance.
You might not know a lot about how the iPhone works, but you know all the details that can be used for interacting with your web site.

Sometimes I also read the extreme that testers shouldn’t bother with Usability/Security/Performance testing, which to me seems like an incredible waste of knowledge and resources.
When testing functionality manually, you can look at quality attributes at the same time, and get a lot of coverage for free.

I’m not saying true specialists aren’t needed, but I’m saying that there are a lot of expertise in your building that at least can be used complementary.
If you are in a situation where you know a lot of these things, but aren’t allowed/encouraged to test these things, I think you should try to convince your managers of a better and more fun way to test your product.

I have been thinking about qualitative research lately, and wondered what the question(s) would look like if testing was seen as a research project.

The testing effort has many positive effects, but one common and important is to provide information about the product, so a good release decision can be made. We cannot prove that the software is good, but we can try our very best to show important ways it can fail.
But at the same time, it is too late to come with important information just before the release is to be made; the information should be provided in a timely manner.
So in a paradoxal way, we try to destroy the product we love, but at the same time give the information to the project, so they can make the testing mission fail.

Testing try to answer “No” to the “Can we ship?” question – many times in many ways – but gives the project the best chances to resolve all issues as early as possible.

Another important question is “How can the product be better?”, and information about this is also being provided all the time. Not too late, but also not too early, in times when we know too little, and the cost is too high to reach the vital information.

Of course, there are many other sources and considerations, but testing often has the best view of the system as a whole, and the best view of some details from a customer perspective.
Testing can’t be complete, but the answers are valid when the testing can be considered saturated, when too much effort is needed to get new information that aren’t too important.

Are there other questions that are more important?
Or are we providing answers without questions?

Cem Kaner has updated his set of publications. I’ve been reading his well written articles over the last ten years.

Have a nice time digging in!

Jerry Weinberg has, in his book ”Perfect Software and other illusions about testing”, expressed a very important observation, namely who is responsible for pinpointing the bug. The tester finds the bug, tries to reproduce it, then adds as much information that he/she has such as log files, configurations, test data and so on. When you estimate time for testing I think you most often consider the time that you do to find the bug and then make a report. As Jerry points out, the time it takes to pinpoint the exact location is not taken into consideration. It might also be unclear who has this responsibility.

This might not be an issue where there is high testability, when it takes little time to report the bug and when the actual pinpointing takes little time. When testing complex systems (a system consisting of several sub system, different hardware, many interfaces, a vast amount of ways to get information on what actually went wrong), the time for pinpointing can be a factor that is underestimated and that can really be a major issue in a project. Reproducing a bug can take several hours or possibly days. If you, after having reported the bug, are asked by the developer to add extra or missing information to the bug it might cost you days to get into the same erroneous state. In some cases it might be close to impossible because of missing equipment or missing/broken units. When it takes this long time it is inevitable that a conflict grows in the project. It is also inevitable that the testers start to doubt how much time they should spend on bug reporting (in the worst case scenario consider not doing it) and if they can help developers.

How do you handle pinpointing? Is there any need to increase testability to make it easier to report bugs and get information from the sysem? How have you discussed and communicated this within our projects? Is your testlead and project manager aware of this time sink and uncertainty?

If you want to taste some of Jerry Weinbergs knowledge I recommend reading this book.

A disclaimer… I am no developer, but I have developed a tool. As I develop I have the mindset of a developer, not the tester. I have done lots of mistakes, intentionally not implemented good/needed things and considered what parts I can get away with in the first release. This tool might not seem big and useful, but I have used it and it has created many interesting results in the past. As I developed this I tried a new method of implementation… all ideas I had on what functions the tool should have, what was supposed to work, what was not supposed to work etc I wrote down in a testideas-document. I then had one column that identified if it worked or not in a specific release. All good feedback I added to that list.

This is the first tool we create at the test eye that is open for the public. At thetesteye we have choosen to publish our material under the license Attribution No Derivatives. My personal aim with this tool was to increase my knowledge of coding. I have used Python and Tkinter as a graphical interface. In the Publications section you will find the link to the tool and the currently released version.

General discussion

The general idea is to use the frequency of words as a way to find errors. The more text you analyze, the higher statistical significance; thus resulting in an easier chance of spotting the erroneous words. This kind of script is very often found as a code example. When I first created a script for this I did not know that. I ran it on a quite large text corpus and found that the company name had been spelled incorrectly 7 times in the copyright text. I also found lots and lots of spelling mistakes as well as some strange API functions that were incorrect.

Use cases

• Run on documentation to find unfrequent words (that usually contains spelling errors)
• Run on code to find variables that are similar but not the same and used incorrectly
• Run on code to find unused variables, thus variables only used once
• Run on code + API documentation to find things that should not be there or code that are not covered anywhere
• Localization specific: When doing translations you might be allowed to have a certain amount of errors, this is one way of finding a few extra faults that you can remove

How I use it

I run the tool on a tree structure. I open the result file in Excel or OpenOffice Calc. I then sort on frequency… start deleting uninteresting records. You can open it in MS Word or something similar to filter out things that are in fact spelled correctly. After a few cleaning ups you might have a list that is worth investigating.

Bugs and Enhancements

The testideas.xls contain the current tests and some of the enhancements that I’ve gotten so far. If you got any suggestions, feel free to mail me at martin.jansson@thetesteye.com.

Let me begin by saying that these are my beliefs ever since I took the ISEB/ISTQB certification. But when I thought of this recently, I think I need to make a statement and try to help all those that are rejected because they are not certified.

————————————–

In the search for a qualifying certification many European companies have chosen to use the ISTQB certification and specifically the Foundation Level as a qualifier. But there are several voices raised against the validity of this certification, many prominent experts do not agree with the content of the syllabus. The problem gets worse because the more people that gets certified, the more this certification becomes “valid”.

The certification is based on an examination that you can take after reading a syllabus by yourself or paying for attending a two-day course and then take the examination. Almost anyone can do this! And it does not say anything about your testing skills. So this certification should not (and cannot) be used as a qualifier of how good one person is as a tester.

So what it really means is that a person that hold this certification has managed to get at least 30 multiple-choice questions right out of 40.
Think about that before you reject job applicants because they aren’t certified (which seems to be the case too often in Europe).

Let us treat software testing as a serious profession and throw out this certification as soon as possible.