Number of bugs vs. Performance of Tester

I will start this blog with an anecdote.

Start of anecdote.

Around four years back, I was working (or better to say testing) in telecom lab when suddenly one of the Project Manager (PM) came to me and said that there is an emergency meeting with the client and you need to be there immediately. I immediately rushed to the conference room and found that the room was full of hot vibrations, which was due to the hot discussions that were going on with the client. It was a telephonic call with the client sitting across the oceans in London. I joined it and the first question that has been fired on me from the client was that how many bugs you have found in the last release that has been delivered to them. Although I know the exact number of bugs, still I said “Around 25 – 30”. The response from the PM, sitting besides me, was “No Sanat, tell us the exact number”. I said “What do you want to retrieve from the exact number of bugs”. And that too in front of the client, I whispered slowly. The response from the PM was shocking. He said “We want to measure the performance of the testing team based on the number of exact bugs that you have found”. I was not able to answer (or to react) on his question at that time but I provided him the exact number of bugs. I immediately asked to leave that room and I done so.

But the concern that was hitting me again and again was why a tester is being rated based on the number of bugs that have been logged by him/her. Although I must be a happy man at that time since I was the person who almost fired 80% of the bugs in that particular product that have been tested by me (with other 4 testers) for more than 3 years. I was thinking on that on a very serious node and in the meantime, one fine day, my manager (Testing Manager) called a meeting with the all the testers (we were 5 at that time, including me). He said that we should fire more and more number of bugs against the release to improve the testing and to show the client that effective testing has been performed.

I totally followed his instructions and that started creating an unhealthy environment between the testers and programmers. All the testers (including me) started logging bugs in the databases with a very high frequency without even thinking of whether it is a bug or not. And without even throwing a second thought on the bugs, the programmer started rejecting the bugs. This panic and uncontrolled situation continued for one week and then a team meeting has been called to analyze the same. The meeting concluded with a node that the decision taken before was incorrect and there must be something that should measure the performance of the testing team.

End of anecdote.

To understand this topic in a better way, first we should understand the common problems in the Software Development Process that redirects any Software to a buggy Software. Some of the problems that I have seen are:

• Poor requirements - if requirements are unclear, incomplete, too general, or not testable, there will be problems.
• Unrealistic schedule - if too much work is crammed in too little time, problems are expected.
• Inadequate testing - no one will know whether or not the program is any good until the customer complains or systems crash.
• Poor planning - requests to pile on new features after development is underway which is extremely common.
• Miscommunication - if developers don't know what's needed or customers have erroneous expectations, problems are guaranteed.

Now I am coming back to the points that need to be considered to analyze any tester’s performance. See the underlined text below. You will get the viewpoint of mine that I always considered for this concern.

A good test engineer has a 'test to break' attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail. Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful. Previous software development experience can be helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers' point of view, and reduce the learning curve in automated test tool programming. Judgment skills are needed to assess high-risk areas of an application on which to focus testing efforts when time is limited.

A good Test engineer should:

• be familiar with the software development process.
• be able to maintain enthusiasm of their team and promote a positive atmosphere, despite what is a somewhat 'negative' process (e.g., looking for or preventing problems).
• be able to promote teamwork to increase productivity.
• be able to promote cooperation between software, test, and QA engineers.
• have the diplomatic skills needed to promote improvements in QA processes.
• have the ability to withstand pressures and say 'no' to other managers when quality is insufficient or QA processes are not being adhered to.
• have people judgment skills for hiring and keeping skilled personnel.
• be able to communicate with technical and non-technical people, engineers, managers, and customers.
• be able to run meetings and keep them focused.

I believe testing cannot be measured accurately. Although there are n number of formulas that I am also using to provide some number as a performance of the individual tester, but still I disagree. Those numbers (or formulas) are good for management because management always believes in number. Testing cannot be measured, it can only be monitored. As I always said “Testing never ends”.

I will conclude this blog with Dr Cem Kaner amazing statement on this concern. “If you really need a simple number to use to rank your testers, use a random number generator. It is fairer than bug counting, it probably creates less political infighting, and it might be more accurate”.

-- Sanat Sharma

Materialized Talks

One week back, I read a great article on http://www.stickyminds.com titled “What's In A Name?” which says that developers are not the only “developers” who are working or developing the product. All the persons (System Engineers, Coders, Programmers, Testers, deploy persons, support functions) are developers. Developers should be called as “Programmers” or “Coders”.

I have added my comment on that article and I got a supporting response from the author, Fiona Charles, who is a Toronto-based test consultant and manager with thirty years of experience in software development and integration projects. See my comments that I published and the response by the author below:

---------------------------------------------------------------------

Comment:
by Sanat Sharma 10/29/2008

Great thought, Fiona. "Developers" are not the only one who are developing the product. It involves other roles also. And that is the reason; some organizations started saying developers as "SDE" means "Software Development Engineers" and testers as "SDET" means "Software Development Engineers in Testing". It all went wrong when we started calling programmers "developers." It is very difficult for me, having an experience of 7+ years in testing and quality, to deal with a programmer turned manager.

Sanat Sharma
http://www.xtremeedge.blogspot.com

Author's Response: 10/29/2008
Thanks, Sanat. That's an interesting example you cite, though I would prefer to leave the "engineer" out of both titles. What's wrong with "programmer" and "tester", after all?

I sympathize with your issues about programmers turned managers. Unfortunately for us, they are in the majority among IT managers. We need to find a way of reaching them collectively, not simply one by one, so we are not constantly having the same conversations or arguments again and again.

-------------------------------------------------------------------

Few days back, I discussed this with one of my programmer colleague and the response I got from him was amazing. I can’t pen down what he said but that was disgusting and shocking. And on top of it, one of the programmers sitting beside us, having around 3+ years of programming work experience laughed at his comments without even thinking of what has been told by both parties.

I realized on that day that sometimes you cannot change the attitude of someone who is not able to understand what you are trying to say. I decided better to keep my mouth shut with these kinds of guy rather than doing long discussions. Because it’s true that “Never argue with a fool, people might not know the difference.”

-- Sanat Sharma