Using Function Points to Estimate Test Cases

What are function points ?
-----------------------------
Function points are a unit measure for software much like an hour is to measuring time, miles are to measuring distance or Celsius is to measuring temperature. It is a measure of software size. A product of five defined data components (inputs, outputs, inquiries, files, external interfaces) and 14 weighted environment characteristics (data comm, performance, reusability, etc.). Examples are
A 1,000 line Cobol program would typically have about 10 function points
A 1,000-line C program would have about eight.

Many attempts have been made to establish a relationship between function points and the effort associated with software development. Much of the difficulty in establishing this relationship is due to the fact that only the relationship between function points and the entire software lifecycle is examined. This blog examines the relationship between function points and testing -- in particular the relationship between test cases and function points.
The relationship between function points and the number of test cases is very strong. Capers Jones estimates that the number of total number of test cases will approximately equal the number of Function Points raised to the 1.2 power (FP 1.2). Data gathering and laboratory research indicates that Function Points times 1.2 estimates the number of acceptance test cases. Again the difference between the two is that FP raised to the 1.2 power estimates all test cases and FP times 1.2 estimates only acceptance test cases. That is, test cases grow at a faster rate than function points. This is intuitive because as an application grows the number of interrelationships within the applications becomes more complex. The test cases Capers estimates includes all test cases not just acceptance test cases.
For example, if a development application has 1,000 function points there should be approximately 4,000 total test cases and 1,200 acceptance test cases. Obviously, the development team should begin creating test cases as soon as possible. If the development team waits until coding has been completed, they will more than likely create far fewer than 4,000 test cases or 1,200 acceptance test cases.

-- Sanat Sharma

International Testing Conference, (8, 9 Dec 06), Hyderabad

I attended the "International Testing Conference" that was on 8th and 9th of December 2006 in Taj Krishna Hotel, Hyderabad. It was really good. Around 25 speakers from different companies (like IBM, Microsoft, Infosys, Wipro, AppLabs) presented different techniques of testing that they are using in their current projects. Some are quite interesting but some are too boring. Well I was quite shocked that how these boring guys are applying their boring thoughts in their respective testing projects.

On one side, each and every speaker talked about the different testing methodologies. Those were useful but only for freshers in Testing area. But on the other side, one person that really impressed me was Michael Bolton. He is really a great testing guy. He simply talks about the current scenarios that is happening with the testing team in every company and how the testing team should work under this environment. I am going to point out some points that I remembered from his presentations.

** -- ** About Automation : Automation should be ROBUST and DYNAMIC. Robust means your automation tool should be designed in such a way that it should test the application in a appropriate way. Dynamic means your automation tool can survive with the application after making some good amount of changes in it. It should be dynamic in terms of editing. Management should take care before going for automation that Automation needs some bandwidth in initial stage.

** -- ** Automation Testers vs. Business testers : There is a big difference between Automation testers and Business Testers. Automation testers are those testers who tests the application according to the test cases that they have created. Those test cases were created keeping only the technical aspects of the application. On the other hand, Business testers are those testes who have business view for that application. These kind of testers tests the application keeping only the business prospective in their mind. Normally Business testers are the higher management person of a company and Automation testers are the proper testing team.

** -- ** "I solve the testing problems that other people can't solve" : This should be true for each Testing Lead/Manager so that they can resolve the testing problems.

** -- ** "THE ONLY THING ONE POWERFUL THAN A GREAT IDEA, IS A GREAT IDEA POWERFULLY EXECUTED".

** -- ** Testing department view : Well, this is a most common view among almost all the testing departments. They viewed the quality of the product as the absence of bugs. But this is not true.

** -- ** Testers were not the QUALITY GATEKEEPERS.

** -- ** About Bug : A bug is anything that threatens the value of the project/product.

** -- ** "TESTING IS QUESTIONING A PRODUCT IN ORDER TO EVALUATE IT".

** -- ** Crime of Testing : It is like a Bank robbery. If you robbed a Bank and police catches you, you will be in jail. And if you are planning to rob a Bank and police catches you, you will be in Jail. So be prepared for performing the Crime of Testing.

** -- ** Asking the Right questions vs. getting the right answers : Now this comparison is very important for testing team. Just think about this.

** -- ** Remember "THE TEST IS WHAT YOU THINK AND DO".

** -- ** Documentation : Your manager says "That should be documented". This can be in terms of test cases, test plans, reports etc. But this really means "That should be documented if and when how it serves our purpose. Who will read it? Will they understand it?".

** -- ** Test cases are not the briefcases. Measuring the quality of a product/project in terms of number of test cases that have been executed is not a right procedure to measure the quality of it.

** -- ** Its up to testing team that whether they want to be steered or wants to work as a puppet.

** -- ** Now the most important question : When one should say that the testing is done?
1. When you have no more questions about the product.
2. When your management believes that the product can be shipped.
3. You addressed all important risks.
4. And most important : When your customer is satisfied.

That's all.


-- Sanat Sharma

Here comes my next blog. This time I am going to cover something about the Bug reporting process and guidelines which is very crucial for the testing team.

This paper is about what I’ve observed. I am not an empowered tester. But I am pretty good because I used to analyze the gifted people. I puzzle out what is that they do have and I don’t. I try to point out those things and then I do those things.

Now the first thing is "What is a Bug?” Bug is either a defect in the system under test (or device under test) or absence of support in the system under test for quality increasing features. It is a mismatch between the requirement specifications and the actual outcome. A behavior is viewed as a bug if the software doesn’t perform as required, specified or expected. Some synonyms are issue, fault, error, CR (Change Request), defect. I don't know why, but personally I like the word Bug.

First we have to understand the causes of a Bug in the software. In my total testing experience till date, I have observed some of the points that are mostly responsible for having bugs in the software. Those are:

1. Miscommunication or no communication between the teams
2. Software complexity
3. Lack of development experience
4. Poorly documented requirement specifications
5. Programming errors
6. Changing requirements on regular basis
7. Time pressures created by either management or client
8. Poorly documented code
9. Absence of any code writing guidelines
10. Poor guidance from development lead

Now the question is how a tester finds a bug? By executing the set of test cases, by monkey testing, by adhoc testing and many other ways. But personally I feel that it’s better to find bugs using the test cases rather than the monkey testing. Or vice versa if you are finding any bugs while monkey testing, add that test case on your test cases document. This will help you a lot. In one of my previous project, I was the person who had fired 75% of the total bugs in the system. But I was not updating the test cases. At last, when the product has been shipped to the customer, the test team has a very less number of test cases as compare to the bugs that have been logged. Although the management has this information that test team is not updating the test cases due to time crunch. But at last, we have to update all our test cases and shipped it to the customer.

We all know that when a tester finds a bug that should be communicated to the development team. A Bug Tracking system is the interface between the development team and the testing team to convey the bugs. It is constructed as a database. I have worked on three different types of Bug Tracking System till date. And all have more and less same guidelines to log a bug.

Now let's start. First thing is when an issue (referring issue here) is found by the tester, the first point is whether that is a bug or not? If that issue is mentioned in the release notes as a limitation or a known issue, it’s not a bug. Also if that issue is already defined in the requirements specifications, it is not a bug. In one of my previous project, I fired a bug with severity as showstopper (blocker) that was already specified as a limitation in the release notes. Instantly, that bug has been marked as Invalid. Although after that, development team lists out all the showstopper bugs as limitations in the next releases. Anyways, this should be taken care by the management.

Now comes the submission part of that bug. First you have to think about the synopsis/summary of the bug. It is very important, as this one line statement will describe the issue in short. A summary on the bug report communicates in a single sentence the essence and significance of the problem we have found. This is very important because when anyone (including your manager) goes through that bug, the summary should give a fare idea of what the problem is. A short and point-to-point summary will help everyone to understand the issue. And this summary is invaluable when it comes time to prioritize the bugs. I personally believe that writing this one line statement is much harder than it looks. Initially when I started testing and firing issues in Bug Tracking System, no one was able to understand what is being mentioned in the summary. To overcome this problem, what can be done is:

* Think about that issue twice that you have found.
* Make two or three summary statements for that
* Show all those summaries to your test lead.
* Ask him to select the appropriate summary for that bug.
* See what he is selecting and why?
* Discuss that summary line.

You have to do this exercise by not more than 6 to 7 times. After that you will be able to write a good summary. That's what I have done. Initially I spent much time to write good summaries because it is the most important sentence in the Bug report.

After summary, the next thing is the description part. Good bug reporting starts with organized and thorough testing. Testing may be automated or manual, but the tester should know what he is trying to achieve? The first thing that should be mentioned in the description is the release (or build) name or version number. The development and the testing team should do this. Although there is a drop down menu in all the Bug Tracking Systems to select the release version number, but this practice is useful for testing team when any comment is being added to that bug. Adding a comment with the release number helps to understand the flow of the bug life cycle with respect to releases. If you are logging in the Bug Tracking System using your company ID, the comment will be added with your company ID number. In this case, it is the responsibility of that person to specify his/her name at the end of the comment.

The tester should describe all the steps that have been performed to create that bug. If the bug is not reproducible, mention it. Try to recreate the scenario thrice. If that is repeatable all the three times, mention that the bug is reproducible. If the equation is something different, mention that also. Like "This failure occurred twice out of three tries". This is very important. Do remember:

If you repeat the scenario thrice or more than that, that will make your understanding clear about that issue and you have a better view for that. How the bug occurs deepens your understanding and helps convince developers and managers about the problem.

Try to dig the issues as deep as you can. I mean it is not a good practice that you have found a bug and you are mentioning the problem only. Try to dig the issue and see from where this problem is generating. If that is from the database, mention the table name and field name where the problem might exists. If that is in the code, try to debug it. If you are not able to dig, at least suggest some resolution to the development team. Its quite easy for the development team to understand and debug the issue if that is clearly written. Use just the words you need and describe only the steps and information that actually pertain to the bug being reported. Make sure that you are proving enough details so that development doesn't have to come looking for you to get them. In one of my previous project, I found a bug in which the application was not able to import a text file. When I dug a bit deeper, I found that the application is not able to import a specific kind of text file having special characters in it. Going more inside it, I found that the presence of only one special character in that text file is creating the problem. I have specified it in the description. And I hope that helped the developer to debug and fix the problem.

One more important thing is that before submitting the bug report in the Bug Tracking System, you should re-read that thoroughly so that if there is any thing left, you can add it. There is a short story of Mr. Perfect that I want to mention here.

Mr. Perfect is sending a love letter to his beauty. He wants to make sure it’s just right. So he doesn’t send it right away. He decided to wait a bit, so he pulls down new mail. One is from a major client. Mr. Perfect adds his address to his address book. He reads that mail and then after re-reading his love letter, he sends it. But unfortunately, the mail client, that he is using, sends the mail to the most recent person that has been added in its address book. Now you can imagine what happened?

So be very careful.

It’s a good practice to attach the screenshots and log files (if any) with the bug. Log files are useful for the development team to debug the problem. Screenshots are useful for development team as well as testing team. Sometimes when the problem is not consistent, the screenshot can show the exact picture of what have been done. In one of my previous project, I found a problem in the application only in the first attempt. The application was crashing giving a unique message that I have never seen before. After that I tried a lot to reproduce it but I couldn't do that. When I discussed that issue with the development team, they simply said it is not possible. And if yes, reproduce it, as they know that I was not able to reproduce it. But luckily I have the screenshot of that problem. I filed a bug with that screenshot as attachment and severity as showstopper, saying that the bug is not being reproducible. Till now, that bug is still in the Bug Tracking System as in open state, but the product is in the market.

Adding a comment in the existing bug is also very important. Sometimes a developer after fixing the problem adds a comment like "Fixed". This is not a good practice. The developer (or fixer) should mention where the problem was and what have been done to fix the issue. As a tester, adding a comment is also very important. I have worked with one of my colleague who used to write only "Verified and found fixed" while verifying the bug. But to keep track of what have been verified, the tester should mention like "Verified and found that .........................” This way the tester can mention the reason of closing the bug or reopening the bug.

Writing good bug reports and managing those reports is a challenging task for the test team especially for the testing manager. These challenges are both technical and political. Political here means related to the need to manage the interpersonal and team dynamics of criticizing the work of others. I once worked on a project where my lead was only concerned about the number of bugs that I have fired in the Bug Tracking System. In my opinion, quality of bugs is more important as compared to quantity of bugs. You should not enjoy testing just because you got a chance to "catch" developers. This kind of attitude will definitely create issues between the developers and the testers. We as a testing team should keep our heads cool during discussion about bug reports with developers. We should discuss bug reports with open mind, accepting the possibility that we are mistaken, but also defend the reasonable expectation of yours. As a tester you should be cooperative with the developers. I once worked in a technology, which was quite new to me at that time. I have to test the release and deliver it to the customer on weekly basis. At that time, I worked with one of the best development team and I realized that working with developers is quite useful for the testing team.

In software industry, Bug report ping-pong is a game played between developer and tester. Here in this game, bug reports are kicked unproductively back and forth. One of the most common reasons for this ping-pong game is that the developer says that this bug is not reproducible. This problem can be resolved through careful bug reporting particularly documenting the steps to reproduce and mentioning that the problem is reproducible or not. Second most common reason for this ping-pong game is inappropriate assignment of the bug to the developer. And the managers can resolve this by assigning the bug to the concerned developer. This article has hopefully challenged some of your existing processes of firing and managing the bug reports.

I have tried to provide an overview of a generic system, based on my experience. As you learn your company system, you may find that items and procedures are little different. But one thing I want to say that as a tester you should have strong opinions about the bug reporting and managing. Having this attitude in mind, you can definitely improve your bug reporting process and can handle the situation easily.

-- Sanat Sharma

My experiments with Testing

Well to say that this is a technical article is totally unfair. Here, I share my learning, experiences and expertise that I gained during my journey as a Trainee to Technical Lead in testing domain.

In general everyone is under impression that Software testing is nothing but only finding bugs (or you may call them issues also) in the application or product. And if someone has a fare idea about that, then the definition could be “Software Testing is the task of trying the software application to see if it works and trying to break it.”

If this is all about the Software Testing, then what is the basic idea to keep this phase in the Software Development Life Cycle (SDLC)? We all are aware of the SDLC. It has some phases that come while developing the Software. The main phases of SDLC are:
1. Feasibility study
2. Requirement Analysis
3. System Design
4. Coding
5. Testing
6. Implementation and Maintenance

Now the first thing is how is Software made? Well for that, we can say that someone has a good idea and define some requirement. Sometimes these guys are from Marketing, sometimes an Engineer (not necessarily a computer engineer), sometimes a very cool hunk, and sometimes an ordinary person (not from Computer’s line). They do some research or they have some good experience in different areas. They define their requirements. That basically includes what they want from the software to be developed. Now the requirement also includes how the software will work and most importantly, when it will be shipped to the customer. After that, the engineers worked on those requirements. They work on the detailed design document, how the software will work behind the scenes, how the user will interact with the software. The decision will be made by the upper management that which software language and tools will be used to develop the software and write the code. After developing the software, testing will be done. The software testers test the product and verify that it works properly. If everything goes fine, the product is shipped to the customer.

Now let’s see the testing part of this cycle. First thing is when this is done? This is done before the product is shipped to the customer. Now what will be the base for the software test team to test the product? Sometimes a tester can rely on the written specifications to explain what software is supposed to do. Using those written specifications, they are supposed to create a good amount of test cases (Quality + Quantity), test the product and shipped to the customer within the defined timeline. Here the bold and the underline point is that the product should be shipped to the customer within the defined timeline. To achieve this, most of the times testing team has to suffer a lot. These are the guys that have been mostly affected to achieve the deadlines. In one of my previous project, my manager instructed me to test the release within two days. I was the only tester in the team and the development team of six had spent more than 30 days to develop the application!!!

Sometime, it happens that the testing team has no requirement specifications documents but they are supposed to test the release thoroughly and fire some good bugs (Quality + Quantity). Now why I am referring to Quality and Quantity both? Quantity should be more means the number of test cases and bugs should be more. Quality means good and deep level test cases with some really showstopper kind of issues. Now come to the requirement specification part. As I am saying sometimes it happens that there are no requirement specifications documents and testing team has to test the application. This is really a painful situation for the testing team. They have to test the release and deliver it to the customer. After that, they are the people who receive some R*E*A*L*L*Y good comments from the customer. In one of my previous projects, the development team developed an application for more than 6 months and we tested it thoroughly. When we showed the application to the customer, he simply said “What is this? This is not the application that I want. This is totally different.”!!! &$#@!&&*!!!! So just remember one thing:

WHEN THERE IS NO REQUIREMENT SPECIFICATIONS, EVERYTHING IS A BUG.

Sometimes, testing team is the only team who has to shrink their testing life cycle to achieve the defined target dates. Or they have to stretch themselves beyond the expectations. In another project of mine (one of the very reputed project of my career), I used to work from 6:00 AM till 12:00 midnight only to deliver the quality release to the customer. Although the satisfactory fact is application is today being used by thousands of users and I proudly claim myself being part of it.

I started my career as a System Administrator guy. Then, I worked with one of the greatest and coolest boss of my career. He is the person from whom I learnt a lot. My boss supported me a lot. A really great guy and a great human being. Anyways, let’s come to business. When I was in system administration team, I always thought why testers are always troubling developers. One of my friends was in testing team and I always used to ask him what the hell they are doing? Development team writes the code and you are cracking it. But that was not true. Testing team &development team both have combined responsibility of delivering a quality release to the customer. Now the point is “What is Quality?”. The best one liner about the quality is:

CUSTOMER SATISFACTION IS QUALITY.

If the customer is happy with your work, you have done a quality work. That means, if all the features, modules and functionalities, that customer wants, are present in the software and all the features, modules and functionalities, that customer don’t want, are absent in the software, you have done a great job. Irrespective of the fact that the development team developed the application in 20 days and testing team tested it for 4 hrs only, if customer is happy, all are happy. And the life goes on and on and on. I mean the company goes on and on and on ……………………..

More often in consumer products, a requirements specification doesn’t exist. In this scenario, a tester must figure out what to test and what software should do on their own. To do that, they used a combination of asking questions of the product manager and the engineers, their knowledge of how the software runs and their intuition based upon their experience. This helps the testing team to create a good amount of test cases and execute them according to the requirement (that is not specified). Using this technique, testing team can fire some good number of bugs in the application. A bug is not just an error or defect in the software. It is deemed a bug if the software doesn’t perform as required, specified or expected.

In brief, I want to summarize some of the good qualities that a tester should have to fight these kinds of scenarios and deliver a QUALITY release to the customer.
· Quick Learner
· Curious
· Well Organized
· Good observer (Mr. Perfectionist!!!)
· Ability to work with a ambiguity and with a minimum of direction
· Self motivated with a excellent written and verbal communications skills

Although all the qualities referred above are important, but one thing I want to say that a tester should have a good communication skills. Why? As a tester, your job is finding bugs and informs this to the development team. So you are always a bad news reader. If you present this bad news in an ambiguous way or with a bad attitude, the recipient’s natural response may be to “kill the messenger”. I have faced this behavior once in my career. By having professional and accurate communication skills, a tester can easily be viewed as a valuable member of the team and not as a “necessary evil”.

As far as bugs are concern, if the testing team is working in a proper way, they can find a huge number of bugs. I am listing some of the points that are majorly responsible for most of the bugs in the software:
· Communication Gap within the teams
· Complexity of the Software
· Changing requirements and programming errors
· Time pressure and poorly documented code
· Errors in Software development tools

So now the question is “What is the goal of software testing?” In my opinion, software testing team goal is to demonstrate that the faults are NOT present and not merely finding the number of faults present. To achieve this, they should get involved from the first phase of the SDLC. Testing team should ensure that all the functionalities are working fine according to the requirements. And most importantly, the customer should be HAPPY. He should get what he wants from the software

To achieve all these goals, we should follow structured approach to testing. Test Planning, Test Design and Test Execution should be conducted in a proper way. I’ll cover more on this in my forthcoming articles.

Finally, in my opinion, we as a testing community should just remember: We are guardians of quality on behalf of business community, particularly where time pressures force developers to take short cuts in normal development process.

-- Sanat Sharma