Assurance Plans Guideline

You just finished building your company’s website. You have tested it yourself and had other company employees test it. The website now goes live. A few weeks later you start getting emails from irate customers who complain that they are unable to place their orders because certain steps in the “Buy Now” process give errors. You quickly fix the problem. A few days later you get complaints about some other issue and you again react quickly to fix the website. This continues for a few months till the complaints finally halt and things stabilize.

At this point you make some enhancements to your website. A few days later a customer email alerts you to the fact that in the process of making this enhancement you “broke” something else on the website. Again you spend time to find and fix the problem but by now you are perplexed and not a little frustrated. These issues have cost you many customers in the last few months and potentially spread ill will across the broader customer community. It seems to you that the only way to have detected these issues before they went “live” was to have employed a large army of software testers, something your company is unable to afford.

Enter automated software testing. While nothing can replace good human testers, broad test coverage requires some degree of software automation for it to be economically feasible. Automated testing tools can provide a huge workforce multiplier and do a very good job complimenting human testers. Every change to your website no matter how small requires thorough testing to ensure that nothing else was affected. This becomes very time consuming very quickly due to the large number of possible cases to test. A strategy whereby tests are automated using software becomes an economic necessity.

There are two classes of automated testing tools. The first kind, functional and regression testing tools, helps to make sure that the website behaves as it should: for example if a customer clicks on button X, page Y is displayed without errors. Functional and regression testing tools are able to automate a large number of scenarios to ensure that your website works as intended. The second type, load testing tools gauge how well your website performs when subjected to a large stress, such as a large number of simultaneous users. I will be discussing load testing in a separate article.

I will now give you an overview of the basic characteristics of functional testing. Before you can begin any kind of functional test automation you will need to identify the test scenarios you wish to automate. Once this is done, you will need to generate test scripts that cover these scenarios.

A functional testing tool will typically record user interactions with a website. As you perform various operations on your website or application, the tool records every step. When you finish recording, it generates an automated script from your interactions with your website. Alternatively you could use the tool to construct the script by hand. Typically testers tend to do a combination of the two. They will use the recorder to generate the basic framework of their scripts and then tweak the scripts by hand to incorporate special cases.

Scripts can be graphical and/or text based in nature. A good functional testing tool does not require users to have a programming background. Users not proficient in programming will work predominantly with graphical scripts. In most tools graphical scripts will typically show all interactions in a tree structure and users can edit any node of the tree to modify the script. Some users however, who have programming backgrounds may wish to program their scripts. These users will typically work with a text script written in a standard language such as JavaScript or VBScript.

Once you have generated your script you will need to insert checks in your scripts to test if your website is functioning correctly. Such checks are usually called checkpoints. A checkpoint verifies that values of a property obtained when testing the website match expected values. Checkpoints enable you to set the criteria for comparing expected values with obtained values. The expected value of a property is derived from recording interactions with the web site. It is viewed and modified from checkpoints. The current value is retrieved during replay (i.e. during the execution of the test case).

There are many different kinds of checkpoints. A page checkpoint verifies the source of a page or frame as well as its statistical properties. You can check for broken links, verify link URLs, image sources, the hierarchy of HTML tags or even the entire HTML source of the Web page or frame. You can also set thresholds for the loading time of a page. A text checkpoint verifies that a given text is displayed or is not displayed in a specified area on a web page. A web object checkpoint verifies the properties of a web object e.g. the value of an HTML INPUT field. A database checkpoint verifies the contents of a database used by your website.

When you replay a test script, the testing tool will open the recorded application and perform the recorded steps in the same sequence they were specified in the script. As it replays the script it will also run through all the checkpoints you have inserted into the script. In addition, you can test your application’s behavior with varying data inputs. For example you can try to submit a page after entering different values in the edit box of a web page. At the end of the replay a detailed report is typically be generated.

Functional test automation allows you to automate the repetitive testing of a large number of scenarios across your website. Functional testing tools are an important weapon in your development arsenal whose use provides a huge productivity gain and allows for small testing groups to accomplish significantly more work. There is a very strong economic case for the use of Functional Testing Tools as part of the development and deployment cycle of a website.

About the Author:
Umair Khan is Founder and Chairman of
Verisium, Inc., a provider of products for automated functional and regression testing, load testing, bug tracking and test and requirement management.

Verisium is the maker of vTest, an automated functional and regression testing tool for web applications.

Share This

Better to fail for real than fail to really fail. Huh?
We know you’ve experienced this. Let’s say you just added some new functionality into your software, and you run a new build. And let’s say that 50% of your test cases fail. What is the first thing you assume?

We’ve asked this same question as our “teaser pitch” last winter to 100 developers and QA professionals who walked up to our booth at a recent conference, and 95 of them had the same answer! The tests must be broken!

This creates a cascading set of bad assumptions that will make your manager recite the adage about “ASS out of U and ME” on the whiteboard at the next project meeting. Here’s why.

• You assume that the problem is not with your application, it is with the test cases themselves being broken or no longer valid.
• So you spend time comparing the test cases with whatever changed in your new build.
• Then you dig into the test scripts to try to figure out why the test case is no longer passing, and rework them until they pass.
• Or you just give up and try validating by clicking through your old Word document test cases. Fun busy work.

How can you possibly call this testing? Rather than using the test to validate the application, you are using the application to test the test case - which is a program you coded!

Yes, unit tests are important for finding structural bugs in your code. But once a unit test tries to get beyond that granular level of testing, it becomes another fragile program in your development environment.

It is outrageous to assume that relying on coded unit test cases alone offers you any value in functional testing. In fact, the whole process is so manual and highly inefficient, that you wonder if you are doing anything more than making busy work for your own team.

Unit testing has its limits. There are methods people have tried to get beyond these limits, but it is like challenging the theory of gravity.

• Attempting to code for reuse - may seem possible but can only get you to the edge of Unit testing’s limits.
• Attempting to test the UI with your QA group, doesn’t really work if you can’t see those middle and back-end layers.

What makes false failures so dangerous? Besides the fact that they are a morale vampire that will make the team give up on testing, false failures impact the overall effectiveness of testing. If you don’t know if a failing test case is even valid, what do you really learn from testing? It is like a detective that never gathers evidence.

Time to declare war on false failures.

Jason has more than 13 years of experience in executing marketing plans, re-engineering business processes and meeting customer requirements for technology and consumer companies such as HP, IBM, EDS, Delphi, TaylorMade, Sun, Realm, Adaptec, Motorola, Sprint and currently with iTKO. Jason writes articles on a variety of subjects including software testing for http://www.itko.com.

Share This

It won’t matter how effective your WinRunner Team is if no-one outside your immediate organization knows about your accomplishments. For this reason, marketing your WinRunner Team is vital to your success. When times get tough, executives look for cost-cutting measures. The QA group is often the first on the chopping block. If these high level executives don’t fully understand and appreciate the value of your service, they will see the cost of WinRunner licenses and maintenance as well as the highly skilled, but also more expensive WinRunner engineers as a nice place to start trimming the budget. They will not have the time or luxury to launch an investigation to see if these services are really necessary.

The next thing you know, you have a nice library of WinRunner scripts, but no tools to execute them and no one with the skills necessary to modify the scripts as applications are updated. However, if these high level executives have personal knowledge about the benefits of software WinRunner in terms they understand, which are time and cost savings to the business they support, they will be less likely to put it on the chopping block.

High level executives don’t have the time or energy to seek you out and find out what benefits the automated testing has to offer to the organization. Instead, you must make the effort to seek them out. Marketing your WinRunner group is the responsibility of the entire team, but the heaviest burden lies upon upper-management and Vice Presidents because they have daily contact with peers at their level and higher.

Demonstrations for High Level Executives

One of the best ways to market your WinRunner team is to demonstrate to Executives, what you have done with WinRunner tool and how it helps their business. When you know an executive will be in town for a day, arrange for a thirty minute meeting. Executives are busy and everyone wants a piece of their time, so limit the time to thirty minutes. It will be sufficient to demonstrate what you have done with the product as well as how it benefits their business.

When you have confirmed that the executive will be able to attend a demonstration, it’s time to find a suitable conference room. Choose one that has live network connections and a screen to display the laptop image. Schedule the conference room for thirty to sixty minutes before the executive arrives in order to prepare. Use this time to set up the laptop, projector, and perform a dry-run of the script. Verify with the development engineering groups that no loads will be affecting the application you plan to demonstrate. Explain how important it is that nothing impact the environment, which will cause the application to go down during the presentation.

Invite as many members of the WinRunner team as possible so that the executive has the opportunity to meet everyone. However, if this is not possible due to the current work-load, invite only key individuals, preferably the ones who created or currently maintain the script that will be part of the presentation. They know most about it and will be able to troubleshoot any problems that arise more efficiently than someone who is not as familiar with the application or script.

Begin the meeting by making introductions and pass out an agenda so that everyone knows where the meeting is going and what will be covered. Give a brief overview of the application that will be demonstrated. The application should be one that the executive is familiar with and the script should run the length of the meeting (or longer). Ideally the application will have a lot of fields, making hand-typed data entry tedious. WinRunnerl will whiz through the application at an impressive speed.

While the script is running, explain how long it takes to manually run one test case verses how long it take WinRunner to execute one test case. Translate this into one test iteration so everyone can see how much time WinRunner saves on a weekly or monthly basis. Mention that the manual testers, who used to perform this testing, are now free to work on other projects, while this one is testing it’s self. At the end of the meeting, bring up the report to show how easy it is to identify which test cases passed and which, failed.

Executives are usually in back-to-back meetings, some of which run over their time limits. Less-important meetings, such as your presentation, may get rescheduled at the last minute. Don’t be discouraged. Simply reschedule the meeting for a later date. These presentations are not a waste of time. Executives who see the benefits of WinRunner and the cost savings will not only hesitate to cut your program out of the budget, but they will also inform their peers, who are struggling with long testing cycles, of your success. Ultimately, your success is their success.

Take Advantage of Status Reports

Status reports are one of the best ways to demonstrate to the business, on a weekly basis, how much time and money they are currently saving by automating the software testing. Status reports should contain the following figures:

Weekly hours saved per application
Year to date hours saved
Number of application automated
Number of scripts
Cumulative hours saved this week for all applications
Cumulative hours saved to date for all applications
Database or Spreadsheet of Project Statistics

Once the business and upper management gets wind of your WinRunner team’s abilities, be prepared for a windfall of questions. You will be asked over and over about the number of applications that have been scripted, time saved through automation, and a host of related questions. The best way to be prepared is to have a database or spreadsheet with your current project statistics on hand. Not only will you appear organized and efficient, but you will not have to drop your current activities to scramble for statistics. Your project database or spreadsheet should show general and application specific statistics.

Let Others Toot Your Horn

Executives who have had positive experiences with you in the past will spread the word when their peers complain about manual testing or show an interest in automating their software testing.

Beef Up Your QA Website

Most organizations have an internal website with sections dedicated to each group within the organization. If your QA organization doesn’t already have a website, it’s time to create one. A QA website can help you streamline activities such as a project queue that prioritized new projects, and conduct customer satisfaction survey’s, and announce your successes to the rest of the company.

Your QA website will do nothing for your PR unless other groups and organizations have to access it in order to interact with your team. You can begin forcing other groups to access your website by creating a project work queue, where they must complete a form in order to have their project entered into the QA work queue. This is your opportunity to lay down the rules rather than be forced to abide by their rules. There are specific facts that need to be clear before QA can prioritize and -assign resources to a new project.

Departmental home pages generally consist of the group mission statement and basic information. Once people have seen it a few times, they will skim right over it and with out a second thought. What a waste of space! Home Page real estate is the most valuable area of your website because it’s the one page that everyone sees. You best real estate should be reserved for facts that demonstrate your team’s success. It’s not that the team mission statement isn’t important, but rather that the mission statement belongs on another page or at the bottom of your home page, after the statistics. These statistics can be arranged in such a way that they display a running total of the cumulative time saved to date for each application and as a whole.

Town Hall Meetings

Take advantage of Executive Town Hall meetings, which are often used to update employees on the success of the business and visions Executives have for the future. They usually include many top level executives, who have connections in other business units who may be in need of automated testing. It’s not unusual for each Vice President to be asked to stand up and say a few words about their team’s current activities. This is a good opportunity to repeat some of your automated testing statistics or, in some cases, a quick presentation.

Don’t be afraid to contact executives, explaining your success with WinRunner and that you would like to share this with the organization during the Town Hall meeting. Executives rarely have the opportunity to see what is really happening in the “trenches” and will be pleased to hear about your success, especially when it is clearly a cost-cutting measure.

About the Author
Danna Henderson has helped many oranizations automate their software testing with complex, data driven scripts. For more information about successful automated testing with WinRunner, contact WinRunner and Software Test Automation Tools.

Share This