When to go for automation?

I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?

Pros of Automation

If you have to run a set of tests repeatedly, automation is a huge win for you
It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nighlty)
Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

Running tests manually can be very time consuming
Each time there is a new build, the tester must rerun all required tests - which after a while would become very mundane and tiresome.

Other deciding factors

What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating

There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?

Yes, and it will cost a little
Yes, but it will cost a lot
No, it is no possible to automate

How important is this test scenario?

I must absolutely test this scenario whenever possible
I need to test this scenario regularly
I only need to test this scenario once in a while

If you answered #1 to both questions – definitely automate that test

If you answered #1 or #2 to both questions – you should automate that test

If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?

Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are

Reevaluate – do I really need to run this test this often?
What’s the cost of doing this test manually?
Look for new testing tools
Consider test hooks

What is Test Frame

work :

It is an executed environment for automated Tests

By using test framework we can reuse all the functions scripted once repeatedly whenever required

Explain what can term as a Good

Framework :

It should be modular and maintainable

- Each module should be separate and independent from all the other modules

- Advantage of this is that functionality available within each module can be readily expanded without effecting any other part of the system

- Should define documentation standards and template

Advantages of Framework

n flexibility to manage all files

n Maintenance

n Easy to change the testdata or scripts.

n Error handling can be done effectively

Explain Key word driven Framework

There is a single script which is used for all the modules. Internally it makes use of some more scripts/ compiled modules, depending on certain conditions. All the scripts are re-usable in nature. They are executed while testing different modules of the application. No script has got object/ window names hard coded in it. All the object/ window names are derived from the excel sheet.

1 - Path of the directory into which scripts are extracted.

2 - “ExcelFiles.xls” This file contains list of data tables to be executed.

It can be found in “..\\WR Scripts\Excel” directory.

This is the Main for loop under which the whole script is executed. It

acts like a wrapper data table

3 - All the relative paths are derived in the script. The data tables,

compiled modules, GUI map files are loaded in the script.

4 - Second data table is opened here which runs in a nested for loop.

This data table contains all the input data . This second data table plays a crucial part in the execution of business scenarios.

5 - There is a switch-case statement that is defined here. It has

equivalent case definitions for various object types, validations,

script/ function calls etc.

6 - Finally at the end of the two for loops all the compiled modules,

GUI map, data tables are closed and unloaded.

WR Scripts |

|--SCRIPTS

|--EXCEL

|--GUI

|--Compiled Modules

|--SetupDocs

|--RESULTS

|--Check list

Let all the Scripts, GUI files, data tables etc.. go into the respective folders only. This enables to place the entire suit of scripts in any directory. And one can load all the assets like GUI, script etc.. dynamically (by deriving the relative path)

Object Names/ : Derived from the header of data table. There is a

Validation/ keyword prefixed to the each parameter of header. This

Special actions key word plays a vital role in deciding which case to be

executed inside the TSL script.

Ex : If it is “WIN_LoginPage” then based on the keyword

i.e, “WIN” the corresponding case statement. will be

executed In which “set_window()”

function will be executed.

Rephrase the above statement (example).

Input values : Derived from the various cells of the data table

Formulating conventions :

· Conventions need to be formulated for different case statements to be executed.

Ex :

1. WIN_LoginPage

                              case "WIN":
                                    winName=colName[i];
                                    set_window(winName,sync_time);
                                    break;

2. BTN_Submit

               case "BTN":
                     btnName=colName[i];
                     button_press(btnName);
                     web_sync(sync_time);
                     break;

What is Data Driven Framework :

Data Driven scripts are those application specific scripts coded in the automation tools and then modified to accommodate varible data…

Variable will be used for key application input fields.

Main script

Load the relevant Script, Excel files from Plan_Data folder

WR Scripts |

|--Tests

|--Scenario

|--GUI

|--plan_data

|--LIB

Explain Method driven Framework

Modular script

Closingestdata access table..

Business Process testing Approach in Quality Center

It is first complete role based test automation system designed to enable functional experts to build , data drive and eecute test automation without any programming knowledge

Its basically is an improved version of keyword driven Testing

And Best part is that it allows English representation of Test cases…

Actually one needs to define test flows througha web based interface by declaring what steps to take and what data to be used…

It can also generates a Test plan based on test definations developed using BPT