Pros and Cons of OpenSource Test Automation Tools

by admin on November 4, 2014

If you believe what the analysts are saying, Opensource is only now becoming mainstream! A practitioner like you would say “What a blinding flash of the obvious!” and move on to your Jenkins.

There are problems with Opensource adoption. Functionality is not a problem by itself. By now, the abilities and weaknesses of all tools are known. What is often missed in the consequent problem – getting the various tools to talk to each other!

Given the provenance of these tools, you would think that all the tools would work cohesively together. But therein lies the rub – they don‟t! Well, ok, they do provide interfaces that allow you to get each of these tools to talk to each other, but they are like shy Englishmen – you have to spend the effort and time to get them to talk to each other. On their own, they do not integrate
and inter-operate.

Let us look at what needs to be accomplished in a typical Agile project:
* A set of requirements have to be built as a part of a sprint
* These are then translated to user stories
* These stories have to have the appropriate levels of priority, risk weightages
*  Test requirements, and test cases are then created and linked back to user stories
* Testing these cases will require, say, Selenium scripts to be created and executed. Again, these scripts have to be associated to builds.
* When these user stories are delivered, (as daily drops), they need to be tested „continuously‟
* The resulting defects have a life cycle of their own but do need to be traced back to the appropriate user story.
* Data across different tests are crunched to be able to throw up predictive „test efforts‟, test effectiveness and other predictive metrics

Now if you chose to manage this project using various Opensource, a typical selection would  look like this:

* Kunagi to manage sprints and user stories
* Test Link to manage test requirements, test cases
* Selenium to manage automation
* Jenkins for continuous integration and
* Bugzilla for defects.
* Jasper for test analytics

Each of these tools is great by itself (and has a great  following in the community). However, (there is always a „however‟ with Opensource tools!), they each come with their own way of setting up users, privileges, databases etc. What is lacking is seamless
integration – the ability to ensure that information and relationships are inherited both upstream and downstream.

The mainstream tools from HP or IBM do not have these issues as they are integrated (in fact working with other tools in the family is a key requirement) from the word “Go”. That integration, however, comes at a huge cost and does not go far enough. Which is probably why you are reading an article on Opensource tools, anyway!  There is a crying need for a single point solution to integrate all Opensource tools. Let us look at the characteristics that would define this integration tool:
Integration, not functionality
The existing toolsets are close to best-in-class. There is no need to add new functionality in the name of augmenting the tool. The relevant tool community knows what works best and it is ideal to leave those decisions to them. What is needed is a way to make them more coherent when they work with each other. This ensures a seamless integration across all the tools across the
entire lifecycle. If it can be done with minimal or no intervention, that is so much icing on the cake!

The key requirement is that the inter-component workflow be logically coherent.

Management & Control

Most Opensource tools are stand alone. What is required is the ability to form a clear, end-to end picture of the status of the project. Sufficient information for project management, estimation, risk analysis, functional and defect prioritization should be garnered. With both upstream and downstream integration, the tool should be able to provide complete lifecycle coverage, and work for different lifecycle models adopted. It should work with scalable development models, and across a multiplicity of projects within an organization. A defect tracking tool should be able to tag defects to requirements picked up from the appropriate
requirement tool.

The ability to link together views across the different lifecycle elements is a functionality sorely missed in Opensource tools. In an ideal scenario, the tool should have the ability to identify a requirement and view its behavior downstream in terms of its behavior during test, its impact on release. Or, take a defect, and trace it backwards to weaknesses in requirement articulation,design or development. It should allow you to manage Agile user stories, link them to tests, link the tests to automated scripts, and link the test execution data to defects.
Today the data generated at various points is only looked at that point with no attempt being made to glean insights from the patterns residing in the data. This, obviously, is because such analytics would mean integrating another tool with its attendant integration problems. Each of the Opensource toolsets stores a rich amount of data. Any integration tool should have, built-in, a powerful, customizable analytics engine that has the ability to weave the data together, and then leverage data along different dimensions. For example, analytics should be able to identify potential defect hotspots given a certain context. This feature will help testers to concentrate their efforts on the weakest interfaces instead of following a spray-and-pray approach. We need the ability to look beyond the data and to comprehensively predict patterns for the future.

Plug & Play Capability
Given the diversity of toolsets in the Opensource world, there is a need to achieve seamless integration across a diverse set of tools. One should be able to plug in different tools for various functionalities during the course of the project life cycle with minimal changes. The replacement could be another Opensource tool or even a COTS tool. Enabling the strong integration & interworking between Opensource toolsets is the need of the hour. These integrated features will help in the faster adoption of Opensource tools while being an invaluable productivity enhancer. It would pay for itself within a very short time merely from the effort saved, leave alone the management and control capabilities or analytics.

#SQASolutionShare on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone

Previous post:

Next post: