If you were ever encouraged to do well at school with the promise of a reward, perhaps a bar of candy, you will already be familiar with the incentive to participate in crowd testing.
Testers are encouraged to take part in crowd testing, perhaps with the offer of a small cash reward for every bug found.
Releasing beta versions is another example, where the aim is that the many will find bugs before the final version is released but now it worth re-visiting the merits of crowd testing as an approach as it is now attracting a growing number of project groups.
Typically the Application Under Test is passed onto several testers in various locations and it is quickly possible to test a number of scenarios:
To set up a crowd testing environment, it helps to have a framework.
Product – It should not be offered up too early in development or you risk ending up with several duplicated defects around known crashes and broken functionality. A good approach would be to do at least a minimal internal verification before throwing it open to a larger audience.
Place – Work out how to manage crowd testing input at the outset. This includes decisions on how the product is going to be shared to the defect tracking systems to the communication channels.
Process – Process is the key. Before you venture into crowd testing, make sure you brainstorm. Think of all the risks involved and mitigation plans. Think of tracking / channelizing to how you are going to advertise, screen and pay. Plan your test life cycles and build releases.
People – A committed team will be the make or break of any project’s success. With crowd testing the following are crucial:
As we all know, bugs get costlier to fix as they near a release and are even higher once the software has gone live.
From our metrics, we see that crowd testers helped find 20% more bugs than those identified by in-house testers.
This saves the cost of a later fix and the cost of increasing the number of core testers to bulk up the project team.
Crowd testers also found more compatibility issues than testers from the project team as they work with a wide range of OS, browsers and so on. For example, crowd testers identified that the drag and drop controls of the ‘add to cart’ page did not work on one browser and that the logo was misaligned outside the frame on another.
Using a crowd helped make sure that all users, irrespective of the software version they will have to hand, will be able to use the system as it has been designed to operate.
Here are a few metrics based on a case study of a crowd testing effort for a retail desktop application, tested with visiting testers from other groups:
Domain : Retail
Testing types : Functional, Exploratory, Crowd
Period : 6 weeks (4 weeks internal + 2 weeks crowd)
Crowd testers : 40% the project team + 40% outside the project team + 20% outside the organization
Test Process : The project team was responsible for initial Functional testing and a few weeks into their formal tests, the application was made available outside the project team / organization for crowd testing.
20% defects came from the crowd.
• The defects from the crowd were concentrated around usability and compatibility
• 73% of the compatibility defects came from the crowd
• 23% of the invalid defects identified by the crowd were because of lack of understanding of functional flow or incorrect environment setting
When you have thought about your product, place, process and people, you are ready for crowd testing. Since the people involved in testing are the most important consideration, this section focuses on tips to sourcing your crowd testers.
Social media and the internet can be used but be sure that it is for the right audience. Make sure your advertisement includes rules and regulations.
You don’t want to end up resolving issues related to understanding of terms. Your brainstorming session should help in this. Keep the rewarding / notification fair and simple.
Be sure to conduct a basic screening of all those who apply and are interested. Have defined criteria of who will fit such as computer / technology savvy, domain expertise and language efficiency.
You can have a rating system and a qualifying guideline before you bring on-board a crowd tester.
Remember this rating guideline should be formulated based on the product that is being tested. It may have to be revisited for a different kind of product.
Have a system in place to ensure you are getting what has been promised by the tester. Clarity is important.
You need to be confident that the tester is interrogating the right aspect of the software or hardware.
This will not only help you to be sure but will also to group your testing effort / results.
Decide where you want the focus of testing to be. Let us say hundreds of testers sign up, while you screen them also have check which combination this resource is going to help you with.
Too many testing one combination and too few in another can lead to incorrect metrics. According to your need, limit the testers in a combination and group them.
This will also help you understand where defects are concentrated and the most highly used platform or browser.
Plan your Knowledge Transfer sessions – Webinars and online videos are effective. Keep short, easy to access and understand and keep a tight focus on what needs to be established.
This is important as you don’t want to end up with too many invalid defects. It is also a good idea to only open out user friendly interfaces for crowd testing initially. See how it works before you try the entire application.
Two is company and three is a crowd – but in this instance, the crowd can provide not only a positive but an invaluable contribution.
Interrogating functionality, discovering what has been overlooked and allowing companies to build rigour before application
By Uma Raj
By Uma Raj
By Abishek Balakumar