Too many cooks can help make quality broth

An insight into the practicalities of Crowd Testing

The crowd testing concept

If you were ever encouraged to do well at school with the promise of a reward, perhaps a bar of candy, you will already be familiar with the incentive to participate in crowd testing.

Testers are encouraged to take part in crowd testing, perhaps with the offer of a small cash reward for every bug found.

Releasing beta versions is another example, where the aim is that the many will find bugs before the final version is released but now it worth re-visiting the merits of crowd testing as an approach as it is now attracting a growing number of project groups.

What does it offer?

Typically the Application Under Test is passed onto several testers in various locations and it is quickly possible to test a number of scenarios:

• Interface – There are tiny monitors that can fit anything from your watch frames to gigantic wall monitors. Do you have enough room for all of it? With your application tested by teams all over the world, you can be sure that is has been tried on all the various sizes and shapes of hardware that are being invented every minute.
• Platform – A room sized memory has now shrunk to a micro SD card or is it grown even smaller as we speak? A huge processor has shrunk to something that fits into a smart phone. Crowd testing can be a solution to test on all of these.
• Compatibility – With versions of Operating Systems and new browsers mushrooming every day, you can be sure that a software lover is always hunting and using the latest offering. What better way than spreading this test effort across the globe?
• Localization – A very small percentage of the world understands English intimately. How do we cater to the majority? Go global with crowd testing and you can ensure the product is road tested by people who don’t use English as a first language.

How to crowd test efficiently

To set up a crowd testing environment, it helps to have a framework.

Product – It should not be offered up too early in development or you risk ending up with several duplicated defects around known crashes and broken functionality. A good approach would be to do at least a minimal internal verification before throwing it open to a larger audience.

Place – Work out how to manage crowd testing input at the outset. This includes decisions on how the product is going to be shared to the defect tracking systems to the communication channels.

Process – Process is the key. Before you venture into crowd testing, make sure you brainstorm. Think of all the risks involved and mitigation plans. Think of tracking / channelizing to how you are going to advertise, screen and pay. Plan your test life cycles and build releases.

People – A committed team will be the make or break of any project’s success. With crowd testing the following are crucial:

• Convinced management – Buy-in at management level has to be a consideration.
• Dedicated control group – To track the incoming defect flow. From simple things like, matter s such as who is rewarded if a defect is duplicated to following up on ‘need more information’ defects.
• Crowd of testers – Although it is a significant advantage to throw the net wide, agreeing an initial screening and skills criterion is a requirement.
• Trustable tracking systems – This should include defect tracking systems, agreed communication channels and process to be sure of the safety and security of the product, such as firewalls and data confidentiality.

Crowd testing benefits

As we all know, bugs get costlier to fix as they near a release and are even higher once the software has gone live.

From our metrics, we see that crowd testers helped find 20% more bugs than those identified by in-house testers.

This saves the cost of a later fix and the cost of increasing the number of core testers to bulk up the project team.

Crowd testers also found more compatibility issues than testers from the project team as they work with a wide range of OS, browsers and so on. For example, crowd testers identified that the drag and drop controls of the ‘add to cart’ page did not work on one browser and that the logo was misaligned outside the frame on another.

Using a crowd helped make sure that all users, irrespective of the software version they will have to hand, will be able to use the system as it has been designed to operate.

Here are a few metrics based on a case study of a crowd testing effort for a retail desktop application, tested with visiting testers from other groups:

Domain : Retail
Testing types : Functional, Exploratory, Crowd
Period : 6 weeks (4 weeks internal + 2 weeks crowd)
Crowd testers : 40% the project team + 40% outside the project team + 20% outside the organization
Test Process : The project team was responsible for initial Functional testing and a few weeks into their formal tests, the application was made available outside the project team / organization for crowd testing.

20% defects came from the crowd.

• The defects from the crowd were concentrated around usability and compatibility

• 73% of the compatibility defects came from the crowd

• 23% of the invalid defects identified by the crowd were because of lack of understanding of functional flow or incorrect environment setting

When do you have the right pick of the crowd?

When you have thought about your product, place, process and people, you are ready for crowd testing. Since the people involved in testing are the most important consideration, this section focuses on tips to sourcing your crowd testers.

Advertise effectively 

Social media and the internet can be used but be sure that it is for the right audience. Make sure your advertisement includes rules and regulations.

You don’t want to end up resolving issues related to understanding of terms. Your brainstorming session should help in this. Keep the rewarding / notification fair and simple.

Screen before you deploy 

Be sure to conduct a basic screening of all those who apply and are interested. Have defined criteria of who will fit such as computer / technology savvy, domain expertise and language efficiency.

You can have a rating system and a qualifying guideline before you bring on-board a crowd tester.

Remember this rating guideline should be formulated based on the product that is being tested. It may have to be revisited for a different kind of product.

Background check 

Have a system in place to ensure you are getting what has been promised by the tester. Clarity is important.

You need to be confident that the tester is interrogating the right aspect of the software or hardware.

This will not only help you to be sure but will also to group your testing effort / results.

Group and limit your resources 

Decide where you want the focus of testing to be.  Let us say hundreds of testers sign up, while you screen them also have check which combination this resource is going to help you with.

Too many testing one combination and too few in another can lead to incorrect metrics. According to your need, limit the testers in a combination and group them.

This will also help you understand where defects are concentrated and the most highly used platform or browser.

Plan your Knowledge Transfer sessions – Webinars and online videos are effective. Keep short, easy to access and understand and keep a tight focus on what needs to be established.

This is important as you don’t want to end up with too many invalid defects. It is also a good idea to only open out user friendly interfaces for crowd testing initially. See how it works before you try the entire application.

And therefore…

Two is company and three is a crowd – but in this instance, the crowd can provide not only a positive but an invaluable contribution.

Interrogating functionality, discovering what has been overlooked and allowing companies to build rigour before application