One of the biggest challenges — and there are many — facing the artificial intelligence (AI) realm today is inherent biases created by limited training data. Researchers have already demonstrated how Amazon’s facial analysis software, for example, distinguishes gender among certain ethnicities less accurately than other services, while Democratic presidential hopeful Senator Elizabeth Warren has called on federal agencies to address questions around algorithmic bias, such as how the Federal Reserve deals with money lending discrimination.
And it’s against that backdrop that “in-the-wild” software-testing company Applause is looking to “re-invent” AI testing with a new service that better detects AI bias through crowdsourcing larger training data sets.
By way of a brief recap, Massachusetts-based Applause, formerly known as uTest, offers companies including Google and Uber a different kind of app-testing platform, one that taps hundreds of thousands of “vetted” real-world users around the world who help to squish bugs and iron out usability issues — it’s all about harnessing the power of the crowd rather than running tests entirely in contrived laboratory settings. The company had raised north of $ 115 million before it was acquired by investment firm Vista Equity Partners in 2017.
Keep it real
A key facet of the Applause platform is not only the sheer number of crowdtesters in its community, but the demographic diversity spanning language, race, gender, location, culture, hobbies, and more. And this will likely be among the main selling points as Applause looks to reappropriate its technology to offer companies access to diverse AI training data.
“Not only will this improve AI experiences for consumers everywhere, the breadth of the community also has the potential to mitigate bias concerns and make AI more representative of the real world,” noted Applause product VP Kristin Simonini.
Applause’s AI training and testing service is offered across five core AI types, covering: voice, optical character recognition (OCR), image recognition, biometrics, and chatbots. So for example, if a company needs to quickly source varied training data for a virtual voice assistant, Applause users in various locales could be called upon to record and submit specific utterances. Equally, they could submit photos of objects or places, or interact with chatbots to iron out any bias. They could even be asked to submit selfies and fingerprints if they’re testing biometric-based security prodcuts.
Perhaps more importantly here, Applause promises speed and scale for both gathering training data and testing the outputs, allowing companies to garner rapid and iterative feedback from end users in real time. So it could work like an ongoing feedback loop, with the gathered data used to improve AI algorithms, and then re-tested on the the Applause community.
“Users want AI to be more natural, more human,” Simonini added. “Applause’s crowdsourced approach delivers what AI has been missing: a diverse and large collection of real human interactions prior to release.”
There are other similar initiatives out there at the moment. Amazon’s Mechanical Turk, for example, can be used for crowdsourcing data for machine learning experiments, while startups such as DefinedCrowd helps create bespoke datasets for AI model training, as does Germany’s Clickworker with a specific focus on machine vision and conversational AI.
With more than a decade of software testing with some of the biggest tech companies in the world, however, Applause is well-positioned to harness its existing presence in the developer community to offer a community of vetted crowdtesters who can only help improve AI applications by reducing bias.