October 24, 2016
Machine testing times for machine learning
By: $name

Increasingly, businesses are integrating machines and software intelligence to make their organisations more efficient, capture larger market share and be the digital disrupters in their industry. This creates new opportunities for software developers and testers and redefines what’s possible within businesses. I’m going to discuss an important but underreported facet of intelligent automation that’s driven by machine learning—how to test these self-learning machines, but before I do so it is useful to define our terms.

Currently, "Artificial Intelligence (AI)", "machine learning" and "self-learning machines" are used interchangeably. I believe this is not entirely accurate. For me, "Self-learning machines" and "machine learning" apply to machines that have been programmed to progressively learn from structured and unstructured data, either assisted by their developer or not. AI, meanwhile, is a much more nebulous term. It describes the future outcome of machine learning: That moment where a machine learns enough from various other sources to become "intelligent" and human-like.

We’re at an important stage in the development of AI: We’ve created machines that can improve their own operation without human input. But how did we get here? There have been three stages:

  1. Self-correcting machines: Machines programmed to make decisions based on external inputs and take corrective actions, e.g. network routers that reroute traffic based on bandwidth congestion that have been around for some time now.

  2. Self-modelling or Resilient machines: Machines capable of not only correcting themselves, but predicting a potential problem and taking action, e.g. resilient storage arrays that redistribute loads in response to temperature increases.

  3. Self-learning machines: The most recent in this progression—not just self-correcting and resilient but self-learning machines, e.g. consumer services such as Netflix or Amazon which learn user behaviour over time to offer personalised services and choices of products.

A self-learning machine will live or die on the quality of its data. And that applies to testing as well. So where should this data come from? Testers have two choices:

  1. Source the data from customers – A good example is A/B testing, where customers can be provided the option of using a new system while maintaining the option of existing system use as well. The data collected from this parallel usage can then be used to determine the quality. A key consideration for A/B testing is that it must be measurable and set up to answer specific questions. One great enabler for A/B testing is crowd testing, where selected talent can be sourced through the crowd that is diverse in terms of location, age, preferences and hence their usage patterns. We believe crowd testing is an important option for various types of testing and are investing in it accordingly.

  2. Demarcated data from prior usage – A key element of teaching the machine to develop the self-learning capability is training data and separating training and testing data is essential. Testing data should be demarcated and set aside before training of the machines begins. Using same data for training and testing leads to overfitting, resulting in higher possibility of the machine incorrectly identifying data relationships that don’t exist. So, in cases where we don’t have enough data to secure a clearly demarcated testing data set, testers can use techniques which use existing data sets to generate new data sets for use in testing like boosting, bagging and stacking.

In this new world the role of the software test engineer will change; as will that of the developer. The distinction between development and test starts to thin out as programmers need to do both continuously to create the desired machine. This isn’t so much about test engineering being made redundant so much as another example of how people will need to retrain to adapt to digital disruption and expand their responsibilities and take ownership of quality.

In the age of intelligent automation, success will come to businesses with the right skills, the right development culture and the right focus on testing and quality. Take a look at your technology capabilities and ask yourself whether you think it’s ready for the future with self-learning machines.

Popular Tags

    More blogs on this topic