Baseline, a machine learning newsletter - April 2022

Welcome to the April 2022 Baseline, the Accenture Federal Services Machine Learning Newsletter, where we share thoughts on important advances in machine learning technologies likely to impact our federal clients. This month we cover the following topics:

• Applying Transformers to new domains
• Simplifying the application of anomaly detection through a new open-source library
• Building robust probabilistic time series forecasters

AlphaCode: Language models can generate code too

Since the introduction of the Transformers architecture in 2017, novel approaches to tasks such as text generation and large document summarization have become milestones of progress within natural language processing (NLP). Extending these capabilities to a new domain, a new system developed from DeepMind, called AlphaCode, utilizes Transformers architecture to generate code solutions to various text-based problem descriptions. AlphaCode solutions already performed at the median level for human competitors at Codeforces, a competitive programming platform, during ten recent contests. The system was first trained on open-source Github code and then tuned on a curated Codeforces dataset composed of coding problems and solutions. For a given problem, AlphaCode generates multiple solutions, which are then filtered and reranked to determine the most promising programs. DeepMind is releasing their dataset of problems and solutions on GitHub, along with extensive testing to evaluate submitted programs. The development of this benchmark dataset along with evaluation tests will allow other models to be more quickly analyzed and evaluated for performance, fueling further innovation in this problem domain.

Anomalib: Simplifying anomaly detection

Anomaly detection in machine learning is a task which aims to differentiate between normal data and abnormal data in a dataset in order to determine outliers. Since anomalies are, by their nature, rare and unusual, it is generally difficult to acquire and label enough examples to perform supervised learning. Therefore, unsupervised anomaly detection, which does not require examples of anomalies to train the model, is often a more practical approach. Generally, unsupervised anomaly detection is performed by comparing potentially anomalous data to a learned normal distribution of typical data.

Researchers at Intel have noted a relative lack of benchmarking algorithms available for unsupervised anomaly detection. In response, they have created a new library, Anomalib, that includes algorithms, experiment tracking, visualization tools, and hyperparameter optimizers to serve as a toolkit for developing and implementing custom unsupervised anomaly detection models. Anomalib will help machine learning developers streamline the implementation of anomaly detection models into production.

A robust probabilistic time series forecasting framework

Time series forecasting is an important task when it comes to decision making, automation, and optimization in business processes. Although traditional statistical methods have previously been successful at this task, recent advances in neural architectures and a greater availability of time series data have enabled deep neural networks to become state-of-the-art in time series forecasting. Traditionally, one challenge to using deep neural networks for time series is their susceptibility to small adversarial input perturbations, both random and intentional. These small variations in the input are easy to overlook by humans but may cause the model to misclassify the outputs. In an effort to develop more robust probabilistic forecasters, researchers at AWS AI Labs and Seoul National University have implemented a framework which rigorously defines perturbations in a mathematical sense and implements a randomized smoothing technique to achieve robustness against perturbations. The researchers verified the efficacy of this technique through extensive experiments on real world datasets. By presenting a solution to make deep forecasting methods more adversarially robust, this work will hopefully drive further innovation in applying deep learning to time series forecasting, and lead to more widespread adoption of these methods.

Accenture Federal Services is a leader in artificial intelligence for the U.S. federal government. Our Machine Learning Center of Excellence, Discovery Lab, and Advanced Research Group continually assess, develop, and adapt the world’s most innovative techniques and emerging technologies for mission-critical applications.