Making synthetic, authentic

Or consider, for a moment, the many ways AI has been instrumental in helping us tackle COVID-19. Convolutional neural network (CNN)-based models have helped detect and classify in real time COVID-19 infections in patients employing chest X-ray images. Researchers even developed a fully automatic deep learning system to generate a diagnosis and prognosis of COVID-19 using computed tomography (CT) scans. AI-based methods have been used to track COVID-19 spread over time and place, to conduct disease surveillance by scanning public spaces for people with potential COVID-19 infection, and to enforce various social distancing measures or lockdowns. An AI-enabled mobile app, called Aarogya Setu and launched by the government of India, allows users to check their safety status based on whether they have crossed paths with Covid-19 positive patients. Most importantly, AI helped researchers dramatically accelerate the genomic sequencing of SARS-CoV-2 and its variants and develop potential treatment approaches and even vaccines.

Moreover, synthetic data can be generated in many structured and unstructured forms: text, tabular, and even media data such as video, images, and sounds. Google’s Waymo, for example, is using AI to generate simulated camera images from sensor data collected by its autonomous-driving vehicles. It is then using those simulated images to train its cars.

In short, synthetic data has become a boon for researchers, data scientists, and organizations that depend upon AI. Perhaps that’s why industry analyst Gartner predicts that, by 2024, 60 percent of the data used for AI development and analytics projects will be synthetically generated. It also explains the rapid proliferation of startups — such as MOSTLY AI, Datagen, and Synthesis AI — that are filling out an entirely new industry focused on synthetic data catering to a wide assortment of industry sectors.

But how do we generate synthetic data rapidly, at great scale, and accurately? One way is to use AI itself to generate that data. We call this generative AI. Another important dimension of the unreal world, generative AI can consist of a wide array of data types, including text, visual data, and multimedia. Examples of this are when technologies draw and paint pictures or use information gathered on the internet to create articles. In essence, generative AI enables computers to learn patterns from a large amount of real-world data and generate new content that mimics those underlying patterns.

One common approach to generative AI is through the use of generative adversarial networks (GANS). GANS, fundamentally, are generative modeling architectures that, during training, pit two neural networks — a generator and a discriminator — against each other. The generator network generates new data or content based on the patterns it is learning from real-world source data. As it does so, the discriminator network’s task is to differentiate between the source real data and the generated data from the generator. This creates a feedback loop in which the generator constantly learns to produce more realistic data, while the discriminator gets better at differentiating fake data from the real data. GANS can generate a wide variety of content, including images, photographs of human faces, realistic photographs, face aging, and 3D objects.

This powerful architecture has made GANS the primary technology used to create deepfakes, which has justly earned the technology considerable scrutiny. But this same type of generative model can also be easily steered toward highly beneficial use cases. For example, they can be used to improve fairness and remove bias in credit and loan decisions by generating training data that removes biases from protected variables (e.g., gender, race, …) while the discriminator tries to guess the values of the protected variables based on the generated data. When it becomes impossible for the discriminator to guess these values, the generator has successfully learned to produce debiased data that can be used to train the credit and loan decision model.

This is heady stuff. More and more enterprises are becoming architects of the unreal world. And as they push AI into more collaborative and creative roles, they are blurring the lines between what’s real and what isn’t. Advances in generative AI, and GANs in particular, are making the creation and use of synthetic data that is incredibly realistic – while being unreal – possible.

As we enter a world with synthetic realness, where AI-generated data — in the form of synthetic data, images, and chatbots, as well as augmented and virtual realities — convincingly reflects the physical world, we are forced to face the questions of what’s real, what’s not, and perhaps more importantly, when do we care? When we see the news, we want to know if the video of the president is real – but when we watch the latest Doritos commercial, maybe it doesn’t really matter. And sometimes, we may actually prefer the unreal, like when we speak to a synthetic nurse about a potential sexually transmitted disease or train an AI model with synthetic data adjusted to counter historical discrimination.

As synthetic realness progresses, conversations about AI that align good and bad with real and fake will shift to focus instead on authenticity. Instead of asking “Is this real?” we’ll begin to evaluate “Is this authentic?” based on four primary tenets:

As conversational AI technologies continue advancing and redefining the state of the art in commercial customer experience, we should expect to see federal agencies similarly follow suit as they strive to improve their own citizen experiences around the government services they deliver. As with chatbots today, synthetic data will surely play a key role in developing those technologies in the future.

2. Open data

One of the earliest federal use cases for synthetic data can be found at the U.S. Census Bureau, which regularly collects vast stores of national data of immense value to many fields of research.

In 2001, the Census Bureau was authorized to integrate person-level micro-data from its longitudinal household survey, called the Survey of Income and Program Participation (SIPP), with IRS tax and earnings data and Social Security Administration retirement and disability benefits data. The resulting data trove offers the most comprehensive information available on how the nation’s economic well-being changes over time, and it’s a goldmine for academics, researchers, economists, and policy makers. With SIPP data, they can examine, for example, national income distributions, the impacts of government assistance programs, and the complex relationships between government tax policy and economic activity at the local levels.

The problem for Census, however, is that the highly detailed nature of the SIPP data makes it particularly sensitive because the micro-data could be used to identify specific individuals. To make the data safe for public use while also retaining its research value, the Census chose to create synthetic data from the SIPP data sets. The result is the SIPP Synthetic Beta (SSB), a Census Bureau product that was first made public in 2007 and continues to be updated and released periodically.

3. Federal healthcare

Similarly, the NIH’s N3C Data Enclave is an open data initiative aimed primarily at advancing COVID-19 research. Since the database was opened to researchers in September 2020, it has grown to include billions of rows of data representing more than 5 million COVID-19 positive patients, making it the largest open U.S. database of data from patient electronic health records. Because of its advanced informatics technologies and data linkages to demographic, mortality, and other information, the database helps researchers create clearer pictures of COVID-19 health outcomes among different communities and enables them to find patterns faster than traditional database methodologies allow. Moreover, the N3C Data Enclave has become useful for research well beyond COVID — researchers have used it to improve our understanding of health equity, diabetes, cancer, HIV, rural mortality rates, and chronic obstructive pulmonary disease as well.

Under the open data initiative, scores of federal agencies and subagencies have already made data sets freely available to researchers on Data.gov, the main web portal for open government data. But some data sets cannot be shared because they could reveal personal details of specific individuals.

As mentioned, synthetic data is increasingly indispensable for medical researchers who must work around data privacy and compliance obstacles. But deep generative models, such as GANS, are also becoming critical tools for creating more robust data sets to train AI-driven diagnostic tools that can be safely applied across many demographic categories.

AI algorithms need to be trained on large, diverse datasets to be generalizable across a variety of populations and to ensure they are not biased in ways that affect their accuracy and reliability. Historical patient data, such as images or scans of certain maladies, often lack the needed diversity and representation to achieve this. For example, a 2020 analysis of data used to train image-based diagnostic AI systems found that approximately 70 percent of the studies that were included used data from three states, and that 34 states were not represented at all. Algorithms developed without considering geographic diversity, including variables such as disease prevalence and socioeconomic differences, may not perform as well as they should across a varied array of real-world settings. To help address these shortfalls, some developers of AI-based diagnostic tools are looking to GANs. GANs are showing much promise in generating realistic images of skin lesions, pathology slides, colon mucosa, and chest X-rays in a range of imaging modalities, according to recent studies.

For these reasons, federal medical researchers are sure to make greater use of generative images and data in their work. But synthetic data will also carry big implications for federal regulatory agencies in the healthcare arena.

In 2021, the U.S. Food and Drug Administration (FDA) unveiled a new “action plan” for how it will regulate AI-based software as a medical device (AI-SaMD). The plan recognizes that FDA’s traditional processes for regulating software and AI products must change to keep pace with today’s innovations – namely, AI, machine learning algorithms and synthetic data embedded in many of today’s devices continue to change over time long after they go to market and are in use. With its action plan for AI-SaMD, FDA voiced an intent to review AI/ML software products from pre-market development to their post-market performance. FDA officials also are reportedly considering the use of synthetic data as an approved method for ensuring that data sets used to train AI/ML models in medical devices are diverse and accurately reflect the patient population in which the technology will be used.

4. Constituent services

The Veterans Affairs Department intends to leverage synthetic data as part of a new initiative to reduce veteran suicides by 10 percent. By parsing synthetic data, department leaders hope they can better clarify veteran challenges and refine predictive risk factors so department programs can be more proactive in addressing the problem. “It’s very clear that we need to use reliable and timely data to identify and address issues impacting our veterans, while also ensuring privacy,” said Dr. Carolyn Clancy, assistant under secretary for Health for Discovery, Education and Affiliate Networks at the Veterans Health Administration (VHA). “We believe synthetic data, modeled to precisely mirror real veteran data, while protecting veteran privacy is a great path forward.”

As AI progresses and models improve, enterprises are building the unreal world. But whether we use synthetic data in ways to improve the world or fall victim to malicious actors is yet to be determined. Most likely, we will land somewhere in the expansive in-between, and that’s why elevating authenticity within your organization is so important. Authenticity is the compass and the framework that will guide your agency to use AI in a genuine way – across mission sectors, use cases, and time – by considering provenance, policy, people, and purpose. Ultimately, it will unlock new attitudes towards and experiences with AI, unleashing the benefits of the unreal world.

To learn more about how The Unreal will affect federal agencies — including potential challenges that federal leaders should look out for and steps that agencies can take to prepare — please read the full PDF.