The human ability to process language is at once profoundly intuitive and extraordinarily complex. You’re doing it now, automatically ascribing meaning to this string of seemingly random symbols. If the words were uproariously funny, wrenchingly sad or deathly dull (sorry!) you’d know it without even having to extend much effort.
The technologies surrounding natural language processing, or NLP, offer the possibility of computers reading text or interpreting the spoken word with the same ease and fluidity, despite the inherent complexity. Driven by artificial intelligence, NLP promises to make government processes simultaneously leaner and more responsive. It could free workers from tedious and repetitive jobs, streamlining service requests and empowering them to devote their energies to higher-value tasks.
Here we’ll offer a brief overview of NLP technologies, consider possible federal use cases and chart a potential path forward for government.
As a sub-field of AI, natural language processing seeks to enable computers to understand human language. The machines can’t pick up every nuance, at least not yet, but they can learn a language well enough to translate text and summarize content. It works with the written word and can also be used to interpret and respond to spoken requests. NLP’s journey to the modern era has been fascinating.
In the 1970s, scientists pursued a symbolic or rules-based approach, which meant a machine had to learn everything about a language’s grammar, dictionary and the specific context in order to understand and generate natural language. In the modern era, machines learn via a statistical approach, training on billions of examples of natural language available in digital form. This approach has yielded far more accurate results with much less effort.
Today, we are applying the exciting advances in deep learning to significantly improve NLP’s accuracy, further expanding its applicability across multiple domains and delivering a range of valuable services including transcription, translation, entity extraction, and semantic and conceptual analysis.
Under the hood, NLP relies on two basic concepts: Natural Language Understanding or NLU, and Natural Language Generation, NLG. In their most common usage, these are the engines powering chatbots and intelligent virtual assistants.
NLU depends on algorithms to break down human speech into computable properties or characteristics called feature vectors, with AI helping to refine the recognition of things like intent, timing and sentiment. In this way, NLU is able to understand input via text or speech.
Ideally, NLU looks beyond words to ferret out meaning, getting to the core of communication even in the face of mispronunciations and spelling errors. Such systems rely on a predefined lexicon and a set of grammar rules. Sophisticated systems leverage machine learning and statistical models to determine the most likely meaning.
Natural Language Generation refers to the computer’s ability to generate text, whether by translating speech to written text, converting data to written language, or converting text to audible speech. Text-to-speech and speech-to-text engines rely on NLG to deliver coherent messages, once again backed by a predefined lexicon and a set of grammar rules.
Many of the biggest names in technology have introduced NLP applications, including Microsoft, Amazon and Google, as well as IBM, which offers NLP applications within its Watson AI platform. The big phone makers all have woven NLP into their virtual assistants, including Siri, Bixby and Alexa. Pure plays like Nuance, Nice Systems and IPSoft can also be sources of innovation. On the academic side, Stanford University has been a leader in developing new NLP iterations.