As you may know, the discipline of Natural Language Processing, or the problem of whether and how we can make machines that can understand and use human language in an intelligent way, is almost as old as the question of what is Artificial Intelligence.

It all dates back to the 50s, when Alan Turing, usually acknowledged as the “father” of AI as a field, proposed a thought experiment known as “the Turing test”. The essence of this thought experiment is simple: Turing stated that a machine can be considered intelligent if it can carry on a conversation so convincingly that a human cannot distinguish whether they are interacting with a machine or another person. The Turing test serves as a benchmark to assess a machine’s intelligence: if a machine is proficient enough at keeping a conversation with a human and at making the human think that they’re talking to another human rather than, for example, a computer program, then that machine can be considered to be intelligent.

Soon enough after Turing proposed his test, several attempts were made at creating Artificial Intelligence or, in other words, a computer program that was able to hold conversations with human users in a convincing, intelligent way. One example is ELIZA, one of the first chatbots in history, which was programmed in the 60s to simulate conversations between psychotherapists and their patients. 

However, until the 1980s, the majority of natural language processing systems relied on intricate sets of manually crafted rules. But things changed a lot in the late 1980s when machine learning algorithms designed for understanding language were introduced, revolutionising Natural Language Processing.

Today, the most advanced NLP programs leverage deep learning models, such as Recurrent Neural Networks (RNNs) and transformers. 

RNNs work as language processors that read a text one word at a time, maintaining a memory of the words it has encountered so far. This memory helps it understand the context of the language, making connections between words as it goes along. 

Transformers, on the other hand, process the entire piece of text at once. They don’t read word by word; instead, they analyse the entire context simultaneously. Transformers are effective at understanding both local and global relationships in language, allowing them to capture information from different parts of the text more efficiently than RNNs.

Pre-training and fine-tuning are integral steps in neural NLP. Models are first pre-trained on vast amounts of unlabeled data, learning the underlying structures and representations of language. Transfer learning is then employed by fine-tuning these pre-trained models on smaller, task-specific datasets, tailoring them for specific applications. This approach allows neural NLP models to benefit from the generalisation power gained during pre-training while adapting to specific language tasks.

Some of the core functionalities that fall under the broader category of NLP are:

Tokenisation

Tokenisation is the fundamental process of segmenting text into discrete units, such as sentences, words, or subword components. It serves as the foundation for virtually all NLP pipelines, enabling models to process and analyze textual data effectively. Advanced tokenization approaches, such as subword tokenization, are particularly useful for handling rare or out-of-vocabulary terms and improving model generalization.

Part-of-Speech Tagging (POS)

POS involves assigning each token its grammatical category, including nouns, verbs, adjectives, and other syntactic roles. This step allows models to interpret not only the semantic meaning of words but also their functional roles within a sentence. POS tagging underpins subsequent processes such as syntactic parsing, disambiguation, and information extraction.

Named Entity Recognition (NER)

NER identifies and classifies real-world entities within text, including persons, organizations, locations, dates, and numerical quantities. NER transforms unstructured text into structured information, facilitating tasks such as knowledge graph construction, question-answering systems, and advanced search applications.

Parsing

Parsing refers to the analysis of syntactic structure within a sentence. Constituency parsing reveals hierarchical phrase structures, whereas dependency parsing captures grammatical relationships between words, delineating roles such as subject, object, and modifier. Parsing is essential for accurate syntactic and semantic interpretation, enabling downstream applications that require a deep understanding of sentence structure.

Lemmatization and Stemming

These are processes designed to reduce words to their canonical or root forms. Stemming applies heuristic rules to truncate word endings, while lemmatization relies on morphological analysis and lexical resources to produce linguistically valid base forms. These operations enhance text normalization, reduce vocabulary size, and improve the performance of tasks such as information retrieval, clustering, and topic modeling.

Language Modeling

Language Modeling involves estimating the probability distribution of sequences of words and predicting the likelihood of subsequent tokens. Language models are fundamental to modern NLP, as they encode syntactic, semantic, and contextual knowledge, supporting a wide range of applications from text generation to comprehension. Contemporary large-scale language models are sophisticated probabilistic systems capable of capturing complex linguistic patterns.

Text Normalization

Text normalisation encompasses a series of preprocessing steps to standardize textual data. This includes lowercasing, punctuation handling, spelling correction, contraction expansion, and normalization of special characters. Normalization reduces variability in text input, ensuring that downstream models operate on a consistent and coherent representation of language.

Coreference Resolution

Coreference resolution identifies when multiple expressions in a text refer to the same entity. Accurately resolving coreferences, such as recognizing that “she” refers to “Maria” or “the organization” refers to “Google,” is critical for coherent text interpretation. This functionality underpins advanced NLP tasks such as summarization, multi-sentence reasoning, and narrative analysis.