- Unveiling the Realm of AI Technologies: A Glimpse into the Augmented Future
- Generative AI and Language Models
- LLMs and Their Applications
- LLMs and Customer Support
- Development, Optimization, Localization, and Personalization Based on LLMs
- Unveiling the Power of Clustering and Topic Modeling
- Enhancing Customer Support Through Hybrid AI: LLMs Meet Clustering and Topic Modeling
- Endnotes
This chapter is from the book
This chapter is from the book
This chapter is from the book
Generative AI and Language Models
One of the most exciting and challenging areas of generative technologies is natural language generation (NLG), which generates natural language text from a given input, such as an image, a keyword, or a prompt. NLG has many applications, including summarization, translation, dialogue, storytelling, and content creation. However, NLG also poses many technical and ethical challenges, such as ensuring the generated texts’ quality, diversity, coherence, and fairness.
A key component of NLG is the language model (LM), a probabilistic model that assigns a probability to a sequence of words or tokens. LMs can generate new texts by sampling tokens according to their probabilities or evaluating the likelihood of existing texts. LMs can be trained on large corpora of text data, such as Wikipedia, books, news articles, or social media posts, using deep learning techniques, such as RNNs or transformers.
Because statistical language models (SLMs) cannot capture long-term dependencies or semantic relations in natural language, this underscores the importance of building these models with responsibility and ethics in mind, as discussed throughout this book.
The development of LMs has gone through several stages, reflecting the advances in computational power, data availability, and algorithmic innovation. There are four main development stages of LMs: statistical language models, neural language models, pre-trained language models, and large language models (LLMs).4
Statistical language models are based on statistical learning models. The idea is to build models based on the n-gram assumption, which states that the probability of a word only depends on the previous n-1 words and not on the rest of the sentence or the document. An n-gram is a sequence of n-words, such as “the cat” or “a big house.” For instance, predicting when the probability of the word “model” would come after the words “large language” is illustrated as P(model|large language). Statistical language models estimate the probabilities of n-grams from a corpus of text data using techniques such as counting, smoothing, or interpolation.
Counting: Counting is simply the frequency of the n-gram in the corpus divided by the total number of n-grams.
Smoothing: Smoothing adds some small values to the counts to avoid zero probabilities for unseen n-grams.
Interpolation: Interpolation combines the probabilities of different n-grams, such as unigrams, bigrams, and trigrams, to balance the trade-off between specificity and generality.
Smoothing and interpolation are often adopted to mitigate the data sparsity problem. Statistical language models are simple and efficient but have limited expressive power and cannot capture long-term dependencies or semantic relations in natural language. For example, statistical language models cannot distinguish between the meanings of “bank” in “I went to the bank” and “The bank was closed” or the contexts of “She saw a bear” and “She saw a bare.” Statistical language models cannot handle the ambiguity of words with multiple meanings, such as “bat,” “right,” or the influence of words that are far apart in the sequence, such as “The man who wore a hat” and “The hat was red.”
Neural language models (NLMs) use neural networks, such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), to learn distributed representations of words and sequences and to model the conditional probability of the next word given the previous words. A groundbreaking work in the field of neural language modeling was the paper “A neural probabilistic language model,”5 which presented the idea of representing words as continuous vectors in a high-dimensional space and learning the probability of the next word based on the sum of the context word vectors. Many studies have opened a new chapter for using language models for representation learning, playing an influential role in the NLP field. For example, word2vec6 utilizes two methods to learn word embedding:
Using context to predict a target word through CBOW (Continuous Bags of Words): CBOW is like a guessing game where the AI tries to predict a word based on the words around it. Imagine a sentence with a missing word; CBOW looks around the neighboring words to guess the missing one, helping the AI better understand the language.
Using a word to predict a target context through Skip-Gram: Skip-Gram is like a word puzzle where the challenge is to find the related words. Given a specific word, Skip-Gram tries to predict the surrounding words, helping the AI grasp the context and relationships between words in a sentence.7
NLMs can overcome some of the limitations of statistical language models, such as capturing longer contexts and learning richer features, but they also have drawbacks, such as requiring more computation and data and suffering from the vanishing or exploding gradient problem.
Pre-trained language models (PLMs) are a type of model that uses transfer learning. PLMs are first trained on a large collection of text data that doesn't have specific labels (this is the pre-training phase). After this, PLMs are fine-tuned on a particular task or domain with more specific data (this is the fine-tuning phase). This process allows PLMs to be highly effective at understanding and generating human language in various applications. As mentioned earlier, RNNs have some drawbacks and limitations, such as the difficulty of learning long-term dependencies, the gradient vanishing or exploding problem, and the sequential nature of computation, which prevents parallelization and reduces efficiency. To address these issues, long short-term memory (LSTM) models were proposed by researchers as one of the most popular and effective variants of RNNs. LSTM models have a special structure consisting of three gates and a cell state, which can regulate the input, output, and forget operations of the recurrent unit. LSTM models can generate natural language texts by updating the cell state and the hidden state at each time step, based on the current input and the previous states, and then producing the next word or token from the hidden state. As one of the first models that demonstrated the effectiveness of PLMs on large-scale unlabeled text data and then transferring the learned knowledge to downstream tasks or domains, embeddings from language models (ELMo) was built as an LSTM-based model that can generate contextualized word embeddings, which are vector representations of words that capture their meanings and usage in different contexts. Unlike traditional word embeddings, such as word2vec or GloVe, which assigns a fixed vector to each word regardless of its context, ELMo can dynamically compute word embeddings based on the entire input sentence or document, using a bidirectional LSTM that encodes both the left and the right contexts of each word. PLMs also adopt the transformer architecture, which is an attention-based neural network that can learn long-range dependencies and parallelize computation. Some of the most influential PLMs are BERT, developed by Google, and GPT, developed by OpenAI. Based on pre-trained context-aware word representations, these models have shown remarkable effectiveness and versatility as general-purpose semantic features that can significantly improve the performance and efficiency of various NLP tasks.
Large language models (LLMs) are the latest and most advanced stage of LMs, which aim to build very large-scale and powerful LMs that can generate natural language texts across multiple domains and tasks, given minimal or no supervision. LLMs rely on massive amounts of computation and data and use sophisticated optimization and regularization techniques, such as self-attention, dropout, or layer normalization, to train billions or trillions of parameters. Some of the most prominent examples of LLMs are GPT-3, GPT-3.5 (Instruct GPT) GPT-4 and GPT-4o, developed by OpenAI.
GPT-3: GPT-3 is a transformer-based model with 175 billion parameters and can generate coherent and diverse texts on various topics and domains, given a few words or sentences as input.
GPT-3.5: In 2022, OpenAI deployed GPT-3.5, which performs more significantly in following instructions, making up facts less often, and generating less toxic output. They used prompts submitted by the customers through Playground and hired human annotators to provide demonstrations of the desired model behavior and rank outputs from the models. GPT-3.5 is fine-tuned based on this data from GPT-3.
GPT-4: In 2023, GPT-4, a 1.8T-parameter model with 16 Mixture of Experts (MoE), was announced by OpenAI to improve the security of the model and enable multimodal capability. However, LLMs also have limitations and risks, such as producing inaccurate, biased, or harmful content or violating the data sources’ privacy or intellectual property rights.
GPT-4o: Launched in 2024, GPT-4o (“o” for “omni”) is a step towards a much more natural human-computer interaction—it accepts any combination of text, audio, image, and video as input and generates any combination of text, audio, and image as output. It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in a conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models.8
The emergence and advancement of LLMs significantly impact the AI community and society at large, as they open up new possibilities and challenges for natural language understanding and generation. LLMs can be seen as a form of generative technologies that can create novel and valuable outputs from minimal or no inputs, such as images, music, art, or texts. They can foster interdisciplinary collaboration and innovation by bringing together researchers and practitioners from different fields and domains and creating new paradigms and methods for natural language understanding and generation.
Despite the exciting progress and impact of LLMs and generative AI, many mysterious and unpredictable perspectives remain. There are some risks associated with LLMs. They can amplify existing biases and harms, such as perpetuating stereotypes, discrimination, misinformation, or manipulation, by learning from unfiltered and unrepresentative data sources, or by being misused or abused by malicious actors. They can also pose ethical and legal dilemmas, such as violating privacy, intellectual property, or human dignity, by exposing sensitive or personal information, infringing on copyrights or trademarks, or generating deceptive or harmful content. Moreover, they can challenge existing norms and values, such as accountability, transparency, or trust, by obscuring natural language generation’s sources, processes, and outcomes or by creating conflicts of interest, responsibility, or authority.