This article gives details on fast face search using vector DB.
Github Link of the project: https://github.com/ranjankumar-gh/face-search/
Libs:
pip install face-recognitionpip install faisspip install picklepip install streamlitScreenshot of the application:
I have created this teaching chatbot using RAG, Llama Model, Qdrant (Vector DB), Langchain, and Streamlit.
Github repository link: https://github.com/ranjankumar-gh/teaching-bot/
Screenshot of the app:
Data: PDF files have been downloaded from the NCERT website for Class IX, subject SST, from the topic “Democratic politics”.
For more details, visit the GitHub link.
An ability is emergent if it is not present in smaller models but is present in larger models. [1]
Scaling up language models has been shown to improve predictably the performance and sample efficiency on a wide range of downstream tasks. Emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. This raises the question of whether additional scaling could potentially further expand the range of capabilities of language models. [1]
Today’s language models have been scaled primarily along three factors:
The following table lists the emergent abilities of large language models and the scale at which abilities emerge. [1]
Tasks that language models cannot currently do are prime candidates for future emergence; for instance, there are dozens of tasks in BIG-Bench[3] for which even the largest GPT-3 and PaLM models do not achieve above-random performance. [1] Similar to emergent abilities, emergent risks could also emerge, such as w.r.t. truthfulness, bias, and toxicity in LLMs, backdoor vulnerabilities, inadvertent deception, or harmful content synthesis.
But Rylan Schaeffer et al., in their paper [3], claim that the sudden appearance of emergent abilities is just a consequence of the way researchers measure the LLM’s performance. The article “How Quickly Do Large Language Models Learn Unexpected Skills?” by Stephen Ornes [4] beautifully summarises the two papers.
Prompt is a text set passed to the GenAI model as instructions. Given the prompt, the model responds with some generated text. A prompt can be questions, statements, or instructions. Prompt engineering facilitates the design of the prompt to enhance the generated text. It is also 1) a tool to evaluate the output of the model and 2) a tool for safety mitigation methods. There is no perfect prompt design. Prompt optimization and experimentation are done iteratively. The following figure 1 depicts a very basic example of the prompt.
Figure 1: A basic example of the prompt
temperature and top_p parameters control the randomness of the output. Before a large language model (LLM) generates a token, it has many possible choices of tokens with different likelihoods assigned. Some tokens are most likely whereas some are least likely. For these two parameters to work, do_sample parameter should be set to True i.e. do_sample=True. It means that we are allowing the next token to be sampled from the set of all likely tokens.
temperature defines how likely the model will choose the least likely token. temperature=0 means the model will generate the same response every time, because it will always choose the most likely token every time. With the higher value of temperature, the model will give more chance to other less likely tokens also. With temperature=1, chances of any probable token for being selected will be mostly equally likely. For example, a value of 0.8 will produce the more diverse output, whereas the value of 0.2 will produce deterministic output. So we can say that temperature induces stochastic behaviour.
top_p also known as nucleus sampling allows the model to only select a subset of all likely tokens. Based on its value, the model will stop sampling as soon as the cumulative probability reaches the value of top_p. Value of 1 means, it will consider all tokens.
top_k parameter tells the model to select the number of tokens exactly equal to its value.
Based on the requirements of the use case, we choose to set these parameters. We need to find the right balance of randomness/diverse vs. deterministic/focused/coherent outputs.
Providing a large language model (AI) with clear, specific, and structured instructions to guide the response of the model is referred to as instruction-based prompting. The most basic prompt consists of two components:
The following diagram depicts a basic instruction prompt. Please note the instruction and data part of the prompt.
Figure 2: Instruction Prompt
To make the model very specific about the output, for example, if we want the output to be either “positive” or “negative”, we can use output indicators. The following diagram depicts the instruction prompt with output indicator.
Figure 3: Instruction prompt with output indicators
Different types of tasks require different formats of the prompt. The following diagram illustrates example formats for summarization, classification, and named-entity recognition.
Figure 4: Prompt format for summarization, classification and NER task
Following is the non-exclusive list of prompting techniques for improving the quality of the output.
As we saw above, common components of prompts are instruction, data, and output indicators. However, prompts are not limited to these components; we can build up a prompt that is as complex as we want. Other common components are
The following is an example from book 1 that uses the above prompt components. This example demonstrates the modular nature of prompting. We can experiment by adding or removing components to see the effect.
Figure 5: Example of prompt showing use of the various components.
Giving examples to the LLM, greatly influences the output of the prompt. This is referred to as in-context learning. Zero-shot prompting uses no example, whereas one-shot prompting uses one example, and few-shot prompting uses two or more examples. The following diagram illustrates the examples of in-context learning.
Figure 6: Examples of in-context learning
While giving the examples, the user and the assistant should be differentiated clearly by mentioning the role as user or the role as assistant. By giving examples, we can be more clear in describing the model. But the model can still choose through random sampling and ignore the instruction.
We already know that we can break the prompt into modular components of the prompt to enhance the output of LLMs. Next level of strategy is to break the problem/task into subproblems/subtasks. We use separate prompts for subtasks and then chain the prompts in a sequence by passing the output of one prompt to the input of the other prompt, thus creating a continuous chain of interactions to solve our problem. This is called chain of prompt operations or prompt chaining. The prompt chaining can help in
Use cases include
Following is the example from the reference book1. This example illustrates the prompt chain that first creates a product name, then uses this name with product features to create a slogan, and finally uses features, product name, and slogan for creating the sales pitch.
Figure 7: Example of prompt chain
Reasoning is an important trait of human intelligence. LLMs as of today resemble this reasoning behaviour by memorization of training data and pattern matching. We need to work with LLMs by leveraging prompt engineering so that they can mimic the reasoning processes, and the output could be enhanced.
Daniel Kahneman in his famous book “Thinking Fast and Slow” introduced the concept of System 1 and System 2 thinking process in humans. According to him, “System 1” represents our fast, automatic, and intuitive thinking mode, while “System 2” is our slower, more deliberate and conscious mode of thinking, which requires effort and attention; essentially, System 1 is “thinking fast” and System 2 is “thinking slow.”.
Figure 8: System 1 and System 2 thinking, Image Source
The majority of LLMs today rely on System 1 thinking, but researchers are working on techniques to encourage more System 2-type behaviour by using prompting methods like “Chain of Thought” to elicit intermediate levels of reasoning before arriving at a final response.
The main aim of chain-of-thought is to push the model towards system 2 thinking i.e. think before answering, and allowing the model to distribute more compute for the reasoning process. Here, reasoning is referred to as thoughts.
Chain of thought – a series of intermediate reasoning steps – significantly improves the ability of large language models to perform complex reasoning [3]. Prompting using chain-of-thought is called chain-of-thought prompting. This prompting technique enables LLMs to tackle complex arithmetic, commonsense, and symbolic reasoning tasks. Chain-of-thought reasoning process is highlighted in the following example taken from the paper[3].
Figure 9: Chain-of-thought example; reasoning process is highlighted – source [3]
Chain-of-thought also requires one or more examples of reasoning. But there is something called zero-shot chain-of-thought that can be achieved by simply using the phrase “Let’s think step-by-step“. Though this phrase does not need to be exactly the same. Small variation should be fine. The following is an example of zero-shot chain-of-thought.
Figure 10: Example of zero-shot chain-of-thought – source[1]
The paper[2] writes about the self-consistency as follows: “It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer.“
We first prompt the language model with chain-of-thought prompting, then instead of greedily decoding the optimal reasoning path, ‘sample-and-marginalize’ decoding procedure is followed. Sample-and-marginalize decoding procedure:
The following diagram, from the paper[4], illustrates the concept.
Figure 11: Example of self-consistency in CoT[4]
This is another effort in the direction of pushing the model towards system 2 level of thinking in humans. Following is a quote from Newell et al.[6]
” A genuine problem-solving process involves the repeated use of available information to initiate exploration, which discloses, in turn, more information until a way to attain the solution in finally discovered.”
Paper[5] explains the Tree-of-Thought (ToT) as follows. ToT generalizes over the popular “Chain of Thought” approach to prompting language models, and enables exploration over a coherent unit of text (“thoughts”) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.
The following diagram from the paper[5] illustrates the various approaches to problem solving with LLMs. Each rectangle box represents a thought.
Figure 12: Various approaches to problem solving with LLMs.
It is important to verify and control the output of the model to avoid breakdown of the model in production and to create a robust AI system. Reasons for validating the output may include
Except from controlling the parameters temperature and top_p, the following are the three ways to control the output of the GenAI model:
To control the structure of the output, e.g., in JSON format, we can provide a few examples in this format for guiding the model to produce the output in the desired format. Still, the model will behave in a certain way, is not guaranteed. Some models may be better than others in following instructions.
Libraries have been developed to constrain and validate the output of generative models such as:
There is another way where we can define grammars or rules that LLM should follow when choosing the next token. For example, in llama-cpp-python we can specify response_format as JSON object, if we want the output in the JSON format.
Text clustering is an unsupervised approach that helps in discovering patterns in data. Grouping similar texts according to their semantic content, meaning, relationships, etc. is the goal of text clustering. This makes it easier to cluster vast amounts of unstructured text and perform exploratory data analysis quickly. With recent advancements in large language models (LLMs), we can obtain extremely precise contextual and semantic representations of text. This has improved text clustering’s efficacy even more. Use cases for text clustering include identifying outliers, accelerating labelling, identifying data that has been erroneously labelled, and more.
Topic modelling facilitates the identification of (abstract) topics that arise in huge textual data collections. Clusters of text documents can be given meaning through this method.
We will learn how to use embedding models for text clustering and text-clustering-inspired method of topic modeling, namely BERTopic, generating labels using LLM given the keywords of the topic.
The following diagram depicts the pipeline for text clustering that consists of the following three steps:
We know that embeddings are the numerical representations of text that attempt to capture its meaning. We can use embedding models that are optimized for semantic similarity tasks, for transforming the documents to embeddings in the first step. We can use the Massive Text Embedding Benchmark (MTEB) leaderboard, for selecting the embedding model for our requirement. For example “thenlper/gte-small” is a small but performant model with fast inference.
It is difficult for clustering techniques to identify meaningful clusters if the dimension of the data is high. These techniques preserve the global structure of the data by finding the low-dimension representations. This techniques act as compression techniques, so these do not remove dimensions arbitrarily. Popular algorithms are Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP). Compared to PCA, UMAP is better in handling nonlinear relationships and structures.
The following diagram depicts the methods of text clustering:
Density-based clustering algorithms calculate the number of dimensions itself and do not force all data points to be part of the cluster. The data points not part of any cluster are marked as outliers. Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) is a hierarchical variation of the DBSCAN algorithm that specializes in finding dense(micro)-clusters without being told the number of clusters in advance.
Topic modelling is the term used to describe the process of identifying themes or hidden topics in a set of textual data. Traditionally, it involves finding a group of keywords or phrases that best represent and capture the essence of the topic. We need to understand the meaning of the topic through these keywords or phrases. Latent Dirichlet Allocation (LDA) is one such algorithm. Let’s discuss BERTopic in the following sections, which is a highly modular text clustering and topic modeling framework.
Steps for performing topic modeling follow three steps of text clustering. The output of the third step of text clustering is fed to the fourth step of topic modeling. The following diagram depicts the steps (4th and 5th) for topic modeling:
4th step calculates the class-based term frequency, i.e., frequency (tf) of word (X) in cluster (C). This term is then multiplied with IDF (inverse document frequency) in the 5th step. The goal is to give more weight to the words in a cluster and less weight to the words appearing across all clusters.
The following diagram depicts the full pipeline from clustering to topic modeling. Though topic modeling follows clustering, they are largely independent of each other, and each component is modular. BERTopic can be customised, and we can choose another algorithm instead of the default ones.
c-TF-IDF does not take into account the semantic structures, so BERTopic leverages the representation models (e.g. KeyBERTInspired, Maximal marginal relevance – MMR, spaCy) to rerank the topics found out in the previously discussed 5th step. This reranking is applied on each topic instead of every document. Many of the representation models are LLMs. Now with this step final pipeline extends to become the following:
The following diagram explains how keywords combined with documents, along with a prompt, can be passed on to LLM for generating the label for the topic given keywords.
Final pipeline is as follows:
Final detailed pipeline:
A common task in natural language processing (NLP) is text classification. Use cases of text classification include sentiment analysis, intent detection, entity extraction, and language detection. This article will delve into how to use LLMs for text classification. We will see representation models and generative models. Under the representation model, we will see how to use task-specific models and embedding models to classify the text. Under the generative models, we will see open source and closed source models. While both generative and representation models can be applied to classification, they take different approaches.
Task-specific models and embedding models are two types of representation models that can be used for text classification. To obtain task-specific models, representation models, like bidirectional encoder representations from transformers (BERT), are trained for a particular task, like sentiment analysis. On the other hand, general-purpose models such as embedding models can be applied to a range of tasks outside classification, such as semantic search.
As it can be seen in the below diagram, when used in the text classification use case, representation models are kept frozen (untrainable). As the task-specific models are specially trained for the given task, when the text is given as input, it can classify the given text as per the task at hand. But when we are using the embedding model, we need to generate embeddings for the texts in the training set. Then train a classifier on the train dataset that has embeddings and corresponding labels. Once the classifier is trained, it can be used for classification.
The factors we should look into for selecting the model for text classification:
BERT is an encoder-only architecture and is a popular choice for creating task-specific models and embedding models, and falls into the category of representation models. Generative Pre-trained Transformer (GPT) is a decoder-only architecture that falls into the generative models category. Encoder-only models are normally small in size. Variations of BERT are RoBERTa, DistilBERT, DeBERTa, etc. For task-specific use cases such as sentiment analysis, Twitter-RoBERTa-base can be a good starting point. For embedding models sentence-transformers/all-mpnet-base-v2 can be a good starting point as this is a small but performant model.
This is pretty straight forward. Text is passed to the tokenizer that splits the text into tokens. These tokens are consumed by the task-specific model for predicting the class of the text.
This is fine if we could find the task-specific models for our use case. Otherwise, if we have to fine-tune the model ourselves, we would need to check if we have sufficient budget(time, cost) for it. Another option is to resort to using the general-purpose embedding models.
We can generate features using an embedding model rather than directly using the task-specific representation model for classification. These features can be used for training a classifier such as logistic regression.
We have the definition of the labels, but we do not have the labeled data, we can utilize what is called “zero-shot classification“. Zero-shot model predicts the labels of input text even if it was not trained on them. Following diagram depicts the concept.
We can use zero-shot classification using embeddings. We can describe our labels based on what they should represent. The following diagram describes the concept.
To assign the labels to the input text/document, we can calculate the cosine similarity with the label embeddings to check which label it is close to.
Generative models are trained for a wide variety of tasks, so it will not work with the text classification out of the box. To make the generative model understand our context, we need to use the concept of prompt engineering. The prompt needs to be cleverly written so that the model understands what it is expected to do, what the candidate labels, etc.
The following diagram summarizes the different categories of the models:
The following diagram depicts the training steps:
We need to prefix each document with the prompt “Is the following sentence positive or negative?“
The following diagram describes the training procedure of ChatGPT:
The model is trained using human preference data to generate text that resembles human preference.
For text classification, following is the sample prompt:
prompt = """Predict whether the following document is a positive or negative movie review:
[DOCUMENT]
If if is positive return 1 and if it is negative return 0. Do not give any other answers.
"""
This article talks about various available solutions, techniques, and underlying architectures for LLM inference and serving. LLM inference and serving is nothing but deploying LLM models and getting/providing access to them. Based on your requirement, whether you want a local deployment for your own use or you want a production-grade deployment, various solutions are available.
The following is an overview of the architecture for LLMs’ inference and serving.
Figure: Typical architecture of LLM Inference and Serving
Before we go into detail, let’s first understand the difference between an inference engine and a server. Inference engines run the models and are in charge of everything needed for the generation process. Inference servers handle incoming and outgoing HTTP and gRPC queries from end users of the application, as well as metrics for measuring the deployment performance of the LLM.
The inference server primarily consists of the query queue scheduler and the inference engine.
Query Queue Scheduler: This component consists of a queue and a scheduler. When a query first arrives, they are added to this queue. The scheduler takes a query from the queue and hands it over to the inference engine. The use of the queue helps the scheduler in picking multiple requests and putting them in the same batch to be processed on the GPU.
Inference Engine: This component consists of batching, model, and query response modules.
Batching: This module is responsible for creating batches of query requests. The batches are created because when calculations on the GPU are done in batches, they are more performant and resource-efficient.
The model represents the LLM model that does the actual inference, e.g., next token prediction. The query response is the final response that is sent back to the original requester. Additionally, the inference server provides the interface for access via HTTP, gRPC, etc.
The metrics module keeps track of metrics such as throughput, latency, etc.
Throughput and latency are two important metrics for evaluating LLM inference and serving. Throughput refers to the number of output tokens an LLM can generate per second. Latency refers to the time it takes for a large language model (LLM) to process an input and generate a response. For latency, the important metric is “Time to First Token” (TTFT), which refers to the amount of time it takes for a language model to generate the first token of its response after receiving a prompt.
To achieve high throughput and low latency, LLM inference engines and servers focus on optimising LLM memory utilisation and performance in the production environment. Though throughput and latency are important factors, based on your specific use case, you may choose additional factors that would influence your decision to select a particular engine/server.
Summary of fact-checking in LLM #llm #hallucinations #misinformation
Following are the questions that should be running in your mind when you are to choose from competing open-source products:
The popularity and traction of GitHub can be inferred from their star histories. You can use star-history.com to make a comparison based on these two metrics. Refer to the tutorial for details.
Following is the comparison of vector databases: qdrant, chroma, weaviate, marqo, milvus, and vespa.
