This article data science blogthon.
Source: www.totaljobs.com
prologue
Until recently, it was common to develop new and improved transducers specifically for a single modality. However, there was an urgent need to develop a multimodal transformer model to tackle the real task.
A multimodal transformer model is a type of model that employs the process of learning representations from different modalities using a single model. Input modalities for machine learning include pictures, text, voice, etc. Multimodal learning models combinations of different modalities of data that commonly occur in real-world applications.
Given the types of real-world tasks often faced in corporations, start-ups, operating companies, and academia, and the demand, current trends, and potential for multimodal transformer models to tackle such tasks, thorough understanding of multimodal transformer models to secure their position in the industry.
In this article, we have compiled a list of five important questions about various multimodal transformer models. You can also use these questions as a guide to familiarize yourself with the topic and craft effective answers to help you succeed in your next interview.
Interview questions about multimodal transformers
Below are some of the questions with detailed answers.
Question 1: Describe the architecture of the CLIP model.
answer: CLIP (Contrasive Language Pre-training) models are neural networks that combine text and vision. Learn visual concepts with the help of natural language supervision.
A dataset of 400 million image/caption pairs was created and contrast learning was employed to pre-train the CLIP model.
Figure 1: Overview of the CLIP model approach
Source: Archive
CLIP = image encoder + text encoder
Image Encoder and Text Encoder —-> create image and caption embeds respectively.
Given a batch of N (image, caption) pairs, CLIP is trained to predict which (image, caption) pairs match. To achieve this, CLIP jointly trains an image encoder and a text encoder to maximize the cosine similarity of the correct pair of image and caption/text embeddings in a batch, while at the same time increasing the cosine similarity of the embeddings to Learn the multimodal embedding space by minimizing. Incorrect pairing (remaining pair). Symmetric cross-entropy loss is optimized with these similarity scores.
Possible classes are built into the text encoder and use pre-trained models for classification. Then all class embeddings are compared with the image embeddings we want to classify and the class with the highest similarity is selected.
CLIP’s zero-shot image classification performance is comparable to fully supervised pre-trained vision models and is more flexible for new classes.
NOTE: We also learn to perform various tasks during pre-training, such as OCR, geolocalization, and action recognition.
Question 2: Why is pre-training like CLIP useful? Why not use a regular classification model instead?
Unlike traditional image classification methods that train an image feature extractor and a linear classifier simultaneously to predict labels, CLIP trains an image encoder and a text encoder to produce a batch of (image, caption/text) training examples. Predict correct pairing. During testing, the trained text encoder synthesizes a zero-shot linear classifier by embedding the class descriptions of the target dataset.
We often use binary information for standard image classification, i.e. whether a class exists or not. As a result, a lot of information is lost. For example, if you train a classifier of cats and dogs using images crawled from the web, the classifier will not be able to recognize if the animal is ‘Aussie’ or its name is ‘Pepper’.
Question 3: Describe the LyoutLM architecture.
answer: When it comes to scanned documents such as invoices, receipts, reports, etc., they are rich in information, so visual and layout information can be extracted and encoded into pre-trained models to recognize text fields of interest. I can do it.
LayoutLM jointly models how text and layout information interact across scanned document images. This is useful for many real-world document image understanding tasks, including information extraction from scanned documents.
LayoutLM uses a modified Transformer architecture with Image embedding, LayoutLM embedding, Text embedding, and 2-D position embedding. 2-D positional embeddings capture relationships or relative positions between tokens in a document. Image embedding captures visual characteristics such as font orientation/style, type, and color.
Furthermore, embedding 2D positional features into the linguistic representation with the help of Transformer’s self-attention mechanism improves consistency with layout information.
Figure 2: LayoutLM architecture
Source: Archive
In addition, LayoutLM also employs multi-task learning objectives, Masked Visual-Language Model (MVLM) loss, and Multi-label Document Classification (MDC) loss, useful for joint pre-training of text and layout.
In particular, LayoutLM is pre-trained on the IIT-CDIP test collection 1.02. It contains over 6 million scanned documents with 11 million scanned document images. This is why LayoutLM can be forwarded to various downstream tasks.
Question 4: Describe the architecture of Wav2Vec 2.0.
answer: Wav2Vec 2.0 is a model for self-supervised learning of speech representations. Mask the audio input in the latent space. It solves contrasting tasks (where true latents are distinguished from distractions) across the quantization of jointly learned latent representations.
Figure 3: Wav2Vec2 architecture
Source: Archive
In terms of architecture, Wav2Vec 2.0 has a multi-layer convolutional function encoder (f) with layer normalization and GELU activation, which takes a raw audio input (X) and produces a potential speech representation (z). To do.1z2. . , zT.) T time steps, the total stride of the encoder determines the number of time steps T.
The latent phonetic representation is subsequently fed into a transformer (g) to form a contextual representation (c1c2. . , cT.) to capture data from the entire sequence.
In addition, the output of the feature encoder is also discretized (qt) represents the target of the self-supervised goal with the help of the quantization module.
We also achieve 4.8/8.2 WER by pre-training the model on 53,000 hours of unlabeled data and fine-tuning on just 10 minutes of labeled data. This shows that speech recognition works even with very little labeled data. This is very important when developing his ASR solution for native languages and dialects where data collection is difficult.
Question 5: What is the role of Gumbel-Softmax in Wav2Vec 2.0?
answer: Discrete sampling is used in many areas of deep learning. For example, in a language model, we have a sampled sequence of word/letter tokens, where each individual token corresponds to a word/letter. So you sample from a discrete space like this, which is different than sampling from a continuous space. Gumbel-Softmax not only helps sampling from a discrete space behave like a continuum, but also keeps the stochastic nature of the nodes intact while keeping the backpropagation step viable..
In Wav2Vec 2.0, The output of the feature encoder (z) is discretized into a finite set of phonetic representations using product quantization. Product quantization requires selecting quantized representations from different codebooks and concatenating them.
Given a set of G codebooks/groups with V entries, one entry is selected from each codebook and the resulting vector (e1…, eG.) and apply a linear transformation (R) to get q.
Gumbel softmax helps select discrete codebook entries in a fully differentiable manner. The feature encoder output (z) is mapped to l logits,
The vth codebook entry for group g is
(Source: Archive
For the forward pass, codeword i is chosen using i = argmaxj pg, j.
During the backward pass: the true gradient of the Gumbel softmax output is used.
Additionally, to facilitate training and codeword utilization, a small random effect is added, the effect of which is controlled with the help of the temperature argument.
Conclusion
In this article, I’ll share five of the most important interview questions about multimodal transformers that you might be asked in a data science interview. You can use these interview questions to work on your understanding of various concepts and create effective answers to present to your interviewer.
In summary, the main points of this article are:
1. CLIP (Contrastive Language Pre-training) model that combines text and visuals. Learn visual concepts with the help of natural language supervision.
2. CLIP consists of an image encoder and a text encoder. Image Encoder and Text Encoder create image and caption embeds respectively.
3. Traditional classification approaches train an image feature extractor and a linear classifier simultaneously to predict labels. CLIP trains an image encoder and a text encoder to predict the correct combination of batches of (image, caption/text) training samples.
4. LayoutLM jointly models how text and layout information interact across scanned document images. This is useful for many real-world document image understanding tasks, including information extraction from scanned documents.
5. Wav2Vec 2.0 is a model for self-supervised learning of speech representations. It is equipped with a multi-layer convolutional feature encoder (f) with layer normalization and GELU activation, which takes a raw audio input (X) and generates potential speech representations (z1, z2. .., zT).
6. Gumbel softmax helps select discrete codebook entries in a fully differentiable manner.
Media shown in this article are not owned by Analytics Vidhya and are used at the author’s discretion.
