Word Embeddings and Pre-training for Large Language Models (BERT, GPT)



Original Source Here

Around 2017

Attention and beyond!

Attention in Language models(LM) by Vaswani et al. (2017), introduced a way of capturing this context in language that outperformed some previous SOTA benchmarks in various downstream NLP tasks. Since then the language models have grown significantly in size. For context (all are transformer-based by the way):

BERT-base(Transformer Encoder) has ~110M parameters
GPT-1 (Transformer Decoder) has ~117M parameters
BERT-large has ~340M parameters
GPT-2 has ~1.5B parameters
GPT-3 has ~175B parameters

The pre-training objective of some of these large pre-trained language models is to predict the next word or next sentence. This turns out to be a good pre-training objective to understand complex word interactions and is useful for different downstream tasks like question answering.

The datasets these models are trained on are non-trivial; for example BERT was trained on Wikipedia (2.5B words) + BookCorpus (800M words), in total 3.2B words.

And why do we care about their size? because these models can capture complex word interactions between words in their context. More than what pre-trained word2vec has to offer, and so these large models learn a word representation independant of word2vec or any other pre-trained word embedding, i.e. pre-training whole models, illustrated below.

How deep learning on text is handled after transformers

Note: You could also throw away Word2Vec embeddings before 2017 and just have your model (say LSTM) learn words embeddings from scratch end-end, but this was not useful as we did not have such large models trained on huge collection of datasets and limiting factors from previous mode architectures like LSTM (one of them is that the context is lost beyond ~100 words) and a problem with sequential compute with them.

But this works well for large transformer based models on learning “strong” representations of language. One might ask if pre-training these large pre-trained models overfit on the pre-training data — it turns out from the results, BERT and these large models are in fact underfitting the pre-training data and so we need even larger models to express complex interactions that allow us to fit these massive datasets better. Therefore more and more large models are showing up like T5 from Google(text-to-text transfer transformer).

Text that these models were trained on were of all sorts data, a hypothetical example: “This pizza is taaaaasty!” (OR) “We should transformify this and laern the word representations better”.

there can be variations of words like “taaaaasty”, misspellings like “laern” and novel items like “Transformerify” which would get an UNK token from tokenizers. Trying to learn meaning of each of these words wouldn’t help, though English has slightly less complex morphology, other languages have more complex morphology. Meaning that, there are massive number of variations of words in other languages compared to English.

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot



via WordPress https://ramseyelbasheer.io/2021/07/31/word-embeddings-and-pre-training-for-large-language-models-bert-gpt/

Popular posts from this blog

I’m Sorry! Evernote Has A New ‘Home’ Now

Image
Original Source Here My Apologies! I was critical about the grandfather of all apps — Evernote a few months back when I wrote Why is Evernote losing its Ground? on Medium. With that said, I also wrote another story on Evernote in November of last year — Has Evernote Made A Comeback After Years Of Stagnation? This was the time when Ian Small and his team were making inroads into the technology and strategically making enhancements and revamping the app across all platforms. Evernote Is At A War-front And Wants To Prove A Point. I must h ave written this in my earlier blogs but just to re-iterate — I have been and am an avid user of Evernote since 2009 and is a part of my life for planning my daily life, projects, content writing, and almost everything. Recently, Evernote has launched its HOME or you may call it a DASHBOARD that provides its users a one-stop place to get access to their important and critical notes and also provides a few AI-enabled capabilities that I wil
Read more

Jensen Huang: Racism is one flywheel we must stop

Image
https://venturebeat.com/wp-content/uploads/2021/07/jensen-huang.jpg?w=1200&strip=all Original Source Here All the sessions from Transform 2021 are available on-demand now. Watch now. Jensen Huang , CEO of Nvidia , received the distinguished lifetime achievement award by the Asian American Engineer of the Year from the Chinese Institute of Engineers (CIE) group. And in his acceptance speech, he made a rare comment beyond Nvidia’s business matters: “Racism is one flywheel we must stop.” The comments came at the end of an acceptance speech where he thanked the group for its award, an annual gathering that recognizes outstanding Asian American scientists, engineers and role models. In a virtual ceremony, Huang was awarded for his contributions as “a visionary and innovator in parallel computing technology that accelerates the realization of AI computing.” He also spoke of his experience as an immigrant and an Asian American. “It is strange to accept a lifetime achievement
Read more

Fully Explained DBScan Clustering Algorithm with Python

Image
Original Source Here Fully Explained DBScan Clustering Algorithm with Python Unsupervised learning in machine learning on density-based clusters Clustering In unsupervised Learning. A photo by Author In this article, we will discuss the machine learning clustering-based algorithm that is the DBScan cluster. The approach in this cluster algorithm is density-based than another distance-based approach. The other cluster which is distance-based looks for closeness in data points but also misclassifies if the point belongs to another class. So, density-based clustering is suited in this kind of scenario. The cluster algorithms come in unsupervised learning in which we don’t rely on target variables to make clusters. In a cluster, the main concern is on maximum dense connected points. The density-based clustering can be useful in arbitrary shapes even with noise. The working function of the density-based cluster is shown below: We select a random data point. Then it will
Read more