Perplexity …

Original Source Here

… in the context of Natural Language Processing.

Perplexity formula

What is perplexity?

Perplexity is an accuracy measurement of a probability model.

A language model is a kind of probability model that measures how likely is a given sentence according to a large corpus of text or the training set (The Wall Street Journal dataset, comments on Youtube in a given language, Brown corpus …).

A unigram model (order 1) is an example of language model which gives the probability of a sentence multiplying the probability of each word in the sentence based on their frequency in the training set.

A bigram model (order 2) is an other example of language model which gives the probability of a sentence multiplying the probability of each word in the sentence taking into account the previous word (except for the first word) based on the frequency of those pairs (or the first word) in the training set.

This can be generalised to order N cases.

Having this in mind, the perplexity of such model will be the inverse of geometric average of each word probability (or pair, or triplet …). Refer to the main image to see the actual formula.

This formula can be intuitively derived from the notion of cross-entropy in this great article [1].

How do we use it in Natural Language Processing model?

An example of application would be to determine how relevent (correct syntax, grammar, insightful …) is a comment among the list of all the comment on the IMDb website (film rating website).

Another application could be found in the translation field. A chinese sentence could be translated in different ways and calculating the perplexity of each possiblity would give a way to chose one translation or the other.

Speach recognition is another interesting application of the usefulness of such measurement. Sounds could be interpreted in different ways (homophones, noise, polysemy …) but our brain or the language model could tell which version is the most probable or would make us feel less perplexe.

Relevent reference

[1] Vajapeyam, S. Understanding Shannon’s Entropy metric for Information (2014).

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot



via WordPress https://ramseyelbasheer.wordpress.com/2021/02/01/perplexity/

Popular posts from this blog

I’m Sorry! Evernote Has A New ‘Home’ Now

Image
Original Source Here My Apologies! I was critical about the grandfather of all apps — Evernote a few months back when I wrote Why is Evernote losing its Ground? on Medium. With that said, I also wrote another story on Evernote in November of last year — Has Evernote Made A Comeback After Years Of Stagnation? This was the time when Ian Small and his team were making inroads into the technology and strategically making enhancements and revamping the app across all platforms. Evernote Is At A War-front And Wants To Prove A Point. I must h ave written this in my earlier blogs but just to re-iterate — I have been and am an avid user of Evernote since 2009 and is a part of my life for planning my daily life, projects, content writing, and almost everything. Recently, Evernote has launched its HOME or you may call it a DASHBOARD that provides its users a one-stop place to get access to their important and critical notes and also provides a few AI-enabled capabilities that I wil...
Read more

Jensen Huang: Racism is one flywheel we must stop

Image
https://venturebeat.com/wp-content/uploads/2021/07/jensen-huang.jpg?w=1200&strip=all Original Source Here All the sessions from Transform 2021 are available on-demand now. Watch now. Jensen Huang , CEO of Nvidia , received the distinguished lifetime achievement award by the Asian American Engineer of the Year from the Chinese Institute of Engineers (CIE) group. And in his acceptance speech, he made a rare comment beyond Nvidia’s business matters: “Racism is one flywheel we must stop.” The comments came at the end of an acceptance speech where he thanked the group for its award, an annual gathering that recognizes outstanding Asian American scientists, engineers and role models. In a virtual ceremony, Huang was awarded for his contributions as “a visionary and innovator in parallel computing technology that accelerates the realization of AI computing.” He also spoke of his experience as an immigrant and an Asian American. “It is strange to accept a lifetime achievem...
Read more

Streamlit — Deploy your app in just a few minutes

Image
Original Source Here Streamlit is an open-source Python library that makes it easy to create and share beautiful, custom web apps for machine learning and data… Continue reading on Medium » AI/ML Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot via WordPress https://ramseyelbasheer.io/2021/07/11/streamlit%e2%80%8a-%e2%80%8adeploy-your-app-in-just-a-few-minutes/
Read more