Natural Language Processing: tokenization and numericalization



Original Source Here

Natural Language Processing: tokenization and numericalization

NLP processing techniques

NLP nowadays is considered to be one of the most booming fields in Deep Learning, offering more and more possible applications, starting from detecting or generating articles and reviews heading into direction of medical applications like diagnosis recognition, not mentioning extensive business opportunities.

How does it work?

The basic initial step is converting texts using Tokenization method, which ‘breaks’ raw text in smaller pieces. Tokens can be words, singles characters or subwords (n-gram characters: a contiguous sequence of n items from a given sample of text or speech). The most common way of tokenization is based on space. Taking space as a delimiter for example, you will get from “Natural Language Processing” 3 tokens : “Natural”, “Language”, “Processing”.

Major techniques for tokenizing are:

  • Split(): split method is used to break the given string in a sentence and return a list of strings by the specified separator.
  • Spacy: Texts are split using rules that are specific for each language, so how the splitting is done for English would be different then for in German. These Prebuilt neural network models are available for English, German, Greek, Spanish, Portuguese, French, Italian, Dutch, Lithuanian and Norwegian, and there is also a multi-language NER model. Additional support for tokenization for more than 50 languages allows users to train custom models on their own datasets as well.
  • Gensim: package for topic and vector space modeling, document similarity.
  • Regular Expression: regex function is used to match or find strings using a sequence of patterns consisting of letters and numbers.
  • NLTK (Natural Language Toolkit) library: breaks the given string and returns a list of strings by the white specified separator. NLTK is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for English written in the Python programming language.
  • Dictionary Based Tokenization: based on the tokens already existing in the dictionary
  • Rule Based Tokenization: based on a set of rules are created for the specific problem
  • Penn TreeBank Tokenization: Tree bank is a corpus, which gives the semantic and syntactic annotation of language

and many others. Which method to use will depend on your text (language, presence of white spaces or characters, special signs, type of genre (medical, scientific, etc.)) and tasks you need to achieve. Below is a good “pros and cons” comparison table for different libraries, which allows you in a more easy way to see, which method can be applicable in which case:

Numericalization

In order to turn tokens into numbers so that we can pass them into the neural network, we do numericalization, which can be achieved via creating a string-to-integer (i.e. stoi) and then use this to encode each of the tokenized sentences. In this way we are getting arrays, representing numericalized words and sentences which we can feed to the model.

You are ready to proceed to the next step: putting texts into batches for a Language Model!

Sources:

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot



via WordPress https://ramseyelbasheer.wordpress.com/2021/02/01/natural-language-processing-tokenization-and-numericalization/

Popular posts from this blog

I’m Sorry! Evernote Has A New ‘Home’ Now

Image
Original Source Here My Apologies! I was critical about the grandfather of all apps — Evernote a few months back when I wrote Why is Evernote losing its Ground? on Medium. With that said, I also wrote another story on Evernote in November of last year — Has Evernote Made A Comeback After Years Of Stagnation? This was the time when Ian Small and his team were making inroads into the technology and strategically making enhancements and revamping the app across all platforms. Evernote Is At A War-front And Wants To Prove A Point. I must h ave written this in my earlier blogs but just to re-iterate — I have been and am an avid user of Evernote since 2009 and is a part of my life for planning my daily life, projects, content writing, and almost everything. Recently, Evernote has launched its HOME or you may call it a DASHBOARD that provides its users a one-stop place to get access to their important and critical notes and also provides a few AI-enabled capabilities that I wil...
Read more

Jensen Huang: Racism is one flywheel we must stop

Image
https://venturebeat.com/wp-content/uploads/2021/07/jensen-huang.jpg?w=1200&strip=all Original Source Here All the sessions from Transform 2021 are available on-demand now. Watch now. Jensen Huang , CEO of Nvidia , received the distinguished lifetime achievement award by the Asian American Engineer of the Year from the Chinese Institute of Engineers (CIE) group. And in his acceptance speech, he made a rare comment beyond Nvidia’s business matters: “Racism is one flywheel we must stop.” The comments came at the end of an acceptance speech where he thanked the group for its award, an annual gathering that recognizes outstanding Asian American scientists, engineers and role models. In a virtual ceremony, Huang was awarded for his contributions as “a visionary and innovator in parallel computing technology that accelerates the realization of AI computing.” He also spoke of his experience as an immigrant and an Asian American. “It is strange to accept a lifetime achievem...
Read more

Streamlit — Deploy your app in just a few minutes

Image
Original Source Here Streamlit is an open-source Python library that makes it easy to create and share beautiful, custom web apps for machine learning and data… Continue reading on Medium » AI/ML Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot via WordPress https://ramseyelbasheer.io/2021/07/11/streamlit%e2%80%8a-%e2%80%8adeploy-your-app-in-just-a-few-minutes/
Read more