Simple NLP in Python With TextBlob: Tokenization
Introduction
The amount of textual data on the Internet has significantly increased in the past decades. There’s no doubt that the processing of this amount of information must be automated, and the TextBlob package is one of the fairly simple ways to perform NLP – Natural Language Processing.
It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, tokenization, sentiment analysis, classification, translation, and more.
No special technical prerequisites for employing this library are needed. For instance, TextBlob is applicable for both Python 2 and 3. In case you don’t have any textual information for the project you want to work on, TextBlob provides necessary corpora from the NLTK database.
Installing TextBlob
Let’s start out by installing TextBlob and the NLTK corpora:
$ pip install -U textblob