Hello everyone and welcome to the introduction to natural language processing lecture.
While there's no official reading assignment if you would like I would suggest reading the Wikipedia
article on natural language processing to get some context for this lecture.
However you don't have to read the whole article.
Just go ahead and browse through it.
Let's talk about natural language processing and why you may want to use it.
Natural Language Processing or an LP serves a lot of use cases when you're dealing with text or unstructured
text data.
Imagine you work for Google News and you want to group news articles by topic or maybe you work for
a legal firm and you need to sift through thousands of pages of legal documents to find the relevant
ones.
This is where natural language processing can help.
We will want to compile the documents in some fashion get features from them so we'll have to featuritis
those documents and then compare their features.
Let's go through a simple example.
Imagine you have two very simple documents.
In this case the document is just blue house and then the second document is Red House.
That means it's just a document of basically a single sentence.
So the first sentence is Blue House document one second sentence red houses document to a simple way
to featuritis text documents to feature is based off a word count.
So we transform blue house into a vectorized word counts.
We basically create a vector count of all the possible words throw all the documents in this case they're
red blue and a house and then we just count how many times those words occur in each document.
That means in this case for document one blue house we get 0 1 1 since Red occurs 0 times blue occurs
once house occurs once.
Similarly her red house we get 1 0 or 1 because red occurs once blue 0 times in house one time a document
represented as a vector of counts is called a bag of words.
And hopefully you saw that back of words in the Wikipedia article if you decided to read up on it.
Once we have these bags of words vectors you can use co-signed similarity on the vectors to determine
similarity of the documents themselves.
This is really useful because we're basically treating each document as a vector of features meaning
we can perform mathematical operations such as the Cocytus similarity taking their dot products and
then dividing it by the multiplication of their magnitudes or other similarity metrics in order to figure
out how similar two text documents are to each other.
We can improve on bag of words by justing word counts based on their frequency in the corpus and the
corpus just stands for the group of all the documents because is something called T.F. IDF or at term
frequency inversed document frequency.
To do this let's go over some definitions.
Term frequency is the importance of the term within that document.
Soldano T.F. term frequency is a function taking in D and T were the number of occurrences of term t
in document D.
Then we have the inverse document frequency which is the importance of the term in the corpus itself
.
So all the documents we'll call that IDF of T equals log capital D divided by lowercase T where capital
D is the total number of documents and lowercase t is equal to a number of documents with the term mathematically
.
Then we can express T.F. IDF as the following equation.
Here we're saying term X within document Y so we can say T F X comma Y is the frequency of x and y and
then we multiply that by the log of the total number of documents divided by the number of documents
containing x.
The reason we do this is so that we can get not just a word count but also some sort of notation on
how important a word is not just relevant to the document but to the entire corpus of all the documents
now that we have gone over the theoretical background the high natural language processing.
We can begin to discuss how we're going to use Python to do natural language processing before we get
started however we'll need to download an additional library go ahead and go to your terminal or command
line and use the install and LPK or pipit install an old T.K. depending on what distribution of Python
you're using the NL T.K. library or package is going to allow us to work with natural language or text
data very easily.
Once you have that package installed we'll go ahead and start our example by building a spam filter
if Python then your portfolio project will have you working the real data from Yelp and online reviews
site.
OK thanks everyone and I'll see at the next lecture