Welcome back.
We are starting a series of lectures
where we'll build on what we've learned so far and see the foundation of an AI
application that can answer questions about the content of private documents.
At the beginning of the course, we set up the environment and the sign up for a
Pinecone account.
We have a Pinecone API key in a .env file
in the current directory.
You should keep the API key secure.
In my case, I'll rotate or generate another key after recording this video.
I'll do the same for all the keys shown in this course.
I'm loading them as environment variables using python .env.
This is the code that does the job.
When dealing with very long pieces of
text, it's necessary to split the text into chunks.
You also want to keep semantically related pieces of text together.
Let's look at an example.
We'll load the data to be split from a
text file, but you can load it from many other document types such as PDFs, CSV
files, spreadsheets, SQL databases, and so on.
In this example, we'll work with a text file.
However, there are Lanchain loaders that
can be used to load data from almost any type of documents, including CSV,
Evernote, file directories, Facebook chats, JSON files, PowerPoint, PDFs, and
many more.
There are also public service loaders
from different services such as Project Gutenberg, Hacker News, Wikipedia, and YouTube.
Additionally, there are proprietary service loaders for Amazon, Azure,
Google, and others.
In this example, I'll take the famous
speech by Winston Churchill, We shall fight on the beaches, which was delivered
to the House of Commons of the Parliament of the United Kingdom in June 1940.
This is the text file and I'll split it into chunks.
The default and recommended text splitter is called Recursive Character Text Splitter.
I'm importing it.
So from lanchain text splitter import
recursive character text splitter.
By default, the characters it tries to
split on are double backslash n, backslash n, and y space.
Now, I'm creating the text splitter object by calling the recursive character
text splitter constructor.
So text splitter equals recursive
character text splitter.
The first argument is the maximum size of
your chunks as measured by the length function.
We'll set a really small chunk size,
which is around a line of text just to show you how it works.
However, you should experiment with different values to see which one works best.
You'll normally use a higher value here.
We'll talk more about chunking strategies
for LLM applications later in the course.
Also note that this is the maximum size
of the chunk, but in reality, it can have a lower size.
The next argument is a chunk overlap.
This is the maximum overlap between
chunks needed to maintain some continuity between them.
And finally, the last argument length function.
It indicates how the length of chunks is calculated.
The default is to just count the number of characters, but
because we'll work with LLMs, and LLMs use tokens not characters, it's pretty
common to pass a token counter here.
I'm creating the chunks.
Chunks equals text splitter dot create documents.
And it takes as an argument a list of characters.
Let's say Churchill speech.
Of course, I have to open and read the
contents of the file as a string in this variable.
I'm doing it now.
I'm opening the file as usual.
So with open files Churchill speech dot
txt as f and the Churchill speech equals f dot read.
This file is in a directory called files in the current one.
You should use a correct relative or absolute path.
I'm running the cell.
Let's see the first chunks.
Chunks of zero.
This is the first one.
This is the second chunk, the third chunk, and so on.
Each chunk is approximately a line.
If you want to see only the text, use
chunks and an index, let's say 10 dot page content.
And I've got the text only.
Let's see how many chunks do I have.
So now you have len of chunks and the chunks.
Good.
There are 300 chunks.
We'll be using OpenAI's text embedding
ADA002 which has a cost.
It's a good idea to calculate the
embedding costs in advance to avoid any surprises.
We'll use the tick token library for this.
I've already shown you how to in the first part of the course, so I'll just
copy and paste the code now.
This is the function and I'm calling it.
So print embedding cost of chunks.
So there are in total 5820 tokens and the
embeddings will cost a fraction of a cent.
It's really cheap.
The next step is to import and instantiate OpenAI embeddings.
I'm importing the required library.
From lang chain embeddings, I'm importing
OpenAI embeddings and the embeddings equals and I'm instantiating the object.
If the API key is loaded in an environment variable, it is not necessary
to pass it as an argument to this function.
The OpenAI embeddings class can be used
to embed text to vectors.
For example, let's turn the first chunk
into a vector of floating point numbers.
So vector equals embedding the object
I've just created dot embed query and the argument will be a text.
It can be any text.
This is the embedding of a string abc.
Let's see the embedding of the first chunk.
So chunks of zero dot page content.
This is the embedding of the first chunk.
Great.
In this video, I've shown you how to split text and embed them into vectors
using lang chain and OpenAI.
We'll take a break and in the next video,
we'll move to Pinecone and see how to insert the embeddings into a Pinecone index.