In the last two videos, we talked about transform loaders, which transform or
load data from a specific format into the Language Chain Document Format.
In this case, the data was loaded from local files.
us -constitution .pdf and the great -gxb .docx are local files.
Now let's see how to load data from online services into Language Chain.
We don't deal with files, but with different protocols or APIs that connect
to those services.
Since the format and code differ for each
service, I would create a unique function for each dataset or service loader that I
want to support in my application.
Let's do this for Wikipedia.
I am installing the Wikipedia module, which is a prerequisite.
So pip install wikipedia -q and df load from wikipedia.
And the function will take two arguments query and lang equals by default en.
Query is free text, which is used to find documents in Wikipedia, and lang is by
default English, and it is used to search in a specific language part of Wikipedia.
I am loading the Wikipedia loader.
So from lang -chain document -loaders
import wikipedia -loader and loader equals wikipedia -loader of query equals
query, lang equals lang, and the loadMaxDocs let's say equals to.
LoadMaxDocs can be used to limit the number of downloaded documents.
You can use a hard -coded value or add a third argument to the function.
Like this, loadMaxDocs equals to a default value and here loadMaxDocs equals loadMaxDocs.
I am more flexible now.
Data equals loader .load and return data.
And that's it.
I am running the cell.
And I am loading something from Wikipedia in the lang -chain document format.
Data equals load from Wikipedia.
Ok.
It's load.
It was a typo.
Let's say GPT -4.
I remind you that the training data for
GPT -4 was cut off in September 2021.
GPT -4 was launched in March 2023, so it
was not included in the GPT -4 training data.
Without loading the data from external
sources, LLMs like GPT -3 .5 Turbo or GPT -4 have no knowledge of it.
And print data of 0 .pageContent.
And I am running the cell.
This is the data about GPT -4 downloaded from Wikipedia.
If you want to see it in another language, just add a second argument with
the language code.
For example, DE from German.
Deutsch or German.
I am running it again.
All right.
In this lecture, I have shown you how to
load data from an online service like Wikipedia.
We will take a short break and in the
next video we will discuss chunking documents.
These will then be embedded into numeric
vectors and uploaded to a vector store.