In this video, we ll go over caching in Lankchain to boost performance and save costs.
What is caching?
Caching is the practice of storing
frequently accessed data or results in a temporary faster storage layer.
In the context of Lankchain, caching helps optimize interactions with DLLMs.
It matters because it reduces API calls and speeds up applications.
When you repeatedly request the same completion from an LLM, caching ensures
that the result is stored locally.
Subsequent requests for the same input
can then be served directly from the cache, reducing the number of expensive
API calls to the LLM provider.
By avoiding redundant API calls, caching
significantly speeds up your application, whether you are building chatbots,
content generators, or any other language -related tools, faster responses enhance
the user experience.
Let's see how it is done.
Lankchain provides an optional caching layer for LLMs and there are two caching
options, in -memory cache and SQLite cache.
Let's implement in -memory cache.
From Lankchain Globals, I am importing set -llm -cache and from Lankchain
OpenAI, I am importing OpenAI.
I am creating an LLM and I will select on
purpose a slower model for demonstration purposes.
LLM equals OpenAI of model name equals
GPT 3 .5 TurboInstruct.
To measure the response time of the
model, use % %time magic command like this.
The % %time magic command in Jupyter
notebook is used to measure the execution time of the code within the current cell.
Next, I will set up an in -memory cache.
From Lankchain Cache, import the
InMemoryCache class and I am setting the InMemoryCache by calling set -llm -cache
with the constructor of InMemoryCache as an argument.
The prompt will be, Tell me a joke that a toddler can understand.
I am making the first request, llm .invoke of prompt.
I am running the code within this cell.
The request was not in cache and took longer.
The total CPU time was 31 milliseconds and the wall time was almost 500 milliseconds.
Let s make the same request.
This time, the response is already in the
cache and it will serve it from there, llm .invoke of prompt.
Take a look at the difference.
Now, the CPU time is 0 and the wall time
is 1 millisecond.
Amazing, isn t it?
Let s talk about SQLite caching.
If you want to use SQLite for caching,
just import the SQLiteCache module and set the SQLiteDatabase path using set
-llm -cache.
To save time, I am just pasting the code
because it s really simple.
That s it.
In this lecture, you learned about Langchain Caching and how to implement it
in memory or using SQLite to ensure efficiency, cost savings, and quick responses.