In this video, we ll talk about streaming.
In the context of LLMs, streaming refers
to the process of delivering the response in a continuous stream of data instead of
sending the entire response at once.
This allows the user to receive the
response piece by piece as it is generated which can improve the user
experience and reduce the overall latency.
I am asking Copilot to write a rock song
about the moon and the raven.
As you can observe, Copilot is streaming
the answer not delivering the entire response at once, but piece by piece.
Let s get back to coding and assign the same task to the LLM.
I am importing chatOpenAi and creating the LLM object and the prompt will be
write a rock song about the moon and the raven and print LLM .invoke of prompt .content.
By default, the model will return the response after completing the entire
generation process.
Let s run the code.
It is running.
Let s enable streaming.
The model will send you pieces of the response as they are completed rather
than waiting until the entire response is finished.
To enable streaming, simply call LLM
.stream with the prompt as an argument.
To observe the streaming, I will do the following.
For chunk in LLM .stream of prompt, print chunk .content and equals empty string
and the flash equals true.
I am running the code.
You can see how it is streaming the response sending it in pieces.
If you want token by token streaming, this requires native support from the LLM provider.
That s it.
In this video, you learned about
streaming, what it is and how to implement streaming in Landshare.