Summary Buffer Memory¶

The same CassandraChatMessageHistory can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.

In [1]:

from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain

from langchain.memory import CassandraChatMessageHistory from langchain.memory import ConversationSummaryBufferMemory from langchain.chains import ConversationChain

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
astraSession = getCQLSession()
astraKeyspace = getCQLKeyspace()

from cqlsession import getCQLSession, getCQLKeyspace astraSession = getCQLSession() astraKeyspace = getCQLKeyspace()

In [3]:

message_history = CassandraChatMessageHistory(
    session_id='summarized-conversation-4321',
    session=astraSession,
    keyspace='langchain',
    ttl_seconds = 3600,
)

message_history = CassandraChatMessageHistory( session_id='summarized-conversation-4321', session=astraSession, keyspace='langchain', ttl_seconds = 3600, )

In [4]:

message_history.clear()

message_history.clear()

Below is the logic to instantiate the LLM of choice. We choose to leave it in the notebooks for clarity.

In [5]:

from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'VertexAI', 'OpenAI' ... manually if you have credentials)

if llmProvider == 'VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from VertexAI')
elif llmProvider == 'OpenAI':
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'VertexAI', 'OpenAI' ... manually if you have credentials) if llmProvider == 'VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from VertexAI') elif llmProvider == 'OpenAI': from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from OpenAI

Use the memory with summary in a Chain¶

In [6]:

memory = ConversationSummaryBufferMemory(
    llm=llm,
    chat_memory=message_history,
    max_token_limit=180,
)

memory = ConversationSummaryBufferMemory( llm=llm, chat_memory=message_history, max_token_limit=180, )

In [7]:

summaryConversation = ConversationChain(
    llm=llm, 
    memory=memory,
    verbose=True
)

summaryConversation = ConversationChain( llm=llm, memory=memory, verbose=True )

With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.

In [8]:

summaryConversation.predict(input='Tell me about William Tell, the hero')

summaryConversation.predict(input='Tell me about William Tell, the hero')

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:

Human: Tell me about William Tell, the hero
AI:

> Finished chain.

Out[8]:

" William Tell is a legendary Swiss hero who lived in the early 14th century. He is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale. According to legend, William Tell was a patriot who was forced by the local Austrian governor to shoot an apple off his son's head as a demonstration of his marksmanship. When he succeeded, he was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries."

In [9]:

summaryConversation.predict(
    input="Any relation to the apple from Cupertino?"
)

summaryConversation.predict( input="Any relation to the apple from Cupertino?" )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss hero who lived in the early 14th century. He is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale. According to legend, William Tell was a patriot who was forced by the local Austrian governor to shoot an apple off his son's head as a demonstration of his marksmanship. When he succeeded, he was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Any relation to the apple from Cupertino?
AI:

> Finished chain.

Out[9]:

' No, there is no relation between the apple from Cupertino and the apple from the William Tell legend. The apple from Cupertino is the logo of Apple Inc., which was founded in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne. The company is known for its line of consumer electronics, including the iPhone, iPad, and Mac computers.'

No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:

In [10]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[10]:

"\nThe human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries."

Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:

In [11]:

summaryConversation.predict(
    input="I heard the two apples may be in fact the very same fruit."
)

summaryConversation.predict( input="I heard the two apples may be in fact the very same fruit." )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss hero who lived in the early 14th century. He is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale. According to legend, William Tell was a patriot who was forced by the local Austrian governor to shoot an apple off his son's head as a demonstration of his marksmanship. When he succeeded, he was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between the apple from Cupertino and the apple from the William Tell legend. The apple from Cupertino is the logo of Apple Inc., which was founded in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne. The company is known for its line of consumer electronics, including the iPhone, iPad, and Mac computers.
Human: I heard the two apples may be in fact the very same fruit.
AI:

> Finished chain.

Out[11]:

" I'm sorry, I don't have any information on that."

This is where the current summary buffer is stored

(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)

In [12]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[12]:

"\nThe human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries."

The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer.

In [13]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries. When asked about any relation to the apple from Cupertino, the AI responded that there is no relation between the two apples. When asked if the two apples may be the same fruit, the AI stated that it didn't have any information on that.

In [14]:

summaryConversation.predict(
    input='Ok, enough with apples. Let\'s talk starfish now.'
)

summaryConversation.predict( input='Ok, enough with apples. Let\'s talk starfish now.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss hero who lived in the early 14th century. He is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale. According to legend, William Tell was a patriot who was forced by the local Austrian governor to shoot an apple off his son's head as a demonstration of his marksmanship. When he succeeded, he was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between the apple from Cupertino and the apple from the William Tell legend. The apple from Cupertino is the logo of Apple Inc., which was founded in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne. The company is known for its line of consumer electronics, including the iPhone, iPad, and Mac computers.
Human: I heard the two apples may be in fact the very same fruit.
AI:  I'm sorry, I don't have any information on that.
Human: Ok, enough with apples. Let's talk starfish now.
AI:

> Finished chain.

Out[14]:

' Sure! Starfish are a group of marine invertebrates that belong to the phylum Echinodermata. They have five arms that can regenerate if they are damaged or lost. Starfish are found in all oceans and many of them have a hard external shell covered in spines. They eat mussels, clams, and other small creatures which they catch with their arms. Some species of starfish can even change color to blend into their environment.'

In [15]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[15]:

"\n\nThe human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries. When asked if there is any relation to the apple from Cupertino, the AI responds that there is no relation between the two apples, as the apple from Cupertino is the logo of Apple Inc. founded in 1976."

Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.

In [16]:

summaryConversation.predict(
    input='I hear that some of those destroy marine ecosystems.'
)

summaryConversation.predict( input='I hear that some of those destroy marine ecosystems.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 

The human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries. When asked if there is any relation to the apple from Cupertino, the AI responds that there is no relation between the two apples, as the apple from Cupertino is the logo of Apple Inc. founded in 1976.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss hero who lived in the early 14th century. He is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale. According to legend, William Tell was a patriot who was forced by the local Austrian governor to shoot an apple off his son's head as a demonstration of his marksmanship. When he succeeded, he was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between the apple from Cupertino and the apple from the William Tell legend. The apple from Cupertino is the logo of Apple Inc., which was founded in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne. The company is known for its line of consumer electronics, including the iPhone, iPad, and Mac computers.
Human: I heard the two apples may be in fact the very same fruit.
AI:  I'm sorry, I don't have any information on that.
Human: Ok, enough with apples. Let's talk starfish now.
AI:  Sure! Starfish are a group of marine invertebrates that belong to the phylum Echinodermata. They have five arms that can regenerate if they are damaged or lost. Starfish are found in all oceans and many of them have a hard external shell covered in spines. They eat mussels, clams, and other small creatures which they catch with their arms. Some species of starfish can even change color to blend into their environment.
Human: I hear that some of those destroy marine ecosystems.
AI:

> Finished chain.

Out[16]:

' Yes, some species of starfish can be damaging to marine ecosystems. For example, the crown-of-thorns starfish is an invasive species that feeds on coral and can devastate coral reefs. To combat this, some countries have implemented programs to remove and control the population of crown-of-thorns starfish in their waters.'

In [17]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[17]:

"\nThe human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries. When asked if there is any relation to the apple from Cupertino, the AI responds that there is no relation between the two apples as the apple from Cupertino is the logo of Apple Inc. founded in 1976. The human then asked if there was any connection between the two apples, to which the AI did not have an answer. The conversation then shifted to starfish."

From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation.

In [18]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks what the AI knows about William Tell, the hero. The AI responds that William Tell is a legendary Swiss hero who lived in the early 14th century and is best known as the figure who shot an apple off his son's head with a crossbow. The story of his life is the subject of a popular folktale, where he was forced to shoot an apple off his son's head as a demonstration of his marksmanship, and, upon succeeding, was released and went on to lead a successful rebellion against the governor. The legend has been a popular subject in art, literature, and popular culture for centuries. The human then asked if there was any relation to the apple from Cupertino, to which the AI responded that there was no relation between the two apples as the apple from Cupertino is the logo of Apple Inc. founded in 1976. The conversation then shifted to starfish, which are marine invertebrates that belong to the phylum Echinodermata, have five arms, and can regenerate if they are damaged or lost. The human then asked if some starfish can be damaging to marine ecosystems, to which the AI responded that some species can, such as the crown-of-thorns starfish which feeds on