Knowledge base (KB), RAG (retrieval-augmented generation) processing

RAG processing using our multimodal, “Computer history” knowledge base

We are currently exploring the role of local knowledge bases (KBs) in RAG (retrieval-augmented generation) AI processing. This post is part of a series documenting our “sandbox” knowledge bases (created over a period of about 20 years) and how we’re using them in various GenAI prototyping projects.

Return to Colab notebook

Toward the end of last year (2024) we had set ourselves up to do our RAG processing locally using a small language model (SLM) and a stand-alone Python program running in the Thonny IDE.

For various reasons, we found that environment  unacceptable, so we recently returned to Colab notebook, which runs in the Cloud and makes processing- and information-sharing possible.

We are happy to be back to Colab!

A new knowledge base: “Computer history”

Also at the end of 2024, we put together yet another local “sandbox” knowledge base, our seventh, which we called “Computer history.” This local knowledge base contains about 40 components in 3 categories:

  1. Images taken by the authors as they cataloged artifacts and archives in the Computer History Museum in Mountain View, California
  2. Short narratives, written in HTML for our personal and family history website, newsfromnan.com, and then archived to PDF, about our our volunteer experiences in the Computer History Museum
  3. A longer narrative, called My High-tech Adventure, that Dick wrote about his career in the high-technology industry

Here are the components of our “Computer history” knowledge base.

Images and files in the "Computer history" knowledge base
Images and files in the “Computer history” knowledge base

The first version of our RAG processing project using “Computer history”

We expect to have multiple versions of this RAG processing project.

In Version 1, we asked the model to search through the names of the image files in the knowledge base, and speculate on the contents of the images based on text in the names.

We were surprised and pleased that with only a minimal system prompt the model was able to understand most of our file naming conventions on its own. The model also did well on describing what the image might look like based on the text in the name. Here are some example chat conversations.

Chat dialog in our "Computer history" processing project
Chat dialog in our “Computer history” processing project
Chat dialog
The model had no problem with the “last name, first name” format in the file name
Chat dialog
Interpreting text in the image file names

Link to our “Computer history” Colab notebook

https://colab.research.google.com/drive/1EMT_jE-QhAS7tc1uBc6Qua5BJ-CcuA9d?usp=sharing

This is a read-only notebook link, but if you are set up with a Google account you can make a copy of it and run it yourself.

Link to a PDF of the “Computer history” program

RAGProcessing_ComputerHistoryKB1.ipynb – Colab

Our plans for future versions of the “Computer history” RAG processing program

In future versions, the model will be asked questions about the contents of the narratives in the knowledge base, to compare the contents of multiple items, and to generate new text drawn from all available sources.

If we can find the right model to analyze the images, we’ll go into more detail about image content, as well.