Using Reflections to Compress LLM Context Data

Traditional software is being replaced by LLM-based software using SPQA, which consumes context about entities instead of traditional databases and queries. LLMs thrive on context but can be overwhelmed by too much input, making it challenging to process large amounts of data. Companies will face the challenge of compressing gigabytes or terabytes of data into something that can be consumed by LLMs. A concept called Reflections, inspired by a Stanford paper on Generative Agents, can help with this compression by turning observations into higher-level thoughts or reflections. Reflections can be used to characterize users, systems, threat actors, activity, markets, cultures, and trends, providing a way to compress raw events into something usable for LLMs. This process involves a multi-phased approach, where data is consolidated and compressed, and then classified using classical ML, before being analyzed by LLMs. However, this compression comes with a loss of original events, making it difficult to go backward and do attribution. The question of what to keep and what to discard is a significant challenge that many companies will work to address. Eventually, this will lead to the development of permanent pipelines flowing into continuous custom model training, but for now, Reflections will remain a crucial part of the process. The use of Reflections will be essential in closing the gap between the large amounts of data companies have and the limited input LLMs can handle, enabling the full potential of LLM-based software to be realized.