Steve HutchinsonBig Pines

Enrichment

A processing step that adds embeddings and metadata - tags, importance scoring, and initial reward values - to raw experience events before they are indexed.

Enrichment is the transformation step between raw experience ingestion and indexed, retrievable memory. A raw experience event contains the input, the system state, the action, and the outcome - but it lacks the derived fields that make it useful for retrieval and reinforcement. Enrichment adds those fields: embeddings across all three lanes (quality, efficient, hybrid), computed importance score based on outcome magnitude and goal relevance, initial reward value seeded from the evaluation score, and classification tags that determine which downstream indexes should receive the event. The enrichment pipeline is also where the multi-lane embedding architecture pays off operationally: enriching once at ingestion time means the index always has all three embedding vectors ready, rather than computing them on demand at query time. Enrichment workers scale horizontally via consumer groups on the ingestion Kafka topic, with each worker processing a partition independently. Enrichment failures route to a dead-letter queue rather than silently dropping events, ensuring that enrichment errors are visible and reprocessable.

This site collects anonymous usage data to understand how people read and navigate the blog. Accepting enables persistent reader preferences across visits.