Steve HutchinsonBig Pines

Experience ingestion

The pipeline stage that accepts incoming experience events and publishes them to the event bus for downstream enrichment, indexing, reasoning, and evaluation.

Experience ingestion is the entry point to the memory pipeline - the boundary where external experience becomes internal data. The ingestion endpoint accepts raw experience events (from the cognitive loop's act-and-evaluate step, from external sensors, or from synthetic event generators during dreaming), validates their structure against the Schema Registry, and publishes them to the memory.experiences.raw Kafka topic. At this stage the event is not yet enriched: no embeddings, no importance score, no initial reinforcement value. Those are added by the enrichment worker downstream. The ingestion layer's job is to accept, validate, and durably record - not to process. This separation means the ingestion endpoint can be extremely fast (validation and publish only) while the more expensive enrichment work happens asynchronously in parallel workers. Schema validation at ingestion time is where malformed or schema-violating events are caught and routed to the dead-letter queue rather than silently corrupting the downstream index.

This site collects anonymous usage data to understand how people read and navigate the blog. Accepting enables persistent reader preferences across visits.