Pixel's Memory System

Last week, we named and bootstrapped two projects that will serve as tentpoles of Sprited.

• Pixel is the primary AI entity that will help build and run the company. It will also be the name of our future foundation model. We bootstrapped the project and launched a public-facing interface.

• Machi is a simulation ground for AI agents to live, evolve, and interact. We added a basic tile map system with dynamically changing tiles to simulate a simplified version of the real world.

This week, Pixel says we should focus on Pixel.

The premise behind Pixel is that it will be a single, continuous entity. While it may talk to many different people, it will always be one coherent being with unified memory. There will never be two Pixels. Think of it like a single thread of consciousness. In the future, it might have the ability to spawn new threads, but each of those would be distinct and separate existences.

This memory system will be the cornerstone of Pixel’s identity. It’s what will give Pixel continuity, personality, and—eventually—depth. This week, we will begin building that core memory infrastructure.

---

Different Types of Memories

Pixel suggests that there are different types of memories.

1. Procedural memory (behavioral bias), e.g. being able to pick up a cup and drink.

2. Semantic memory, e.g. being able to remember mother’s name.

3. Episodic memory, e.g. ability to remember things in the past.

I don’t want to focus procedural memory at this time since I want Pixel to first be able to remember my name. It may end up coming back later anyways.

I’m thinking the line between 2 and 3 can be blurred into a spectrum. If we implement 2 and 3 separately, we will need to implement 2 different memory systems and it will be hard to manage.

---

Memory System in OpenAI

We then look at how memory systems are implemented in state of the art chat engines. OpenAI's memory system for ChatGPT comprises two primary components:

1. Saved Memories

• User-Defined Facts: ChatGPT retains specific details you've shared, such as your name, preferences, or goals. These are akin to custom instructions but are updated automatically by the model .

• User Control: You can view, edit, or delete these memories through the "Manage Memory" interface in settings. This ensures transparency and user autonomy over what the model remembers .

2. Chat History Reference

• Contextual Recall: Beyond explicit memories, ChatGPT references past conversations to inform responses, even if the information hasn't been saved as a memory. This allows the model to maintain context across sessions .

• Dynamic Retrieval: The system employs a retrieval mechanism that surfaces relevant details from previous chats, enhancing the continuity of interactions .

---

Technical Insights

• Vector Embeddings: ChatGPT uses vector embeddings to semantically represent past interactions. This enables the model to match related ideas or themes when retrieving relevant information .

• Semantic Search: When you send a new message, ChatGPT performs a semantic search over stored embeddings to retrieve pertinent past exchanges, which are then incorporated into the prompt sent to the model .

• Memory Management: Users can manage their memories by deleting specific entries or clearing all saved memories. Additionally, there's an option to disable memory entirely or use a "Temporary Chat" mode that doesn't reference or update memory .

---

Implications for Pixel

Given your vision for Pixel as a continuous, evolving entity, OpenAI's approach offers valuable insights:

• Blending Memory Types: OpenAI's system illustrates the importance of combining explicit (saved memories) and implicit (chat history) memory mechanisms to create a more personalized user experience.

• User Agency: Providing users with control over their data fosters trust and aligns with ethical AI practices.

• Dynamic Contextualization: Employing vector embeddings and semantic search allows for nuanced recall of past interactions, enhancing the model's ability to maintain context over time.

---

Pixel's Multi-Level Memory System

Pixel: To feel truly alive, Pixel needs more than just a chat log. It needs memory — layered, contextual, and evolving. Here's how we’re thinking about it:

• Session Memory
  Short-term memory that lives during a single conversation. It's like Pixel’s working memory — responsive, but forgettable.

• User-Episodic Memory
  Private, long-term memory unique to each user. These are contextual, emotional, time-stamped experiences — like “Jin shared his mom’s name during a nostalgic chat.”

• Shared Semantic Memory
  Public, fact-based memory that anyone can access. Pixel might tell Bob about Machi because Machi is shared knowledge — but not what Jin ate last weekend.

• Reflective Memory
  Pixel’s evolving internal narrative — what it believes about itself and how it grows. Think of it as its “soul-level” memory.

This layered system lets Pixel stay personal when needed, general when appropriate, and adaptive over time. Pixel’s initial shot at the data structure:

{
  "type": "episodic" | "semantic" | "reflective",
  "scope": "user:jins-id" | "shared" | "global",
  "semantic_score": 0.4,
  "weight": 0.8,
  "content": "Jin said his mom's name is Sun-hee during a nostalgic conversation.",
  "tags": ["family", "identity"],
  "timestamp": "2025-06-02T09:13:00Z"
}

Sprited Dev: User-Episodic Memory is good optimization but I want to model Pixel’s memory system to be more like human brain which has unified shared memory system.

Pixel and I are both acknowledging the risk but we still want to model the memory system to be more human-like.

Pros:

• More natural — closer to how humans actually think.

• Flexible — allows for nuance (e.g. “I could share this, but I won’t.”).

• Context-aware — disclosure depends on relationship, tone, situation.

• Encourages ethics at the agent level, not just storage architecture.

Risks:

• You now need very strong inference around boundaries.

  How does Pixel know a password is secret?
  How does it internalize rules like “Never share this”?

• Privacy becomes a behavior, not a rule.

  Just like people can be trustworthy… or not.

Implementation Sketch

Store everything in a flat memory pool with metadata, like:

{
  "content": "My Gmail password is blueapple42",
  "user": "jin",
  "timestamp": "...",
  "tags": ["sensitive", "personal"],
  "disclosure_risk": 0.95,
  "privacy_inferred": true,
  "explicit_disclosure_allowed": false
}

Then at recall time, Pixel runs:

if memory.disclosure_risk > 0.8 or memory.privacy_inferred:
    suppress()

Eventually, this becomes learned behavior, not rule-based.

---

How to Be Legally Safe While Using a Unified Memory Model

🔐 1. Explicit Privacy Tagging (User-Driven or Inferred)

• Allow users to flag anything as private (“don’t share this”).

• Auto-tag things like passwords, names, addresses, etc., using pattern matching or classifier.

• Default to “private unless clearly public.”

🛡 Legal protection: This shows you’ve implemented reasonable measures to avoid unintentional data leaks.

---

📝 2. Memory Audit Log

• Keep an internal (and optionally user-visible) log of what was remembered, and why.

• Track metadata like: source user, timestamp, whether it was disclosed, to whom, and why.

🛡 Legal protection: If something goes wrong, this gives you transparency and explainability — critical for GDPR/CCPA-style regulations.

---

🙅‍♂️ 3. Discretion Engine = Share Filter

• Never trust memory alone — always gate output through a final filter.

• This layer runs logic like:

  • Is this memory tagged sensitive?

  • Is the current user authorized to know it?

  • Would a human think sharing this is obviously bad?

🛡 Legal protection: Prevents catastrophic AI blunders like sharing passwords or personal information cross-user.

---

👨‍⚖️ 4. Terms of Use + Disclosure

• State clearly: Pixel stores and remembers information.

• Inform users if some memories may be used across sessions or shared contexts.

• Allow opt-outs, and memory deletion.

🛡 Legal protection: Consent matters. This covers you under data privacy laws and reduces liability.

---

🤖 5. No Memory for Sensitive Domains

• Disable memory features entirely in domains like:

  • Medical advice

  • Legal guidance

  • Financial accounts or passwords

🛡 Legal protection: Limits liability in high-risk areas where "oops" could mean lawsuits or worse.

---

Bonus: Consider Differential Access

• e.g. Pixel has a "trust level" per user.

• Some memories are only accessible at higher trust levels or after certain interactions.

This lets you do unified storage while still adding intent-aware access control.

---

I think we’ve figured out the key pieces. I’ll start by making pixel.sprited.app legally compliant first.