Three-Layer Memory for a 3B On-Device LLM Companion: Giving a Small LLM a Long Memory#

Here's a fact that breaks most intuitions about building an AI companion: an on-device language model on an iPhone doesn't have a 200,000-token context window like a top-tier cloud model. It has around 4096. And that's not just the input — system instructions and the model's own reply share the same budget. If you design a feature as if there's a GPT-4-class model with near-unlimited memory in someone's pocket, it breaks by the third day of conversation: the model either "forgets" the start of the chat or starts confusing what's actually happening.

I'm currently building Lanternly, a journaling companion with an AI that runs entirely on-device via Apple Foundation Models (I wrote about the framework itself separately: Apple Foundation Models: On-Device LLM in an iOS App). My first memory architecture didn't survive contact with reality: either the context overflowed and the model started drifting, or I had to cut the history so aggressively that important details got lost. The fix wasn't a clever truncation trick — it was splitting memory into three independent layers, each with its own role, storage location, and owner. Here's how it's built, and why.

Why 4096 Tokens Is a Different Architecture, Not a Smaller One#

Developers who've only worked with cloud LLMs tend to solve memory one way: "send more history in the prompt." With a 200,000+ token window that almost always works — you can stuff in dozens of messages, whole documents, chunks of a knowledge base, and the model sorts it out. That creates a false sense that context management is an implementation detail you can postpone rather than an architectural decision.

That doesn't fly on-device. A 4096-token budget splits between instructions (the system prompt with persona, tone, rules), the reply itself (you need to leave room, or generation gets cut off mid-sentence), and whatever's left — usually 2500–3000 tokens for the entire conversation context. That's a few dozen short turns, no more. So context management isn't an optimization; it's a requirement without which the feature doesn't work at all. What follows isn't one clever "smart history trimming" trick — it's three separate mechanisms, each solving a different piece of the problem.

Layer 1 — The Model's Context: A Sliding Window with Summarization on Overflow#

The first layer is what physically lands in the prompt for a specific model call: system instructions plus the last N turns of the conversation. The naive fix is to just drop the oldest messages when things get tight. It works, but you pay for it with the conversation itself: if the person mentioned something important five messages ago and the window has already shifted, the model will never see it.

The working solution is a sliding window that, on overflow, doesn't discard old turns but folds them into a short summary via one extra model call. The upper half of the window collapses into 2–3 sentences of summary; the lower half stays verbatim. That's the difference between "forgetting" and "compressing" — the model loses exact wording but keeps the gist.

import FoundationModels
 
/// Layer 1 — the model's own context: a sliding window that collapses
/// into a rolling summary instead of silently dropping older turns.
struct ConversationWindow {
    private let maxTokens: Int
    private(set) var turns: [ChatTurn] = []
    private(set) var rollingSummary: String?
 
    init(maxTokens: Int = 3200) {
        self.maxTokens = maxTokens
    }
 
    /// Rough heuristic: ~4 characters per token (good enough for budgeting,
    /// not for billing — on-device models don't charge per token anyway).
    private func estimatedTokens(_ text: String) -> Int {
        text.count / 4
    }
 
    private var currentTokens: Int {
        turns.reduce(rollingSummary.map(estimatedTokens) ?? 0) {
            $0 + estimatedTokens($1.text)
        }
    }
 
    mutating func append(_ turn: ChatTurn) async {
        turns.append(turn)
        guard currentTokens > maxTokens, turns.count > 4 else { return }
        await collapseOldest()
    }
 
    /// Move the oldest half of the window into a rolling summary and keep
    /// only the freshest turns verbatim. This is the difference between
    /// "forgetting" and "compressing" — the model still knows the gist.
    private mutating func collapseOldest() async {
        let cut = turns.count / 2
        let evicted = Array(turns.prefix(cut))
        turns.removeFirst(cut)
        rollingSummary = await summarize(evicted, previous: rollingSummary)
    }
 
    private func summarize(_ evicted: [ChatTurn], previous: String?) async -> String {
        guard case .available = SystemLanguageModel.default.availability else {
            return previous ?? ""
        }
        let session = LanguageModelSession(instructions: """
        Summarize the earlier part of this conversation in 2-3 neutral \
        sentences. Keep names, decisions and open questions; drop small talk.
        """)
        let transcript = evicted.map { "\($0.role): \($0.text)" }.joined(separator: "\n")
        let prompt = previous.map { "Previous summary: \($0)\n\nNew turns:\n\(transcript)" } ?? transcript
        return (try? await session.respond(to: prompt).content) ?? (previous ?? "")
    }
}
 
struct ChatTurn {
    enum Role: String { case user, assistant }
    let role: Role
    let text: String
}

Important: this layer is ephemeral. It exists only inside a single LanguageModelSession and is rebuilt from scratch on every model call, from data owned by layers 2 and 3. It doesn't persist anything on its own.

Layer 2 — Chat History on Disk: SwiftData + iCloud#

The second layer answers the question layer 1 deliberately ignores: what actually happened in the conversation, in full, uncompressed? This is a complete, unaltered log of every turn — stored via SwiftData locally on the device and synced across the person's devices through iCloud.

The distinction from layer 1 is fundamental: layer 1 is a working, truncated projection for the current model call; layer 2 is the source of truth. It's what summaries get built from, it's what the person can scroll back through to reread an old conversation, and it's what the person can delete — in whole or in part — without that breaking the current model session in any way.

import SwiftData
 
/// Layer 2 — chat history on disk: the full, uncompressed log, kept
/// independent from whatever the model's context window currently holds.
@Model
final class ChatMessage {
    var id: UUID = UUID()
    var role: ChatTurn.Role = .user
    var text: String = ""
    var createdAt: Date = .now
 
    init(role: ChatTurn.Role, text: String) {
        self.role = role
        self.text = text
    }
}

The practical payoff: if the app loses track of a conversation because the context overflowed, the person loses nothing — the full history is intact on disk. It simply doesn't fit inside the budget of a single model call right now, and that's a different problem, not one that should be solved by the same code.

Layer 3 — "Memory About You": Facts You Can See and Erase#

The third layer is a short, editable digest of durable facts about the person: preferences, important people, habits — things worth remembering across entirely different conversations, even once layer 1 has long since forgotten and rereading all of layer 2 for one detail would be needlessly expensive.

Facts arrive two ways. The person can add them manually. And the model can, best-effort and without blocking the main reply, try to extract one durable fact from a long-enough message. If there's no fact, the model replies with a special marker and nothing gets saved; if a fact resembles one that already exists, it's discarded as a duplicate.

import SwiftData
 
/// Layer 3 — "Memory about you": a short, editable list of durable facts.
/// Lives on disk (and syncs via iCloud), independent of any single chat
/// session. The user can see, edit and delete every entry.
@Model
final class MemoryFact {
    var id: UUID = UUID()
    var text: String = ""
    var source: MemorySource = .manual
    var createdAt: Date = .now
 
    init(text: String, source: MemorySource = .manual) {
        self.text = text
        self.source = source
    }
}
 
enum MemorySource: String, Codable {
    case automatic  // extracted by the model from a conversation
    case manual     // added directly by the user
}
 
/// Best-effort auto-extraction: after a long-enough message, ask the model
/// for at most one durable fact. Never blocks the reply if it fails.
enum MemoryExtractor {
    static func extract(from text: String, existing: [MemoryFact]) async -> MemoryFact? {
        guard text.count > 25,
              case .available = SystemLanguageModel.default.availability else { return nil }
 
        let session = LanguageModelSession(instructions: """
        Extract ONE durable fact about the person from their message \
        (a preference, an important person, a habit worth remembering). \
        One short phrase, no quotes. If there is no durable fact, reply NONE.
        """)
        guard let response = try? await session.respond(to: text) else { return nil }
        let fact = response.content.trimmingCharacters(in: .whitespacesAndNewlines)
        guard !fact.isEmpty, fact.uppercased() != "NONE", fact.count < 120 else { return nil }
 
        let isDuplicate = existing.contains {
            $0.text.localizedCaseInsensitiveContains(fact)
        }
        return isDuplicate ? nil : MemoryFact(text: fact, source: .automatic)
    }
}

The key architectural decision here isn't technical — it's product-level: every fact is tagged with its source (automatic / manual), and the person sees the entire list in settings, can edit any entry's wording, or delete it without a trace. Nothing sits hidden "between the lines" of the model, and nothing comes back after deletion.

Assembling Context: Fitting Three Layers into a Token Budget#

Before every model call, the three layers get packed into a single instructions string that has to fit inside Foundation Models' real limit — 4096 tokens for the entire exchange, reply included. Order matters: layer 3 goes first, because facts about the person are cheap (a couple of lines) and give the model the most value per token spent. Then comes the compressed part of layer 1 (the rolling summary), if it exists. Only what's left of the budget goes to the verbatim tail of recent turns, starting with the freshest.

/// Packs all three layers into one instructions string that fits inside
/// the model's real context limit (Foundation Models: ~4096 tokens total,
/// input + output). Order matters: identity facts are cheap and high-value,
/// so they're never the first thing dropped.
struct PromptBudget {
    let totalTokens = 4096
    let reservedForResponse = 512
    let reservedForInstructions = 300
 
    var availableForContext: Int { totalTokens - reservedForResponse - reservedForInstructions }
 
    func assemble(memory: [MemoryFact], summary: String?, recentTurns: [ChatTurn]) -> String {
        var budget = availableForContext
        var blocks: [String] = []
 
        // Layer 3 — identity facts first: small, stable, high signal.
        if !memory.isEmpty {
            let text = memory.map { "- \($0.text)" }.joined(separator: "\n")
            blocks.append("About the person:\n\(text)")
            budget -= text.count / 4
        }
 
        // Layer 1, compressed part — the rolling summary of evicted turns.
        if let summary, !summary.isEmpty, budget > 200 {
            blocks.append("Earlier in the conversation: \(summary)")
            budget -= summary.count / 4
        }
 
        // Layer 1, verbatim tail — fill what's left, newest turns first.
        var tail: [String] = []
        for turn in recentTurns.reversed() {
            let cost = turn.text.count / 4
            guard cost < budget else { break }
            tail.insert("\(turn.role.rawValue): \(turn.text)", at: 0)
            budget -= cost
        }
        blocks.append(contentsOf: tail)
 
        return blocks.joined(separator: "\n\n")
    }
}

This is the code that turns three separate layers into one seamless experience for the person, while the layers themselves stay independent and each solve their own part of the problem.

On-Device vs. Cloud: Why You Can't Just Copy MemGPT or mem0#

The temptation to borrow a ready-made pattern from the cloud-agent world is strong. MemGPT, and the Letta framework built on the same idea, treat the context window as OS-style virtual memory: core memory is the model's "RAM," while archival and recall stores are "disk," and the model itself decides what to page in by calling dedicated tools mid-conversation. mem0 takes a different route — after every turn, a separate LLM call extracts atomic facts and classifies an operation on them (add, update, delete, or no-op), checking against similar entries in a vector store.

Both approaches are solid and clever — and both assume a 200,000+ token budget and that an extra model call costs almost nothing in time or money. On a small on-device model with a 4096-token ceiling and no vector database of its own, that literally doesn't scale: every extra "let the model decide what to remember" call eats into the same budget the model needs for the actual reply. So layer 3 in this architecture doesn't hand the model control over its own memory — it's a deterministic, cheap mechanism with a fixed extraction prompt, a hard length cap, and simple substring-based deduplication; the app, not the model, decides when to run it and what fits the budget. Less flexible than MemGPT or mem0, but predictable and cheap enough to run after every single message, right on the phone.

Privacy as Architecture, Not a Settings Toggle#

Three-layer memory has a side effect that matters more, in practice, than any token optimization: it's structurally the opposite of a black box. History and facts live in SwiftData under the person's own iCloud account — not in someone else's cloud, not in a server-side vector store the person can't reach. Layer 3 doesn't hide what the app "knows" about the person: the fact list is visible, every entry can be corrected or deleted, and once deleted it doesn't resurface from a hidden cache or vector index — because no hidden index exists.

This isn't just a nice detail for one particular journaling companion. It's an architectural conclusion that applies to any long-memory feature built on a small on-device model: if you don't have the budget for MemGPT-grade cloud infrastructure, splitting memory into three layers doesn't give you a lesser substitute for "real" long-term memory — it gives you memory that honestly tells the person what it stores and why.

To close, three questions worth answering before designing memory for your own on-device LLM feature:

• What should land in this specific model call right now — and can you actually compute the budget instead of guessing?
• What needs to be kept in full, even if the model never sees it as one chunk?
• Which handful of facts deserve to outlive any single conversation — and are you willing to show them to the person exactly as stored?

Answer all three clearly, and a small on-device model's memory stops being its weak point and becomes a deliberate architectural choice — with a level of transparency that cloud black boxes usually can't match.