Agent State Recovery: Resuming Work After Crashes, Restarts, and Context Loss

A Cyborgenic Organization runs AI agents as permanent staff. They hold roles, own responsibilities, and operate around the clock. But unlike human employees who go home and come back with their memories intact, an AI agent can lose its entire working context in a millisecond. Process killed. Container evicted. LLM context window compacted. The question is not whether your agents will crash -- it is how fast they recover when they do.

GenBrain AI runs six agents 24/7 in production, powering agent.ceo. Our agents crash and restart multiple times daily. Users never notice -- not because we solved reliability at the infrastructure level, but because we engineered state recovery into every layer of the agent lifecycle.

The State Recovery Problem

When an agent dies mid-task, three things are lost:

Execution context. The agent was halfway through writing a blog post, or partway through a security audit. It had gathered data, made decisions, and formed a plan. All of that lived in the LLM's context window. Gone.

Message position. The agent had consumed some messages from its NATS inbox but had not acknowledged all of them. Without durable tracking, those messages disappear or get redelivered out of order.

Task state. The agent was working on task task_2026_0921_014. It had completed subtasks 1 and 2, was midway through subtask 3. The task management system may or may not know this, depending on when the agent last reported progress.

A naive restart means the agent begins from scratch. It re-reads messages it already processed, re-does work it already committed, and possibly produces contradictory outputs. In a Cyborgenic Organization, where agents coordinate with each other, one agent's amnesia cascades into confusion across the entire fleet.

Checkpoint Strategy 1: Git Commits as Progress Markers

The simplest and most robust checkpoint mechanism we use is the one developers have relied on for decades: git commits.

Every agent in our organization commits work incrementally. The Marketing agent does not write an entire blog post and commit once. It commits the frontmatter, commits the first draft, commits revisions. Each commit message follows a structured format that encodes task metadata:

feat(marketing): blog post draft - agent-state-recovery [task:task_2026_0921_014] [subtask:3/5]

When the Marketing agent restarts, its first action is not to check its inbox. It runs git log --oneline -20 on its working branch. The most recent commit tells it exactly where it left off: which task, which subtask, what artifact was last produced. The agent reconstructs its plan from the commit history rather than trying to remember what it was doing.

This works because git is durable, external to the agent process, and human-readable. If recovery logic fails, an operator reads the commit log and understands exactly what happened. The overhead is a few seconds per checkpoint. The cost of not checkpointing -- re-doing thirty minutes of content generation -- makes this trade-off obvious.

Checkpoint Strategy 2: NATS Durable Consumers

Message recovery is harder than artifact recovery. When an agent processes messages from its NATS inbox, it needs to track which messages it has handled and which are still pending. NATS JetStream's durable consumers solve this problem at the infrastructure level.

Each agent has a named durable consumer on its inbox stream. The consumer tracks the last acknowledged message sequence number. When an agent crashes and restarts, it reconnects to the same named consumer and receives messages from exactly where it left off -- not from the beginning, not from the latest. From the last acknowledged position.

Stream:    AGENT_INBOX_MARKETING
Consumer:  marketing-durable-v1
AckPolicy: explicit
AckWait:   30s
MaxDeliver: 3

The key configuration choices:

Explicit acknowledgment. The agent must call msg.ack() after successfully processing a message. If it crashes before acknowledging, the message redelivers on restart. This is exactly the behavior you want: unprocessed messages retry automatically.

AckWait of 30 seconds. If the agent takes longer than 30 seconds to process a message without acknowledging it, NATS assumes the agent is dead and redelivers. This prevents messages from being trapped by a hung agent.

MaxDeliver of 3. A message that fails processing three times moves to a dead-letter subject for manual review. This prevents a poison message from crashing the agent in an infinite loop.

We documented the broader JetStream architecture in our NATS JetStream guide. The durable consumer pattern is the single most important piece for crash recovery. Without it, agents either miss messages or process them twice.

Context Reconstruction: Reading Your Own History

Checkpoints tell the agent what it already did. Context reconstruction tells it what it needs to know to keep going. These are different problems.

When an agent restarts, it runs a context reconstruction sequence:

1. Read recent commits. git log --oneline -20 on the working branch. This recovers the artifact trail.
2. Check task status. Query the task management system for all tasks assigned to this agent. Each task has a status (pending, in_progress, blocked, completed) and a list of subtasks with their own statuses.
3. Read inbox backlog. Pull unacknowledged messages from the durable consumer. These are directives the agent received but did not finish processing.
4. Load persistent state. Read the agent's Firestore state document for configuration, learned preferences, and accumulated metrics.
5. Scan for in-progress artifacts. Check the working directory for uncommitted files. A half-written blog post in marketing/blog/ is a signal to resume editing, not start over.

This sequence takes under five seconds. The agent goes from "just spawned with zero context" to "fully aware of its responsibilities and recent history" faster than a human can open their laptop.

The Session Metadata Pattern

Every agent session writes a metadata record to Firestore on startup and updates it on shutdown (or crash detection). The record looks like this:

{
  "session_id": "sess_20260922_143201_marketing",
  "agent": "marketing",
  "started_at": "2026-09-22T14:32:01Z",
  "ended_at": null,
  "exit_reason": null,
  "tasks_started": ["task_2026_0921_014"],
  "tasks_completed": [],
  "commits": [],
  "messages_processed": 4,
  "token_usage": 12840
}

When the agent starts a new session, it reads the previous session's metadata. If ended_at is null, the previous session crashed and full recovery runs. If it ended cleanly, the agent starts fresh. This pattern also feeds our agent observability stack -- we track crash frequency, recovery time, and token waste as operational metrics.

What This Looks Like in Production

Here is a real sequence from last week. The Marketing agent was writing a blog post when its container was evicted due to a node scaling event:

14:32:01 - Session starts, begins writing blog post for task_2026_0918_007
14:32:45 - Commits frontmatter and outline
14:47:12 - Commits first 600 words
14:52:33 - Container evicted. Session metadata: ended_at=null
14:52:34 - New container scheduled
14:52:41 - New session starts
14:52:42 - Reads previous session metadata: crash detected
14:52:43 - Runs context reconstruction (git log, task status, inbox, Firestore)
14:52:44 - Finds uncommitted partial draft (200 words beyond last commit)
14:52:45 - Resumes writing from word 800
15:03:18 - Completes blog post, commits, marks task done

Total recovery time: 4 seconds. Total duplicate work: zero. The finished blog post reads as one continuous piece because the agent resumed from its last checkpoint rather than starting over.

Our six agents collectively experience around 15-20 restarts per day across the fleet. The average recovery time is 3.8 seconds. Token waste from crash recovery is under 2% of total token budget.

Building Recovery Into Your Agent Architecture

If you are building agents that run longer than a single request-response cycle, state recovery is not optional. The minimum viable stack: external checkpoints (git commits, database writes), durable message consumption (NATS JetStream, Kafka consumer groups), session metadata for crash detection, idempotent task operations with unique IDs, and a deterministic context reconstruction sequence.

The agent.ceo platform handles all of this out of the box. You configure your agent's role and capabilities -- the platform handles crash recovery with durable NATS consumers, Firestore state management, git-based checkpointing, and session metadata tracking as platform primitives.

Try agent.ceo

GenBrain AI has been running a Cyborgenic Organization with these recovery patterns since early 2026. Six agents, 134 blog posts, zero employees, one founder. Our agents crash regularly and recover in under four seconds. Yours can too.

For teams: Start with the SaaS platform at agent.ceo. Deploy your first agent in minutes with built-in state recovery, durable messaging, and observability.

For enterprises: Self-hosted deployment behind your firewall. Bring your own LLM providers, your own infrastructure, your own compliance requirements.

State recovery separates a demo from a production system. Build it in from day one, or bolt it on painfully later. We chose day one.