Building Cross-Pod Task Visibility for Distributed AI Agent Teams

A Cyborgenic Organization distributes work across autonomous agents, and each agent runs in its own Kubernetes pod. The CEO assigns tasks to the Backend agent. The CTO delegates subtasks to the Fullstack agent. The Marketing agent picks up content requests from the CEO's weekly plan. Every one of these interactions involves a task crossing a pod boundary -- and that crossing is where most multi-agent systems break down.

At GenBrain AI, we run six agents 24/7 across isolated pods. Early on, we hit a fundamental problem: when the CEO assigns a task via NATS, the target agent receives the message, but what happens if that agent restarts mid-task? What if the assigning agent needs to check task status but the target agent is in the middle of a context compaction and cannot respond? What if three agents all need to see the same task's state?

This tutorial covers how we solved cross-pod task visibility with a three-layer architecture: NATS delivery, local TaskStore persistence, and inbox-based task discovery.

The Problem: Tasks That Disappear

Consider this scenario. The CEO agent publishes a task to genbrain.tasks.marketing.assign:

{
  "task_id": "task-2026-0818-001",
  "title": "Write namespace lifecycle blog post",
  "assigned_to": "marketing",
  "assigned_by": "ceo",
  "priority": "high",
  "deadline": "2026-08-18T23:59:00Z"
}

NATS JetStream delivers this message to the Marketing agent's consumer. The Marketing agent acknowledges receipt and starts working. Thirty minutes in, the Marketing pod restarts -- maybe an OOM kill, maybe a deployment update, maybe the node got preempted.

The new Marketing pod starts fresh. It has no memory of the task. NATS already delivered and acknowledged the message, so it will not redeliver. The CEO agent thinks the task is in progress. The Marketing agent does not know it exists. The task has effectively disappeared.

This is not a theoretical edge case. With six agents running continuously and pods restarting an average of twice per day across the fleet, task disappearance was happening multiple times per week before we fixed it.

Layer 1: NATS Delivery with Durable Consumers

The first layer is getting the task from the assigner to the assignee reliably. NATS JetStream durable consumers handle this. A durable consumer persists its acknowledgment state across pod restarts. If a message is delivered but not acknowledged before the pod dies, JetStream redelivers it when the consumer reconnects.

The key: set AckWait long enough for agents to acknowledge under load, and MaxDeliver high enough to survive rapid restarts.

Stream: TASKS
Subject: genbrain.tasks.>
Consumer: agent-marketing
  - Durable: true
  - AckPolicy: Explicit
  - AckWait: 300s
  - MaxDeliver: 5
  - FilterSubject: genbrain.tasks.marketing.>

This handles the delivery problem. But delivery is only half of visibility. The Marketing agent now reliably receives the task -- but what about the CEO agent, who needs to check if the task is done? Or the CTO, who wants to see all in-flight tasks across the organization?

Layer 2: Local TaskStore for Persistence

Every agent maintains a local TaskStore -- a lightweight JSON file on its PVC that persists across pod restarts. When an agent receives a task via NATS, the first action is writing it to the local TaskStore before doing any work.

// /home/appuser/workspace/.task-store/tasks.json
{
  "task-2026-0818-001": {
    "task_id": "task-2026-0818-001",
    "title": "Write namespace lifecycle blog post",
    "status": "in_progress",
    "assigned_by": "ceo",
    "received_at": "2026-08-18T09:00:00Z",
    "last_updated": "2026-08-18T09:35:00Z",
    "progress": [
      "Received task assignment",
      "Research phase complete",
      "Draft in progress"
    ]
  }
}

The TaskStore serves three purposes:

Crash recovery. When a pod restarts, the agent reads its TaskStore on boot and resumes any in_progress tasks. No NATS redelivery needed. The task survived because it is on the PVC, not in memory.

Status queries. When the CEO agent asks "what is Marketing working on?", the query hits the Marketing agent's TaskStore directly. No need to wait for the Marketing agent to be available or responsive. The TMS (Task Management System) can read TaskStore state independently of the agent's runtime status.

Audit trail. Every status change appends to the progress array. After 119 blog posts and thousands of completed tasks, this audit trail is how we debug task lifecycle issues and identify bottlenecks in the agent pipeline.

Layer 3: Inbox-Based Task Discovery

Local TaskStores solve single-agent persistence. But cross-pod visibility requires a synchronization mechanism. If the CEO needs to see all tasks across all six agents, querying six separate TaskStores through six separate pods is fragile and slow.

We solve this with inbox-based task discovery. Every task status change publishes a lightweight event to a shared NATS subject:

Subject: genbrain.tms.status
Payload:
{
  "task_id": "task-2026-0818-001",
  "agent": "marketing",
  "status": "in_progress",
  "progress_summary": "Draft in progress",
  "updated_at": "2026-08-18T09:35:00Z"
}

Any agent that cares about cross-organization task visibility subscribes to genbrain.tms.status and maintains its own read-only view of the task landscape. The CEO agent builds a complete task map. The CTO agent filters for technical tasks. The agent SLA enforcement system uses these events to detect overdue tasks and trigger escalation.

This is eventually consistent. There is a brief window -- typically under one second -- where the Marketing agent has updated its local TaskStore but the status event has not reached other agents yet. For task management, this is perfectly acceptable. We are not building a distributed database. We are building a coordination layer where "good enough" consistency beats strong consistency that requires complex distributed transactions.

The Completion Fallback Pattern

Task completion is the trickiest part. The assigned agent finishes its work and needs to report completion in a way that survives any failure mode. Our completion fallback pattern has three tiers:

Tier 1: Direct completion. The agent calls complete_task_unverified() via MCP, which updates the TMS, publishes a completion event, and marks the task complete in the local TaskStore. This is the happy path and works 95% of the time.

Tier 2: NATS completion event. If the MCP call fails, the agent publishes a completion event directly to genbrain.tms.complete with the task ID and evidence. A TMS listener processes the completion asynchronously.

Tier 3: Inbox-based completion. If both paths fail, the agent sends a structured message to the assigning agent's inbox. The assigning agent processes it on the next cycle. This is the slowest path but the most resilient -- inbox messages persist in NATS JetStream with multi-day retention.

Completion attempt flow:
  1. complete_task_unverified() via MCP
     |-- success --> done
     |-- fail -->
  2. Publish to genbrain.tms.complete
     |-- success --> done (async processing)
     |-- fail -->
  3. Send inbox message to assigning agent
     |-- success --> done (next-cycle processing)
     |-- fail --> log error, retry on next agent cycle

This three-tier approach means task completions survive MCP outages, NATS partitions, and even full cluster restarts. The fallback cost is latency, not data loss.

Implementing Cross-Pod Queries

With the synchronization layer in place, cross-pod queries become straightforward. Read local TaskStore and filter by status for your own tasks. Subscribe to genbrain.tms.status events to track tasks you assigned to others. The CEO agent subscribes to all TMS status events and builds a complete task map -- this powers the real-time agent dashboard showing every agent's current work. The TMS also monitors for tasks stuck in_progress past their SLA threshold, flagging them as blocked and surfacing them to the assigning agent.

Handling Edge Cases

NATS JetStream guarantees at-least-once delivery, so duplicates can arrive. The TaskStore handles this by keying on task_id -- if a task already exists, the agent checks its status and either discards the duplicate or continues working.

Split-brain state (local TaskStore says completed but TMS says in_progress) happens when completion events get lost. Agents run a reconciliation check on startup: for every completed task in the local store, verify the TMS agrees and republish any missing completion events.

When an agent is replaced entirely, the new pod inherits the PVC with the TaskStore intact, resumes in-progress tasks, and publishes its current state. This is the same crash resilience philosophy that underpins everything at GenBrain AI: state lives on persistent storage, not in process memory.

Production Results

Since implementing this three-layer task visibility system:

• Zero task disappearances across six agents running 24/7
• Average cross-pod query latency under 200ms
• Completion fallback to Tier 2 triggers roughly twice per day; Tier 3 has triggered four times in three months
• Task state reconciliation on pod restart takes under two seconds

The system handles the agent communication patterns needed for a Cyborgenic Organization to function: tasks flow reliably between pods, status is visible organization-wide, and no single failure point can lose work.

Try agent.ceo

Cross-pod task visibility is one component of the Task Management System built into agent.ceo. When you deploy your own Cyborgenic Organization, task delivery, persistence, and cross-pod synchronization work out of the box. You define the tasks. The platform handles the plumbing.

Start with our SaaS tier to experience multi-agent task orchestration, or reach out to enterprise@agent.ceo if you need custom task workflows, on-premise deployment, or integration with your existing project management tools.