· 21 min read

I Deleted 3,873 Lines at Midnight: How I Built a Local-First Multi-Agent DAG Engine in Go

When I first learned Go, I wanted to build a complete project in the language so that I could learn it properly while building something useful. I wanted to build the entire system from scratch, and figure things out along the way.

Around the same time, I was exploring LLM agents and how they could be used to automate tasks by breaking them down into smaller steps.

So, I decided to build my own LLM Orchestrator from scratch in Go, learning and figuring things out as I went.

Before I get into how I designed the engine and the challenges I ran into, here is the core idea of the system in simple terms:

flowchart TB
    subgraph DAG["DAG Compiler & Validator (Go)"]
        direction TB
        SPEC["Workflow Definition
YAML / Visual Canvas Nodes"]
        CYCLE["DFS Cycle Detector
Zero Recursion / Safe DAG"]
        KAHN["Kahn's Topo Sorter
Level Staging: Stage 0 → N"]

        SPEC --> CYCLE --> KAHN
    end

    subgraph EXEC["Stage-Level Concurrent Runner"]
        direction TB
        SEM["Worker Semaphore Pool
Bounded Goroutine Limiter"]
        A1["Agent: Researcher
Gathers Context & Web Data"]
        A2["Agent: Coder
Generates Structured Artifacts"]
        A3["Agent: Reviewer
Validates & Summarizes"]
        INTERP["Upstream Output Pipes
Dynamic Prompt Interpolation"]

        SEM --> A1 & A2
        A1 & A2 --> INTERP --> A3
    end

    subgraph STATE["State & Checkpointing Layer"]
        direction TB
        CHECK["Stage Checkpoint Commit
Atomic Disk Persistence"]
        RECOVER["Failure Recovery Manager
Resume from Last Clean State"]

        CHECK -. "On Crash" .-> RECOVER
    end

    subgraph TELEMETRY["Real-Time UI Observability"]
        direction TB
        WS["WebSocket Event Hub
Broadcasts Node Status & Tokens"]
        CANVAS["Interactive Visual Builder
Live Node Glow & Execution Traces"]

        WS --> CANVAS
    end

    %% End-to-End Pipeline Connections
    KAHN ==> SEM
    A3 --> CHECK
    SEM -- "State Updates" --> WS
    CHECK -- "State Saved" --> WS
    CANVAS -. "Edit / Run" .-> SPEC

    %% Excalidraw Color Styles
    style DAG fill:#EFF6FF,stroke:#3B82F6,stroke-width:2px,color:#1E3A8A
    style EXEC fill:#FFFBEB,stroke:#F59E0B,stroke-width:2px,color:#78350F
    style STATE fill:#FDF2F8,stroke:#EC4899,stroke-width:2px,color:#831843
    style TELEMETRY fill:#ECFDF5,stroke:#10B981,stroke-width:2px,color:#064E3B

    style SPEC fill:#DBEAFE,stroke:#2563EB,color:#1E40AF
    style CYCLE fill:#DBEAFE,stroke:#2563EB,color:#1E40AF
    style KAHN fill:#DBEAFE,stroke:#2563EB,color:#1E40AF

    style SEM fill:#FEF3C7,stroke:#D97706,color:#92400E
    style A1 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style A2 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style A3 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style INTERP fill:#FEF3C7,stroke:#D97706,color:#92400E

    style CHECK fill:#FCE7F3,stroke:#DB2777,color:#831843
    style RECOVER fill:#FCE7F3,stroke:#DB2777,color:#831843

    style WS fill:#D1FAE5,stroke:#059669,color:#065F46
    style CANVAS fill:#D1FAE5,stroke:#059669,color:#065F46

A workflow is defined as a Directed Acyclic Graph (DAG) of discrete tasks. Tasks have explicit dependencies, assigned agent personas, and input prompts. The Go backend validates the graph, computes the optimal parallel execution stages using topological sorting, executes tasks concurrently across worker pools, feeds upstream outputs directly into downstream prompts, and streams real-time state changes over WebSockets to an interactive visual builder.

Building it sounded straightforward on paper. In practice, getting it to run reliably without race conditions, data corruption, or memory leaks turned into one of the most intense engineering challenges I’ve tackled.

The Midnight Meltdown & The Architecture Pivot

My journey with this codebase did not start smoothly.

When I first sat down to write the engine, I fell into the classic developer trap: premature optimization and architecture astronauting. I thought, “If I’m building an orchestrator, it has to be a horizontally scalable distributed system from day one.”

I started coding a distributed architecture with Redis-based leader election, distributed leases, heartbeat daemons, and cluster node synchronization. I spent days wiring together lease renewal loops, Redis pub/sub coordination, and split-brain recovery logic.

Then came the night of May 12th. I was staring at nearly 4,000 lines of code across 36 files. The orchestrator couldn’t even execute a basic two-node task reliably because the distributed state machine was constantly deadlocking or dropping leases during local tests. The abstractions were getting in the way of the actual product.

At 11:58 PM, I hit a breaking point. I stopped everything, took a deep breath, and ran a hard reset.

The Git log still has the commit:

commit a8f92f842430c5b0107bdd1cde8ce2f5c8495857
Author: AniruthKarthik <aniruthkarthik10@gmail.com>
Date:   Tue May 12 23:58:28 2026 +0530

    I MESSED UP THE FLOW, DOING AGAIN FROM SCRATCH
 36 files changed, 82 insertions(+), 3873 deletions(-)

I literally deleted 3,873 lines of code in a single command. I stripped out every single piece of distributed infrastructure, wiped the queues and dead-letter topics, and started over with just a clean, single struct: internal/core/task.go.

A few days later, after experimenting with Redis coordination one last time, I made the official architectural pivot in commit e907376:

commit e907376ca5f9420d49599cd72a30ab0f7baa32ea
Author: AniruthKarthik <aniruthkarthik10@gmail.com>
Date:   Mon May 18 10:25:44 2026 +0530

    pivot: move to single-node local architecture
 14 files changed, 82 insertions(+), 509 deletions(-)

I scrapped the distributed coordinator completely and refocused on building a local-first execution runtime.

Why Go? Because for a local runtime, Go is an absolute superpower:

  1. Goroutines and Channels: I can spawn thousands of concurrent tasks without worrying about thread overhead or Python GIL lockups.
  2. Predictable Memory and Latency: No heavy JVM warmup or Python garbage collector pauses choking the UI stream.
  3. Single Binary Distribution: The entire server, API, and embedded SPA frontend compile into a single static binary. You run ./bin/orch and everything works out of the box.

Once I embraced the local-first philosophy, the system finally started taking shape.

Compiling Graphs: Topological Planning & Stage Concurrency

The heart of the orchestrator is transforming declarative workflow definitions into an execution plan. Whether a workflow is defined in YAML or created through the visual drag-and-drop canvas, the engine treats it as a directed acyclic graph where nodes represent tasks and directed edges represent dependencies.

Before executing a single task, the graph must pass strict static validation in internal/dag/validator.go:

  • Reference Integrity: Every dependency identifier must exist in the task map.
  • Self-Dependency Guards: A task cannot list itself as a dependency.
  • Acyclicity: The engine runs a Depth-First Search (DFS) with a recursion stack. If any node in the current traversal path is revisited, the engine rejects the graph immediately with an ErrCircularDependency before any LLM API is called.

Once validated, internal/dag/topo.go breaks the graph into discrete, parallelizable execution stages using Kahn’s algorithm:

func (p *TopologicalPlanner) BuildExecutionPlan(
    workflow *core.Workflow,
) (*ExecutionPlan, error) {
    if err := p.validator.Validate(workflow); err != nil {
        return nil, err
    }

    tasks := workflow.GetTasks()
    plan := NewExecutionPlan(workflow.ID)
    indegree := p.buildIndegreeMap(tasks)
    adjacency := p.buildAdjacencyList(tasks)
    processed := make(map[string]bool)

    level := 0
    for {
        currentLevelTasks := p.collectZeroIndegree(indegree, processed)
        if len(currentLevelTasks) == 0 {
            break
        }

        stage := ExecutionStage{
            Level:   level,
            TaskIDs: currentLevelTasks,
        }
        plan.AddStage(stage)

        for _, taskID := range currentLevelTasks {
            processed[taskID] = true
            p.decrementNeighbors(taskID, adjacency, indegree)
        }
        level++
    }

    return plan, nil
}

This staging model is key to how the orchestrator handles concurrency.

All tasks in stage[0] have an indegree of 0, meaning they have no dependencies and can run immediately in parallel. As tasks in stage[0] finish, the indegrees of their downstream neighbors are decremented. Once all dependencies for a downstream task are resolved, that task enters the next stage.

In the executor loop (internal/executor/executor.go), each stage is dispatched using a sync.WaitGroup bounded by a semaphore concurrency limiter:

var wg sync.WaitGroup
errChan := make(chan error, len(stage.TaskIDs))

for _, taskID := range stage.TaskIDs {
    task, _ := workflow.GetTask(taskID)
    if task.Status == core.TaskCompleted {
        continue
    }

    wg.Add(1)
    go func(t *core.Task) {
        defer wg.Done()
        if err := e.executeTask(taskCtx, execCtx, workflow, t); err != nil {
            errChan <- err
            if workflow.GetFailurePolicy() != core.FailurePolicyContinueOnFailure {
                workflowCancel()
            }
        }
    }(task)
}

wg.Wait()

After every successful stage, the orchestrator serializes the entire workflow state and persists a snapshot (workflow.CreateCheckpoint()). If the server crashes or an unexpected system reboot occurs, the supervisor can inspect the database, look up the latest checkpoint, and resume execution right from the start of the uncompleted stage instead of re-running the entire workflow and re-billing expensive API calls.

The Visual Client (With Candid Vibecoding Admission)

Now, to be completely transparent: I am really not great with frontend development.

My natural comfort zone is backend systems, concurrency, APIs, and databases. I didn’t want to spend three weeks wrestling with CSS layouts, SVG edge math, and state machines in React. But at the same time, trying to debug an agent DAG through raw terminal logs and JSON dumps is like flying blind. When five agents are running in parallel, you need to see which node is active, which edge is passing data, and where a failure occurred in real time.

So, I basically vibecoded the entire frontend using an LLM.

I used React Flow, TypeScript, and Vite, connecting it to the Go backend via WebSockets.

flowchart TB
    subgraph WORKFLOW["React Flow Visual Multi-Agent Workflow"]
        direction TB
        subgraph S1["Stage 0: Parallel Collection"]
            direction TB
            N1["Research Topic
Agent: Researcher
Status: COMPLETED"]
            N2["Competitor Scan
Agent: Scraper
Status: COMPLETED"]
        end

        subgraph S2["Stage 1: Verification"]
            direction TB
            N3["Fact Checker
Agent: Reviewer
Status: RUNNING"]
        end

        subgraph S3["Stage 2: Aggregation"]
            direction TB
            N4["Executive Summary
Agent: Writer
Status: PENDING"]
        end

        N1 --> N3
        N3 --> N4
        N2 --> N4
    end

    style WORKFLOW fill:#EFF6FF,stroke:#3B82F6,stroke-width:2px,color:#1E3A8A
    style S1 fill:#DBEAFE,stroke:#2563EB,color:#1E40AF
    style S2 fill:#FEF3C7,stroke:#D97706,color:#78350F
    style S3 fill:#FDF2F8,stroke:#EC4899,color:#831843

    style N1 fill:#DCFCE7,stroke:#16A34A,color:#14532D
    style N2 fill:#DCFCE7,stroke:#16A34A,color:#14532D
    style N3 fill:#FFFBEB,stroke:#F59E0B,color:#78350F
    style N4 fill:#F3F4F6,stroke:#9CA3AF,color:#374151

And because I vibecoded it in rapid bursts, the frontend initially had plenty of quirks and frustrating bugs that I had to track down:

  1. The Canvas Magnification Break: The workflow canvas would randomly collapse to zero height or scale unpredictably inside flex containers. I had to explicitly lock the builder wrapper height with style={{ height: 'calc(100vh - 3rem)' }} and clamp React Flow’s zoom boundaries (minZoom={0.2}, maxZoom={1.5}) in commits 290973b and f1bc5d6.
  2. The Handle Dot Rage: In standard React Flow, connecting two nodes requires clicking a tiny 6-pixel dot handle and dragging it precisely onto another 6-pixel handle. Trying to connect five tasks in a row was infuriating. In commit 9a1b0d4, I patched the canvas to allow dropping connection lines anywhere onto the destination node body.
  3. Off-Screen Spawning: Initially, clicking “Add Task” placed new nodes at hardcoded coordinates (0, 0) or directly on top of the previously selected node. You had to drag the node across the entire canvas every time you added one. In commit 7312ecc and d9971f9, I added viewport-aware node placement and dedicated “Add Child” buttons directly onto each task card.

It is far from a textbook frontend masterpiece, and there are still UI styling quirks I want to refine. But it gave me a live, interactive window into the simulation that made debugging the engine possible. I also recorded a walkthrough showing the visual DAG builder and live execution: LLM Orchestrator Demo Video.

The Context Problem: Feeding Upstream Results into Downstream Prompts

Once the DAG planner and executor were running, I hit an unexpected realization during end-to-end testing: my agents were running completely blind.

The DAG was ordering the tasks properly, the stages were executing concurrently, and the nodes were turning green in the UI. But when I inspected the generated output of downstream tasks, they had no clue what the upstream tasks had done! A “Summarize Report” task that depended on “Research Market Trends” was just generating a generic essay because it never received the output of the research task.

The execution engine was handling scheduling, but it was not handling data flow.

To solve this, I built a three-part context delivery pipeline in internal/executor/executor.go and internal/agents/agents.go:

flowchart TB
    subgraph STAGE["Upstream Completed Tasks"]
        direction LR
        U1["Market Research
JSON Output"]
        U2["Competitor Analysis
JSON Output"]
    end

    subgraph PIPELINE["Three-Part Context Delivery Pipeline"]
        direction TB
        subgraph STEP1["1. Direct Dependency Injection"]
            direction TB
            DEP["Collect dep_outputs Map
Injected into task.Input"]
        end

        subgraph STEP2["2. Template Interpolation"]
            direction TB
            INTERP["Regex Parser
Replaces {{Task.field}} in Prompt"]
        end

        subgraph STEP3["3. Context Stitching & Budgeting"]
            direction TB
            STITCH["ContextStitcher Safety Gate
Clamps to 32k Chars & Truncates"]
        end

        STEP1 --> STEP2 --> STEP3
    end

    subgraph AGENT["Downstream Agent Execution"]
        direction TB
        PROMPT["Structured System Prompt
+ Sanitized Context"]
        CALL["Model Provider API Call
Groq / OpenAI / Anthropic"]
        PROMPT --> CALL
    end

    STAGE ==> STEP1
    STEP3 ==> PROMPT

    style STAGE fill:#EFF6FF,stroke:#3B82F6,stroke-width:2px,color:#1E3A8A
    style PIPELINE fill:#FFFBEB,stroke:#F59E0B,stroke-width:2px,color:#78350F
    style AGENT fill:#ECFDF5,stroke:#10B981,stroke-width:2px,color:#064E3B

    style U1 fill:#DBEAFE,stroke:#2563EB,color:#1E40AF
    style U2 fill:#DBEAFE,stroke:#2563EB,color:#1E40AF

    style STEP1 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style STEP2 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style STEP3 fill:#FEF3C7,stroke:#D97706,color:#92400E
    style DEP fill:#FFFBEB,stroke:#F59E0B,color:#78350F
    style INTERP fill:#FFFBEB,stroke:#F59E0B,color:#78350F
    style STITCH fill:#FFFBEB,stroke:#F59E0B,color:#78350F

    style PROMPT fill:#D1FAE5,stroke:#059669,color:#065F46
    style CALL fill:#D1FAE5,stroke:#059669,color:#065F46

1. Direct Dependency Injection (dep_outputs)

In internal/executor/executor.go, right before a task is handed to a worker or agent, the executor pulls the outputs of all direct dependency tasks and packs them into the task’s input payload:

if len(task.Dependencies) > 0 {
    if task.Input == nil {
        task.Input = make(map[string]any)
    }
    depOutputs := make(map[string]any, len(task.Dependencies))
    for _, depID := range task.Dependencies {
        depTask, err := workflow.GetTask(depID)
        if err != nil {
            continue
        }
        depOut := depTask.GetOutput()
        if len(depOut) == 0 {
            continue
        }
        depOutputs[depID] = depOut
        depOutputs[depTask.Name] = depOut
    }
    if len(depOutputs) > 0 {
        task.Input["dep_outputs"] = depOutputs
    }
}

In internal/agents/agents.go, the AgentExecutor inspects dep_outputs and automatically constructs a structured markdown section in the agent’s system prompt:

## Outputs from upstream tasks you depend on
The following are the COMPLETE outputs of the tasks that ran before you. 
Use them as the primary context and input for your work.

### Task: Market Research
{ ... full JSON response ... }

2. Regex-Based Template Interpolation

Sometimes an agent prompt needs fine-grained access to a specific field from an upstream task rather than dumping the entire JSON blob. I implemented regex-based interpolation allowing users to write prompts like:

Analyze the following competitors mentioned in {{Market Research.competitors}} 
and draft a pricing comparison against my product.

The agent compiler parses {{TaskName}} to inject the entire artifact or {{TaskName.field}} to inject specific keys directly into the user prompt.

3. Context Stitching & Token Budgeting

If an upstream task produces a 40-page report, dumping five such artifacts into a downstream task will blow past model context limits or spike API costs unnecessarily.

In internal/agents/stitching.go, I introduced ContextStitcher. It acts as a safety gate, monitoring cumulative characters and token estimates. If artifacts exceed the configured budget (e.g. 32,000 characters), it stitches together the most relevant direct dependency artifacts and cleanly truncates or omits peripheral data with an explicit marker: [FURTHER ARTIFACTS OMITTED DUE TO CONTEXT LIMITS].

Real-Time Observability & The Bounded Event Bus

An orchestrator without real-time observability is impossible to operate. I didn’t want the frontend polling the REST API every 500 milliseconds—it creates unnecessary server load, introduces UI lag, and makes state transitions feel jittery.

Instead, I built an asynchronous, pub/sub event bus (internal/events/bus.go).

type EventBus struct {
    subscribers map[EventType][]subscription
    mutex       sync.RWMutex
    eventChan   chan Event
    workerCount int
    nextID      SubscriptionID
}

The bus decouples the core execution loop from telemetry. When a task starts, completes, retries, or records token usage, the executor fires an event to a buffered channel:

func (b *EventBus) Publish(event Event) {
    select {
    case b.eventChan <- event:
    default:
        // Channel full — drop event to avoid blocking callers in high-load scenarios.
    }
}

A dedicated worker pool reads from eventChan and dispatches events to registered subscribers:

  • The Audit Logger: Persists immutable records of every task transition and LLM interaction.
  • The Token Tracker: Accumulates running prompt, completion, and total token counts per model in memory.
  • The WebSocket Hub: Broadcasts events directly to connected browser clients.

In the frontend, useWebSocket.ts listens for incoming events and instantly turns node borders blue on TaskStarted, green on TaskCompleted, or red on TaskFailed, while streaming live execution logs into the slide-over drawer.

The Forensic Debugging Journey (The Nightmare Bugs)

Engineering posts that only show the clean, finished product always feel dishonest to me. The real work of building systems is spent banging your head against obscure bugs, strange edge cases, and unexpected runtime breakdowns.

Here is an honest breakdown of the five biggest nightmare bugs that broke my system during development:

+=============================================================================+
|                      THE FORENSIC DEBUGGING AUDIT LOG                       |
+---+----------------------------+--------------------------------------------+
| # | BUG CLASSIFICATION         | ROOT CAUSE & ARCHITECTURAL RESOLUTION      |
+---+----------------------------+--------------------------------------------+
| 1 | Concurrent Map Mutation    | Go shallow-copying map[string]any pointers |
|   | & Store Data Race          | Fix: Recursive DeepCopyMap & DeepCopySlice |
+---+----------------------------+--------------------------------------------+
| 2 | Silent WebSocket Hijack    | Logging middleware hiding http.Hijacker    |
|   | Interface Drop             | Fix: Explicit Hijack() on responseWriter   |
+---+----------------------------+--------------------------------------------+
| 3 | Sibling Zombie Tasks &     | Stage-only cancel leaking background tasks |
|   | Unbounded Token Bleed      | Fix: Cascading workflowCtx cancellation    |
+---+----------------------------+--------------------------------------------+
| 4 | PascalCase JSON Amnesia    | Missing struct tags + server binary cache  |
|   | & The Phantom Fix          | Fix: camelCase tags + binary rebuild       |
+---+----------------------------+--------------------------------------------+
| 5 | Re-execution Stale Trap    | Re-runs skipping COMPLETED task records    |
|   | & Zombie Outputs           | Fix: Explicit PENDING reset & error wiping |
+---+----------------------------+--------------------------------------------+

Bug 1: The Concurrent Map Mutation & Data Race

  • The Symptom: Under parallel load, tasks in the same execution stage would randomly fail with strange type errors, or worse, one task’s input payload would mysteriously appear inside a completely unrelated task’s input.
  • The Root Cause: In Go, maps and slices are reference types. When converting tasks from the domain model to storage records (TaskToRecord), I originally wrote a simple loop:
    // THE FLAWED CODE
input := make(map[string]any)
for k, v := range task.Input {
    input[k] = v
}
    This is a shallow copy. If v is a nested map[string]any or a slice of JSON parameters (which is true for almost all agent payloads), both the in-memory task and the store record pointed to the exact same underlying memory block. When two goroutines executed concurrently in the same stage and mutated their inputs, they triggered concurrent map read/write panics and cross-task state corruption.
  • The Fix: In commit c8885f1, I wrote a recursive deep-copy engine in internal/core/utils.go:
    func DeepCopyValue(v any) any {
    switch val := v.(type) {
    case map[string]any:
        return DeepCopyMap(val)
    case []any:
        return DeepCopySlice(val)
    case []string:
        return DeepCopyStringSlice(val)
    default:
        return v
    }
}
    I enforced DeepCopyMap and DeepCopySlice across all task constructors, store saves, and checkpoint captures, ensuring total memory isolation across parallel goroutines.

Bug 2: The Silent WebSocket Hijacker Drop

  • The Symptom: The backend compiled cleanly, the HTTP REST endpoints worked, but the browser could never establish a WebSocket connection. The client sat in a perpetual reconnect loop, and the server logged:
    WebSocket upgrade error: webserver doesn't support hijacking
  • The Root Cause: In internal/api/server.go, I had implemented custom HTTP middleware to log request latency and status codes. To capture the HTTP status code, I wrapped Go’s standard http.ResponseWriter in a custom struct:
    type responseWriter struct {
    http.ResponseWriter
    statusCode int
}
    In Go, WebSocket upgrading (upgrader.Upgrade) works by type-asserting the incoming http.ResponseWriter to http.Hijacker in order to take over the raw underlying TCP connection. Because my responseWriter struct wrapped http.ResponseWriter without explicitly implementing Hijack(), the type assertion failed, and Gorilla WebSocket threw an error every single time.
  • The Fix: In commit aaa4bdb, I explicitly forwarded the Hijack method on the custom wrapper:
    func (rw *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
    hj, ok := rw.ResponseWriter.(http.Hijacker)
    if !ok {
        return nil, nil, fmt.Errorf("webserver doesn't support hijacking")
    }
    return hj.Hijack()
}
    The moment this was added, the WebSocket connection upgraded immediately and live DAG updates began flowing to the UI.

Bug 3: Sibling Zombie Tasks & Unbounded Token Bleed

  • The Symptom: If a task in a parallel stage failed due to an invalid prompt or model error, the workflow was marked as failed, but my API logs showed that other tasks in that stage continued executing in the background, making calls to Anthropic and OpenAI and burning API credits.
  • The Root Cause: In internal/executor/executor.go, I was instantiating a context with context.WithCancel for individual stages (stageCancel). If a task failed in a fail-fast workflow, calling stageCancel() only signaled sibling tasks in that immediate loop, but did not cleanly propagate cancellation upstream or cancel active HTTP client request bodies.
  • The Fix: In commit ba2c4e0, I introduced a top-level workflowCtx tied to the entire workflow lifecycle:
    workflowCtx, workflowCancel := context.WithCancel(context.Background())
defer workflowCancel()
    If any task fails under the FailFast policy, workflowCancel() is called immediately. Because workflowCtx is passed directly into provider HTTP clients, all active HTTP connections to OpenAI, Anthropic, or Groq are aborted within milliseconds, cutting off unnecessary token spend instantly.

Bug 4: PascalCase JSON Amnesia & The Phantom Fix

  • The Symptom: The workflow builder UI loaded, but the workflow table was completely blank. No workflows showed up, task counts displayed as zero, and updating a task node crashed the UI with undefined property errors.

  • The Root Cause: In internal/store/models.go, my database record structs (WorkflowRecord, TaskRecord) were defined without JSON tags:

    type WorkflowRecord struct {
    ID          string
    Name        string
    Description string
    Status      string
}

    Go’s standard encoding/json serializes untagged struct fields using their exact Go names (PascalCase: ID, Name, Status). But the React/TypeScript frontend expected standard camelCase (id, name, status). Because the JSON keys didn’t match, TypeScript silently set fields to undefined.

    To make it more embarrassing, in commit 2e74563 I added the missing json:"..." tags, refreshed my browser, and… the UI was still completely broken! I spent nearly an hour double-checking TypeScript interfaces before realizing: I had edited the Go code but forgot to recompile and restart the backend server binary! The old binary with PascalCase JSON was still happily serving traffic on port 8080.

  • The Fix: In commit 620e536, I rebuilt the binary, restarted the service, and added explicit json:"..." struct tags across all models.

Bug 5: The Re-Execution Stale State Trap

  • The Symptom: Clicking “Execute” on an existing workflow that had previously completed or failed resulted in an instant no-op. The UI showed no new logs, tasks never transitioned back to running, and old output artifacts remained frozen.
  • The Root Cause: In internal/api/server.go, the execute endpoint loaded the workflow and its task records from the database and passed them directly to the executor. But because the records in the database still had Status: "COMPLETED", the executor loop checked:
    if task.Status == core.TaskCompleted {
    continue
}
    and skipped every single task! The engine assumed the work was already finished.
  • The Fix: In commit b6ef25a, I updated handleExecuteWorkflow to explicitly reset execution state before launching the executor:
    workflow.Status = core.WorkflowPending
workflow.StartedAt = nil
workflow.FinishedAt = nil
for _, task := range workflow.Tasks {
    task.Status = core.TaskPending
    task.StartedAt = nil
    task.FinishedAt = nil
    task.Output = nil
    task.Error = ""
}
    This guaranteed that every re-run starts from a clean slate.

Honest Evaluation: What Worked, What Failed, and Real Benchmarks

When building an engine like this, it is easy to accumulate features that sound great on paper but turn out to be clunky or unnecessary in practice. Being honest about what didn’t work was just as important as writing the core code.

1. Removing the Fake Dry-Run Mode

At one point, I added a “Dry Run” toggle in the UI and backend (FUTURE_FEAT.md). The idea was to allow users to simulate workflow execution without making actual LLM API calls.

In practice, the implementation in internal/agents/agents.go was returning hardcoded stub text:

// What dry run was doing:
return map[string]any{
    "status":  "simulated",
    "output":  fmt.Sprintf("This is a simulated response for task: %s", task.Name),
}

This turned out to be actively misleading. Because downstream tasks rely on upstream outputs for context and schema validation, passing fake stub text caused downstream tasks to fail or behave completely differently than they would with real LLM responses. It wasn’t a true simulation; it was just a fake path that cluttered the code. In commit 7c3455e, I ripped the dry-run feature out completely.

2. Disabling the “Auto-Route” Heuristic

Early on, the UI included an “Auto-Route (Recommended)” option in the provider dropdown. The vision was that the orchestrator would inspect the task description and automatically pick the optimal model based on cost and reasoning requirements.

In reality, the logic in internal/executor/router.go was just a naive heuristic that looked for any registered executor agent. If you didn’t have an exact agent role configured, it picked whatever model was available or threw an error. Calling it “Auto-Route” gave users a false sense of intelligence. In commit 23ced85, I disabled the auto-route option in the UI and required explicit provider and model selection. Pragmatic determinism is always better than half-baked magic.

3. Concurrency Stress Benchmarks

To verify that the engine could handle high throughput without race conditions, I built an end-to-end concurrency stress test in internal/executor/stress_test.go:

func TestConcurrencyStress(t *testing.T) {
    s := store.NewMemoryStore()
    eb := events.NewEventBus(100)
    // ... setup dependencies ...
    exec := NewExecutor(wr, ar, art, mem, tr, tp, eb, s)
    exec.WithConcurrencyLimit(50)

    var wg sync.WaitGroup
    numWorkflows := 20

    for i := 0; i < numWorkflows; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()
            // Spawn workflow with 5 dependent tasks
            // ...
            _ = exec.Execute(workflow)
        }(i)
    }
    wg.Wait()
}

Running this test across 20 concurrent workflows (100 total tasks executing topological planning, stage dispatching, and state persistence) completed in 0.06 seconds:

=== RUN   TestConcurrencyStress
--- PASS: TestConcurrencyStress (0.06s)
=== RUN   TestSupervisor_CheckStuckTasks
--- PASS: TestSupervisor_CheckStuckTasks (0.00s)
=== RUN   TestOutputValidationMiddleware
--- PASS: TestOutputValidationMiddleware (0.00s)
PASS
ok      github.com/AniruthKarthik/llm-orchestrator/internal/executor    0.062s

Because memory isolation is strictly guaranteed by recursive deep copying and the topological stages are grouped via Kahn’s algorithm, the Go runtime dispatches parallel tasks with microsecond overhead.

Lessons learned and What’s Next

Building LLM Orchestrator taught me several lessons about agent architectures that I wouldn’t have understood by just reading documentation:

  1. Local-First Beats Premature Distributed Complexity: Starting with Redis coordination and distributed leases was a mistake. Most agent pipelines don’t need multi-node consensus; they need ultra-fast local execution, instant feedback, and clean state persistence. Stripping out the distributed layer made the system 10x more reliable and easier to develop.
  2. Concurrency Requires Absolute Memory Isolation: In Go, shallow-copying maps of arbitrary JSON data (map[string]any) is an open invitation for data races. If multiple goroutines operate on task inputs or outputs, recursive deep copies are mandatory.
  3. Context Injection Is the Real Bottleneck: Scheduling tasks in a DAG is relatively easy. The hard part is ensuring that downstream agents receive the exact right context from upstream tasks without exceeding token limits or blowing up API costs.
  4. Visual Debugging Is Essential: You cannot operate an agent system through terminal JSON logs alone. Even a vibecoded UI with rough edges is infinitely better than flying blind when tracking complex multi-agent workflows.

What’s Next

The engine is currently running stably, but there is still plenty of room for improvement:

  • Conditional Branching (If/Else Nodes): Enabling dynamic DAG execution where a task output can conditionally skip or activate downstream branches.
  • Vector DB & RAG Integration: Connecting ContextStitcher directly to a local vector store so long-running workflows can retrieve historical context on demand.
  • Polishing the Frontend: Refactoring the React Flow canvas to clean up remaining CSS quirks and improve mobile responsiveness.

If you are interested in looking through the code, checking out the architecture, or running the engine locally, the repository is open source on GitHub: AniruthKarthik/llm-orchestrator.