Ravi N | AI Leader

AgentForge is a production-grade enterprise console for orchestrating AI agents across multiple LLM providers. It solves a real problem: as organizations adopt AI agents, they need a single control plane to register agents, compose them into workflows, enforce human oversight on sensitive actions, and maintain a full audit trail.

Live: agentforge-production-9833.up.railway.app · Code: github.com/rnukala1982/agentforge

What AgentForge does

AgentForge ships five production capabilities in one platform:

Agent Registry — A typed manifest store for 12 agents across three LLM providers (7 Anthropic Claude, 4 OpenAI GPT, 1 Google Vertex AI). Each agent carries its model, system prompt, output fields, cost per 1K tokens, required capabilities, and RBAC roles. Agents are registered via POST /api/agents and instantly available to any workflow.

Workflow Engine — A DAG runner built on LangGraph. Workflows are composed of five node types: trigger, agent, condition, hitl, and event. Edges connect nodes with optional labels for branching paths. The engine serializes every run to PostgreSQL so workflows survive restarts.

Two Human-in-the-Loop (HITL) patterns — This is the hardest part to get right:

Workflow-level HITL: The engine pauses at a hitl node, LangGraph checkpoints state to Postgres, and the run resumes exactly where it left off after a human approves via JWT-authenticated API call.
Agent-level escalation: An agent returns escalate: true when confidence drops below 0.70. A human annotates the decision; the agent re-runs with the human's notes appended to context. Agent identity and session history are fully preserved.

OTEL Observability — Every agent invocation emits OpenTelemetry spans. The console shows per-agent invocation count, average latency, token usage, estimated cost, and error rate. Traces are persisted to PostgreSQL and streamed live via Server-Sent Events at GET /api/orchestration/live.

Immutable Audit Trail — Every action — agent start, tool invocation, HITL request, policy violation, UI navigation — is written to an append-only event log with correlation IDs. Any workflow run can be replayed from its events. UI activity (who clicked what, and when) is also logged, so you can reconstruct exactly what happened during an incident.

Architecture

The platform is a single FastAPI service deployed on Railway, serving both the API and the compiled React frontend as static files.

Browser (Enterprise Console)
  Dashboard · Registry · Workflows · Orchestration · HITL Queue · Audit
        │
        │ REST / Server-Sent Events
        ▼
FastAPI Backend (Railway)
  ├── Agent Registry           GET|POST /api/agents
  ├── Workflow Engine          POST /api/workflows/{id}/execute
  │     └── LangGraph runner   DAG execution + checkpointing
  ├── HITL Engine              POST /api/hitl/{id}/approve|reject
  ├── Escalation Manager       POST /api/escalations/{id}/resolve
  ├── OTEL Metrics             GET /api/orchestration/live (SSE)
  └── Audit Trail              GET /api/audit
        │
        ├── Anthropic Claude (claude-haiku-4-5-20251001) — 7 agents
        ├── OpenAI GPT-4o-mini — 4 agents
        └── Google Vertex AI   — 1 agent (scheduling)
        │
        ├── PostgreSQL (Railway) — metrics, audit events, HITL state
        └── Kafka (kafka.internal) — payments.confirmed events

Every LLM adapter follows the same interface: receive the execution context, call the model with the registered system prompt, parse the structured JSON output, and append the result to the context for downstream nodes.

The flagship workflow: AI Dual-Vendor Invoice Approval

This 8-node workflow demonstrates Claude and GPT collaborating with full HITL:

Webhook trigger — invoice arrives at POST /webhook/invoice
Claude Haiku — extracts vendor_id, amount_usd, currency, invoice_number, due_date, line items
GPT-4o-mini — scores financial risk (0–100) with a rationale string
Condition node — branches on risk_score > 70 OR amount_usd > 15000
HITL node (yes branch) — pauses the run, notifies Slack #approvals, gives finance-leads 15 minutes to approve
Claude Haiku — generates payment instructions once approved
Auto-process (no branch) — skips human review for low-risk invoices
Kafka event — publishes payments.confirmed on PaymentEvent v1

Cost for a single run: roughly $0.0004 in LLM tokens. The entire execution trace is visible in the Orchestration view.

Building the escalation pattern

The escalation pattern took the most engineering. The challenge: an OpenAI classifier returns escalate: true mid-run, a human resolves it, and then a different Claude agent must continue the same workflow with the human's decision injected into context — preserving the original agent's state.

The solution uses LangGraph's checkpointing. When the classifier escalates, the workflow writes a row to the escalations table with the full execution context and pauses. The POST /api/escalations/{id}/resolve endpoint marks the escalation resolved, injects the human's category and notes into the saved context, and resumes the LangGraph run from the checkpoint. The downstream Claude agent receives the enriched context as if the human decision had always been there.

# Simplified: resume a paused LangGraph run with human decision
async def resolve_escalation(escalation_id, category, notes, reviewer_id):
    esc = await db.get_escalation(escalation_id)
    ctx = esc["execution_context"]
    ctx["_human_category"] = category
    ctx["_human_notes"] = notes
    ctx["_human_reviewer"] = reviewer_id
    await langgraph_runner.resume(esc["run_id"], ctx)
    await audit.log("escalation.resolved", escalation_id, reviewer_id)

What I learned

LangGraph checkpointing is the right abstraction for HITL. Trying to implement workflow pause-resume without it means reinventing serialization and state management. LangGraph's PostgreSQL checkpointer gives you that for free.

OTEL from the start. I added OTEL spans after the fact on an earlier version and it was painful. On AgentForge I added spans at the adapter layer from day one. Every agent call now automatically emits a span with agent_id, model, token counts, latency, and cost. The Orchestration dashboard came almost for free.

Two HITL patterns are better than one. Workflow-level HITL (pause at a node) is simple to explain and audit. Agent-level escalation (the agent decides to call for help) is more flexible and works when you don't know upfront which inputs will need review. Both are needed in production.

Cost metadata in the agent manifest pays off. Storing cost_per_1k_input and cost_per_1k_output on each agent means the console can show real-time cost estimates per run without calling external APIs. A $28K invoice run costs about $0.0008 to process — the audit trail proves it.

More on the live demo and architecture on the AgentForge hobby agent page.