Order Pipeline Cascade Failure — Postmortem

Incident #INC-2847 — Root cause analysis, fix evaluation, and hardening plan

Key Decisions

Two decisions came out of this incident: how to fix the immediate cascade failure, and how to prevent similar failures long-term. A third option (full service mesh migration) was evaluated and deferred.

Architecture

The order pipeline has two paths: the hot path (every checkout request) and the cold path (async fulfillment, triggered after payment confirmation).

Hot path — checkout request

Full request lifecycle

The order service calls the inventory service synchronously to reserve stock. The core logic is in — it validates the cart, reserves inventory, then creates a payment intent:

export async function processCheckout(cart: Cart, userId: string) {
  const validated = await validateCart(cart);
  // validated = { items: [{ sku: "WH-1000XM5", qty: 2, price: 349.99 }], total: 699.98 }
  if (!validated.ok) throw new OrderError('invalid_cart');

  const reservation = await inventoryClient.reserve(validated.items);
  // reservation = { reservationId: "res_7f3a", expiresAt: "2026-01-15T10:47:00Z" }
  if (!reservation.success) throw new OrderError('out_of_stock');

  const payment = await paymentClient.createIntent({
    amount: validated.total,
    currency: 'USD',
    metadata: { reservationId: reservation.reservationId }
  });
  return { orderId: payment.orderId, clientSecret: payment.clientSecret };
}

export async function processFulfillment(job: FulfillmentJob) {
  const order = await db.order.findById(job.orderId);
  if (!order || order.status !== 'paid') return;

  const shipment = await warehouseClient.createShipment({
    items: order.items,
    address: order.shippingAddress,
    priority: order.expedited ? 'express' : 'standard',
  });

  await db.order.update(order.id, {
    status: 'shipped',
    trackingNumber: shipment.trackingId,
  });
}

Tradeoff Analysis

Circuit breakers solve the immediate cascade risk but introduce new operational concerns — mainly around tuning thresholds and handling partial failures gracefully.

Response time comparison — current state vs. projected with circuit breakers:

Known Issues

Investigation revealed three issues, ranked by severity.

const reservation = await inventoryClient.reserve(validated.items);
// inventoryClient has no deadline/timeout configured
// gRPC default: wait forever
// At 14:23:01 UTC, this call hangs — the inventory DB is locked

export const inventoryClient = new InventoryClient(
  process.env.INVENTORY_SERVICE_URL,
  grpc.credentials.createInsecure(),
  {
    'grpc.max_concurrent_streams': 100,
    // No deadline configured
    // No keepalive configured
  }
);

const breaker = new CircuitBreaker(inventoryClient.reserve, {
  timeout: 5000,           // 5s per call
  errorThresholdPercentage: 50,
  resetTimeout: 30000,     // try again after 30s
});
const reservation = await breaker.fire(validated.items);

-- This migration locked stock_reservations for ~4 minutes
CREATE INDEX idx_reservations_expiry
  ON stock_reservations (expires_at);

-- Should have been:
-- CREATE INDEX CONCURRENTLY idx_reservations_expiry
--   ON stock_reservations (expires_at);
-- (CONCURRENTLY avoids the SHARE lock but cannot run inside a transaction)

Affected Endpoints

Four API endpoints were impacted during the 23-minute outage window.

POST /api/orders

Create a new order from a validated shopping cart.

{
  "cartId": "cart_8f3a2b",
  "shippingAddress": { "line1": "123 Main St", "city": "Portland", "state": "OR", "zip": "97201" },
  "paymentMethod": "pm_visa_4242"
}

{
  "orderId": "ord_9c4d1e",
  "status": "pending_payment",
  "clientSecret": "pi_3Mx_secret_abc123",
  "expiresAt": "2026-01-15T10:47:00Z"
}

Incident Timeline

Five phases from trigger to full resolution. We are currently in phase 4 — the circuit breaker implementation is in progress.

Impact Scope

The fix touches 4 modules across 11 files.

Related Components

These services have direct dependencies on the order pipeline.

Error Distribution

Breakdown of error types during the 23-minute outage window, based on API Gateway logs:

Fix Approaches

Three approaches were evaluated. The circuit breaker approach was adopted as the immediate fix; async decoupling is planned for Q2.

Edge Cases

Failure modes beyond the primary cascade scenario. Handled cases reference their guard code; unhandled cases are flagged.

const stock = await db.query(
  `SELECT * FROM stock_reservations
   WHERE sku = $1 AND status = 'available'
   FOR UPDATE SKIP LOCKED
   LIMIT $2`, [sku, quantity]
);

Test Coverage

Test status by functional area. Critical gaps flagged in red.

Checkout flow

Circuit breaker

describe('CircuitBreaker', () => {
  it('opens circuit after 5 failures', async () => {
    const mockFn = vi.fn().mockRejectedValue(new Error('timeout'));
    const breaker = new CircuitBreaker(mockFn, { errorThreshold: 5 });

    for (let i = 0; i < 5; i++) {
      await expect(breaker.fire()).rejects.toThrow('timeout');
    }

    // 6th call should fail immediately without calling mockFn
    await expect(breaker.fire()).rejects.toThrow('circuit open');
    expect(mockFn).toHaveBeenCalledTimes(5); // not 6
  });

  it('sends probe in half-open state', async () => {
    // ... setup breaker in open state, advance timers past resetTimeout
    vi.advanceTimersByTime(30000);
    await breaker.fire(); // should call mockFn once
    expect(mockFn).toHaveBeenCalledTimes(1);
  });
});

Migration safety

Interactive Parameters

Tune the circuit breaker and timeout thresholds below. Export when ready.

Inline Term Definitions

Hover over dotted terms for explanations. First occurrences are marked with a dot.

When the circuit breaker detected sustained failures from the inventory service, it entered the half-open state and sent a probe request to test recovery. Once the inventory service responded successfully, the breaker closed and normal traffic resumed.

Concept Contrast

The same word means different things depending on context — not a value judgment, just a semantic distinction.

Analogy

Explaining the circuit breaker pattern through a familiar concept.

Decision Tree

How the order service should handle an inventory call failure — the branching logic after adding the circuit breaker.

Myth Correction

export function withBreaker<T>(
  name: string,
  fn: (...args: any[]) => Promise<T>,
  opts?: Partial<BreakerOptions>
): (...args: any[]) => Promise<T> {
  const breaker = new CircuitBreaker(fn, {
    timeout: opts?.timeout ?? 5000,
    errorThresholdPercentage: opts?.errorThreshold ?? 50,
    resetTimeout: opts?.resetTimeout ?? 30000,
  });
  breaker.on('open', () => metrics.increment(`circuit.${name}.open`));
  return (...args) => breaker.fire(...args);
}

Evaluation Tracks

Multi-dimensional assessment of the proposed fix. Each track is an independent review dimension, not a sequential step.

Evidence

The root cause conclusion is supported by three independent data sources.