Circuit Breaker: Stop Calling a Broken Service

When a weather service is down for an hour, every screen that asks it waits for its timeout, again and again, and retries only make the wait longer; a circuit breaker remembers the failures, stops calling for a while, and lets one trial call through to see whether the service is back.

Freelancer · Software Development · OOP · Updated

By

After this lesson you can

  • Stop calling a broken service for a while, then let one trial call through
  • Read a circuit breaker's state diagram: closed, open and half-open, and what moves it between them
  • Spot the smells that point to Circuit Breaker: every caller waiting on a service that's down

Key takeaways

  • The smell: every caller waits on a service that's down, one timeout after another, and nothing remembers that the last calls failed.
  • The fix: a breaker counts failures; after three in a row it opens and answers at once with a fallback, without calling.
  • After a while it goes half-open: one trial call. If it works, the breaker closes; if it fails, it opens again.
  • Closed, open and half-open are a state diagram, Pass 2's State pattern again: what a call does depends on the breaker's state.
  • Every language has a library for it: Polly (.NET), Resilience4j (Java), pybreaker (Python) and cockatiel (JavaScript and TypeScript).

Before this: Retry: Try Again, a Little Later, and State: What You Can Do Depends on Where It Is (Pass 2's state diagrams).

Last Tuesday, SkyCast was down for an hour, not a hiccup. Every time a screen in Maya's house asked for the weather, it waited five seconds for an answer that never came: the kitchen display, the phone app, the wall panel, again and again, all hour. With the last lesson's retries, each screen waited even longer, and every hub in town kept knocking on SkyCast's door while it was trying to get back up. Retry is for hiccups. For an outage, the hub needs something that notices and stops calling: a circuit breaker.

The symptom: everyone waits on a service that's down

To show an hour in a few lines, these examples use a stand-in clock that moves only when told to, and a stand-in SkyCast that, while it's down, uses up 5 seconds of that clock on every call (its timeout) before failing:

FakeClock clock = new FakeClock();
FakeSkyCast skyCast = new FakeSkyCast(clock);

foreach (string screen in new[] { "Kitchen display", "Phone app", "Wall panel", "Kitchen display" })
{
    int started = clock.Now;
    string weather;
    try
    {
        weather = skyCast.Weather();
    }
    catch (SkyCastDownException)
    {
        weather = "unavailable";
    }
    Console.WriteLine(quot;{screen}: {weather}, waited {clock.Now - started} s");
}
Console.WriteLine(quot;SkyCast was called {skyCast.Calls} times while it was down");

class SkyCastDownException : Exception { }

// Stands in for the real clock: time moves only when the example says so.
class FakeClock
{
    public int Now { get; private set; }
    public void Advance(int seconds) => Now += seconds;
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast
{
    private readonly FakeClock clock;
    public bool Down { get; set; } = true;
    public int Calls { get; private set; }
    public FakeSkyCast(FakeClock clock) { this.clock = clock; }

    public string Weather()
    {
        Calls++;
        if (Down)
        {
            clock.Advance(5); // SkyCast's timeout
            throw new SkyCastDownException();
        }
        clock.Advance(2); // slow, but an answer
        return "72F, sunny";
    }
}
Kitchen display: unavailable, waited 5 s
Phone app: unavailable, waited 5 s
Wall panel: unavailable, waited 5 s
Kitchen display: unavailable, waited 5 s
SkyCast was called 4 times while it was down
class SkyCastDown extends RuntimeException { }

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
    int now = 0;
    void advance(int seconds) { now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
    private final FakeClock clock;
    boolean down = true;
    int calls = 0;
    FakeSkyCast(FakeClock clock) { this.clock = clock; }

    String weather() {
        calls++;
        if (down) {
            clock.advance(5); // SkyCast's timeout
            throw new SkyCastDown();
        }
        clock.advance(2); // slow, but an answer
        return "72F, sunny";
    }
}

void main() {
    FakeClock clock = new FakeClock();
    FakeSkyCast skyCast = new FakeSkyCast(clock);

    for (String screen : List.of("Kitchen display", "Phone app", "Wall panel", "Kitchen display")) {
        int started = clock.now;
        String weather;
        try {
            weather = skyCast.weather();
        } catch (SkyCastDown down) {
            weather = "unavailable";
        }
        IO.println(screen + ": " + weather + ", waited " + (clock.now - started) + " s");
    }
    IO.println("SkyCast was called " + skyCast.calls + " times while it was down");
}
Kitchen display: unavailable, waited 5 s
Phone app: unavailable, waited 5 s
Wall panel: unavailable, waited 5 s
Kitchen display: unavailable, waited 5 s
SkyCast was called 4 times while it was down
class SkyCastDown(Exception):
    pass


class FakeClock:
    """Stands in for the real clock: time moves only when the example says so."""

    def __init__(self):
        self.now = 0

    def advance(self, seconds):
        self.now += seconds


class FakeSkyCast:
    """Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes."""

    def __init__(self, clock):
        self.clock = clock
        self.down = True
        self.calls = 0

    def weather(self):
        self.calls += 1
        if self.down:
            self.clock.advance(5)  # SkyCast's timeout
            raise SkyCastDown()
        self.clock.advance(2)  # slow, but an answer
        return "72F, sunny"


clock = FakeClock()
skycast = FakeSkyCast(clock)

for screen in ["Kitchen display", "Phone app", "Wall panel", "Kitchen display"]:
    started = clock.now
    try:
        weather = skycast.weather()
    except SkyCastDown:
        weather = "unavailable"
    print(f"{screen}: {weather}, waited {clock.now - started} s")
print(f"SkyCast was called {skycast.calls} times while it was down")
Kitchen display: unavailable, waited 5 s
Phone app: unavailable, waited 5 s
Wall panel: unavailable, waited 5 s
Kitchen display: unavailable, waited 5 s
SkyCast was called 4 times while it was down
class SkyCastDown extends Error {}

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
  now = 0;
  advance(seconds) { this.now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
  #clock;
  down = true;
  calls = 0;
  constructor(clock) { this.#clock = clock; }

  weather() {
    this.calls++;
    if (this.down) {
      this.#clock.advance(5); // SkyCast's timeout
      throw new SkyCastDown();
    }
    this.#clock.advance(2); // slow, but an answer
    return "72F, sunny";
  }
}

const clock = new FakeClock();
const skyCast = new FakeSkyCast(clock);

for (const screen of ["Kitchen display", "Phone app", "Wall panel", "Kitchen display"]) {
  const started = clock.now;
  let weather;
  try {
    weather = skyCast.weather();
  } catch (error) {
    if (!(error instanceof SkyCastDown)) throw error;
    weather = "unavailable";
  }
  console.log(`${screen}: ${weather}, waited ${clock.now - started} s`);
}
console.log(`SkyCast was called ${skyCast.calls} times while it was down`);
Kitchen display: unavailable, waited 5 s
Phone app: unavailable, waited 5 s
Wall panel: unavailable, waited 5 s
Kitchen display: unavailable, waited 5 s
SkyCast was called 4 times while it was down
class SkyCastDown extends Error {}

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
  now = 0;
  advance(seconds: number): void { this.now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
  #clock: FakeClock;
  down = true;
  calls = 0;
  constructor(clock: FakeClock) { this.#clock = clock; }

  weather(): string {
    this.calls++;
    if (this.down) {
      this.#clock.advance(5); // SkyCast's timeout
      throw new SkyCastDown();
    }
    this.#clock.advance(2); // slow, but an answer
    return "72F, sunny";
  }
}

const clock = new FakeClock();
const skyCast = new FakeSkyCast(clock);

for (const screen of ["Kitchen display", "Phone app", "Wall panel", "Kitchen display"]) {
  const started = clock.now;
  let weather: string;
  try {
    weather = skyCast.weather();
  } catch (error) {
    if (!(error instanceof SkyCastDown)) throw error;
    weather = "unavailable";
  }
  console.log(`${screen}: ${weather}, waited ${clock.now - started} s`);
}
console.log(`SkyCast was called ${skyCast.calls} times while it was down`);
Kitchen display: unavailable, waited 5 s
Phone app: unavailable, waited 5 s
Wall panel: unavailable, waited 5 s
Kitchen display: unavailable, waited 5 s
SkyCast was called 4 times while it was down

Four screens, four five-second freezes, four calls to a service that can't answer. Over an hour, that's every screen, every time. With three retries each, every briefing would make three calls and wait even longer.

  • The symptom: every caller waits the full timeout on a service that's down, one after another, and the service gets called the whole time it's trying to recover.
  • The cause: each call decides on its own. Nothing remembers that the last three calls failed, so the fourth one makes the same mistake.

What would you try?

More retries, or a longer timeout. Both make each screen wait longer, and both send SkyCast more calls while it's down. That's the wrong direction.

A "SkyCast is down" switch that someone flips by hand. It works, if someone is awake at 6:00 to flip it, and remembers to flip it back.

Ask SkyCast whether it's up before each call. That's another call to the same far-away service, and it fails the same way.

What we want is for the hub itself to notice "this keeps failing", stop calling for a while, and then check, carefully, whether things are better.

The fix: remember, and stop calling for a while

Your house already has one. Every circuit in the house runs through a switch in the breaker box. When a circuit draws far too much power, say a space heater and a hair dryer on one plug, the breaker trips: it cuts that circuit off before the wires overheat. The rest of the house keeps its lights. And it stays off until someone deals with it and flips it back, once, to see whether the problem is gone.

A house's breaker box with its door open: a column of switches, all pointing to ON except one in the middle, flipped to OFF, labeled Kitchen

A circuit breaker in code does the same for calls to another system. It sits between the hub and SkyCast, and it's always in one of three states:

  • Closed (the normal state, calls flow, like a closed circuit): every call goes through to SkyCast. The breaker counts failures in a row.
  • Open (tripped): after 3 failures in a row, the breaker stops calling. Every request gets the fallback, "unavailable", at once, without waiting and without bothering SkyCast.
  • Half-open (testing): after 30 seconds open, the breaker lets one trial call through. If it works, the breaker closes again. If it fails, the breaker opens for another 30 seconds.

These rules are a state diagram, the kind you drew for packages in Pass 2: rounded boxes for the states, arrows labeled with what moves the breaker between them:

A circuit breaker's state diagram: it starts closed, calling SkyCast and counting failures (a call that works counts back to 0). 3 failures in a row open it: it answers at once, without calling. After 30 s it is half-open and lets one trial call through: success closes it, failure opens it again

That makes the breaker Pass 2's State pattern again: what a call does depends on which box the breaker is in. In closed, it calls and counts; in open, it answers at once; in half-open, it calls once and decides.

Circuit Breaker's roles: the morning briefing asks the circuit breaker, which remembers failures and decides whether to call SkyCast, far away. When open, it answers "unavailable" at once. Its state, closed, open or half-open, sits in a dashed fence labeled "what changes: the breaker's state"

The classic version: a breaker with three states

The breaker is handed the clock (a function that says what time it is), so the example's stand-in clock can move time forward. It's handed the call to make and a fallback to give when it won't call:

FakeClock clock = new FakeClock();
FakeSkyCast skyCast = new FakeSkyCast(clock);
CircuitBreaker breaker = new CircuitBreaker(() => clock.Now, failuresToOpen: 3, openSeconds: 30);

void Briefing(string screen)
{
    int started = clock.Now;
    string weather = breaker.Call(skyCast.Weather, fallback: "unavailable");
    Console.WriteLine(quot;{screen}: {weather}, waited {clock.Now - started} s (breaker {breaker.State})");
}

Briefing("Kitchen display");
Briefing("Phone app");
Briefing("Wall panel");
Briefing("Kitchen display");
Console.WriteLine("30 s later, still down:");
clock.Advance(30);
Briefing("Phone app");
Console.WriteLine("30 s later, SkyCast is back:");
clock.Advance(30);
skyCast.Down = false;
Briefing("Wall panel");
Console.WriteLine(quot;SkyCast was called {skyCast.Calls} times in all");

class CircuitBreaker
{
    private readonly Func<int> now;
    private readonly int failuresToOpen;
    private readonly int openSeconds;
    private int failures;
    private int openedAt;
    public string State { get; private set; } = "closed";

    public CircuitBreaker(Func<int> now, int failuresToOpen, int openSeconds)
    {
        this.now = now;
        this.failuresToOpen = failuresToOpen;
        this.openSeconds = openSeconds;
    }

    public string Call(Func<string> action, string fallback)
    {
        if (State == "open")
        {
            if (now() - openedAt < openSeconds)
            {
                return fallback; // open: answer at once, without calling
            }
            State = "half-open"; // time's up: let this one call through as a trial
        }
        try
        {
            string result = action();
            State = "closed"; // it worked: back to normal
            failures = 0;
            return result;
        }
        catch (SkyCastDownException)
        {
            failures++;
            if (State == "half-open" || failures >= failuresToOpen)
            {
                State = "open"; // stop calling for a while
                openedAt = now();
            }
            return fallback;
        }
    }
}

class SkyCastDownException : Exception { }

// Stands in for the real clock: time moves only when the example says so.
class FakeClock
{
    public int Now { get; private set; }
    public void Advance(int seconds) => Now += seconds;
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast
{
    private readonly FakeClock clock;
    public bool Down { get; set; } = true;
    public int Calls { get; private set; }
    public FakeSkyCast(FakeClock clock) { this.clock = clock; }

    public string Weather()
    {
        Calls++;
        if (Down)
        {
            clock.Advance(5); // SkyCast's timeout
            throw new SkyCastDownException();
        }
        clock.Advance(2); // slow, but an answer
        return "72F, sunny";
    }
}
Kitchen display: unavailable, waited 5 s (breaker closed)
Phone app: unavailable, waited 5 s (breaker closed)
Wall panel: unavailable, waited 5 s (breaker open)
Kitchen display: unavailable, waited 0 s (breaker open)
30 s later, still down:
Phone app: unavailable, waited 5 s (breaker open)
30 s later, SkyCast is back:
Wall panel: 72F, sunny, waited 2 s (breaker closed)
SkyCast was called 5 times in all
class SkyCastDown extends RuntimeException { }

class CircuitBreaker {
    private final IntSupplier now;
    private final int failuresToOpen;
    private final int openSeconds;
    private int failures = 0;
    private int openedAt = 0;
    String state = "closed";

    CircuitBreaker(IntSupplier now, int failuresToOpen, int openSeconds) {
        this.now = now;
        this.failuresToOpen = failuresToOpen;
        this.openSeconds = openSeconds;
    }

    String call(Supplier<String> action, String fallback) {
        if (state.equals("open")) {
            if (now.getAsInt() - openedAt < openSeconds) {
                return fallback; // open: answer at once, without calling
            }
            state = "half-open"; // time's up: let this one call through as a trial
        }
        try {
            String result = action.get();
            state = "closed"; // it worked: back to normal
            failures = 0;
            return result;
        } catch (SkyCastDown down) {
            failures++;
            if (state.equals("half-open") || failures >= failuresToOpen) {
                state = "open"; // stop calling for a while
                openedAt = now.getAsInt();
            }
            return fallback;
        }
    }
}

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
    int now = 0;
    void advance(int seconds) { now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
    private final FakeClock clock;
    boolean down = true;
    int calls = 0;
    FakeSkyCast(FakeClock clock) { this.clock = clock; }

    String weather() {
        calls++;
        if (down) {
            clock.advance(5); // SkyCast's timeout
            throw new SkyCastDown();
        }
        clock.advance(2); // slow, but an answer
        return "72F, sunny";
    }
}

FakeClock clock = new FakeClock();
FakeSkyCast skyCast = new FakeSkyCast(clock);
CircuitBreaker breaker = new CircuitBreaker(() -> clock.now, 3, 30);

void briefing(String screen) {
    int started = clock.now;
    String weather = breaker.call(skyCast::weather, "unavailable");
    IO.println(screen + ": " + weather + ", waited " + (clock.now - started) + " s (breaker " + breaker.state + ")");
}

void main() {
    briefing("Kitchen display");
    briefing("Phone app");
    briefing("Wall panel");
    briefing("Kitchen display");
    IO.println("30 s later, still down:");
    clock.advance(30);
    briefing("Phone app");
    IO.println("30 s later, SkyCast is back:");
    clock.advance(30);
    skyCast.down = false;
    briefing("Wall panel");
    IO.println("SkyCast was called " + skyCast.calls + " times in all");
}
Kitchen display: unavailable, waited 5 s (breaker closed)
Phone app: unavailable, waited 5 s (breaker closed)
Wall panel: unavailable, waited 5 s (breaker open)
Kitchen display: unavailable, waited 0 s (breaker open)
30 s later, still down:
Phone app: unavailable, waited 5 s (breaker open)
30 s later, SkyCast is back:
Wall panel: 72F, sunny, waited 2 s (breaker closed)
SkyCast was called 5 times in all
class SkyCastDown(Exception):
    pass


class CircuitBreaker:
    def __init__(self, now, failures_to_open, open_seconds):
        self.now = now
        self.failures_to_open = failures_to_open
        self.open_seconds = open_seconds
        self.failures = 0
        self.opened_at = 0
        self.state = "closed"

    def call(self, action, fallback):
        if self.state == "open":
            if self.now() - self.opened_at < self.open_seconds:
                return fallback  # open: answer at once, without calling
            self.state = "half-open"  # time's up: let this one call through as a trial
        try:
            result = action()
        except SkyCastDown:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.failures_to_open:
                self.state = "open"  # stop calling for a while
                self.opened_at = self.now()
            return fallback
        self.state = "closed"  # it worked: back to normal
        self.failures = 0
        return result


class FakeClock:
    """Stands in for the real clock: time moves only when the example says so."""

    def __init__(self):
        self.now = 0

    def advance(self, seconds):
        self.now += seconds


class FakeSkyCast:
    """Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes."""

    def __init__(self, clock):
        self.clock = clock
        self.down = True
        self.calls = 0

    def weather(self):
        self.calls += 1
        if self.down:
            self.clock.advance(5)  # SkyCast's timeout
            raise SkyCastDown()
        self.clock.advance(2)  # slow, but an answer
        return "72F, sunny"


clock = FakeClock()
skycast = FakeSkyCast(clock)
breaker = CircuitBreaker(lambda: clock.now, failures_to_open=3, open_seconds=30)


def briefing(screen):
    started = clock.now
    weather = breaker.call(skycast.weather, fallback="unavailable")
    print(f"{screen}: {weather}, waited {clock.now - started} s (breaker {breaker.state})")


briefing("Kitchen display")
briefing("Phone app")
briefing("Wall panel")
briefing("Kitchen display")
print("30 s later, still down:")
clock.advance(30)
briefing("Phone app")
print("30 s later, SkyCast is back:")
clock.advance(30)
skycast.down = False
briefing("Wall panel")
print(f"SkyCast was called {skycast.calls} times in all")
Kitchen display: unavailable, waited 5 s (breaker closed)
Phone app: unavailable, waited 5 s (breaker closed)
Wall panel: unavailable, waited 5 s (breaker open)
Kitchen display: unavailable, waited 0 s (breaker open)
30 s later, still down:
Phone app: unavailable, waited 5 s (breaker open)
30 s later, SkyCast is back:
Wall panel: 72F, sunny, waited 2 s (breaker closed)
SkyCast was called 5 times in all
class SkyCastDown extends Error {}

class CircuitBreaker {
  #now;
  #failuresToOpen;
  #openSeconds;
  #failures = 0;
  #openedAt = 0;
  state = "closed";

  constructor(now, failuresToOpen, openSeconds) {
    this.#now = now;
    this.#failuresToOpen = failuresToOpen;
    this.#openSeconds = openSeconds;
  }

  call(action, fallback) {
    if (this.state === "open") {
      if (this.#now() - this.#openedAt < this.#openSeconds) {
        return fallback; // open: answer at once, without calling
      }
      this.state = "half-open"; // time's up: let this one call through as a trial
    }
    try {
      const result = action();
      this.state = "closed"; // it worked: back to normal
      this.#failures = 0;
      return result;
    } catch (error) {
      if (!(error instanceof SkyCastDown)) throw error;
      this.#failures++;
      if (this.state === "half-open" || this.#failures >= this.#failuresToOpen) {
        this.state = "open"; // stop calling for a while
        this.#openedAt = this.#now();
      }
      return fallback;
    }
  }
}

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
  now = 0;
  advance(seconds) { this.now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
  #clock;
  down = true;
  calls = 0;
  constructor(clock) { this.#clock = clock; }

  weather() {
    this.calls++;
    if (this.down) {
      this.#clock.advance(5); // SkyCast's timeout
      throw new SkyCastDown();
    }
    this.#clock.advance(2); // slow, but an answer
    return "72F, sunny";
  }
}

const clock = new FakeClock();
const skyCast = new FakeSkyCast(clock);
const breaker = new CircuitBreaker(() => clock.now, 3, 30);

function briefing(screen) {
  const started = clock.now;
  const weather = breaker.call(() => skyCast.weather(), "unavailable");
  console.log(`${screen}: ${weather}, waited ${clock.now - started} s (breaker ${breaker.state})`);
}

briefing("Kitchen display");
briefing("Phone app");
briefing("Wall panel");
briefing("Kitchen display");
console.log("30 s later, still down:");
clock.advance(30);
briefing("Phone app");
console.log("30 s later, SkyCast is back:");
clock.advance(30);
skyCast.down = false;
briefing("Wall panel");
console.log(`SkyCast was called ${skyCast.calls} times in all`);
Kitchen display: unavailable, waited 5 s (breaker closed)
Phone app: unavailable, waited 5 s (breaker closed)
Wall panel: unavailable, waited 5 s (breaker open)
Kitchen display: unavailable, waited 0 s (breaker open)
30 s later, still down:
Phone app: unavailable, waited 5 s (breaker open)
30 s later, SkyCast is back:
Wall panel: 72F, sunny, waited 2 s (breaker closed)
SkyCast was called 5 times in all
class SkyCastDown extends Error {}

type BreakerState = "closed" | "open" | "half-open";

class CircuitBreaker {
  #now: () => number;
  #failuresToOpen: number;
  #openSeconds: number;
  #failures = 0;
  #openedAt = 0;
  state: BreakerState = "closed";

  constructor(now: () => number, failuresToOpen: number, openSeconds: number) {
    this.#now = now;
    this.#failuresToOpen = failuresToOpen;
    this.#openSeconds = openSeconds;
  }

  call(action: () => string, fallback: string): string {
    if (this.state === "open") {
      if (this.#now() - this.#openedAt < this.#openSeconds) {
        return fallback; // open: answer at once, without calling
      }
      this.state = "half-open"; // time's up: let this one call through as a trial
    }
    try {
      const result = action();
      this.state = "closed"; // it worked: back to normal
      this.#failures = 0;
      return result;
    } catch (error) {
      if (!(error instanceof SkyCastDown)) throw error;
      this.#failures++;
      if (this.state === "half-open" || this.#failures >= this.#failuresToOpen) {
        this.state = "open"; // stop calling for a while
        this.#openedAt = this.#now();
      }
      return fallback;
    }
  }
}

// Stands in for the real clock: time moves only when the example says so.
class FakeClock {
  now = 0;
  advance(seconds: number): void { this.now += seconds; }
}

// Stands in for SkyCast: while it's down, each call waits 5 s for an answer that never comes.
class FakeSkyCast {
  #clock: FakeClock;
  down = true;
  calls = 0;
  constructor(clock: FakeClock) { this.#clock = clock; }

  weather(): string {
    this.calls++;
    if (this.down) {
      this.#clock.advance(5); // SkyCast's timeout
      throw new SkyCastDown();
    }
    this.#clock.advance(2); // slow, but an answer
    return "72F, sunny";
  }
}

const clock = new FakeClock();
const skyCast = new FakeSkyCast(clock);
const breaker = new CircuitBreaker(() => clock.now, 3, 30);

function briefing(screen: string): void {
  const started = clock.now;
  const weather = breaker.call(() => skyCast.weather(), "unavailable");
  console.log(`${screen}: ${weather}, waited ${clock.now - started} s (breaker ${breaker.state})`);
}

briefing("Kitchen display");
briefing("Phone app");
briefing("Wall panel");
briefing("Kitchen display");
console.log("30 s later, still down:");
clock.advance(30);
briefing("Phone app");
console.log("30 s later, SkyCast is back:");
clock.advance(30);
skyCast.down = false;
briefing("Wall panel");
console.log(`SkyCast was called ${skyCast.calls} times in all`);
Kitchen display: unavailable, waited 5 s (breaker closed)
Phone app: unavailable, waited 5 s (breaker closed)
Wall panel: unavailable, waited 5 s (breaker open)
Kitchen display: unavailable, waited 0 s (breaker open)
30 s later, still down:
Phone app: unavailable, waited 5 s (breaker open)
30 s later, SkyCast is back:
Wall panel: 72F, sunny, waited 2 s (breaker closed)
SkyCast was called 5 times in all

Follow the diagram down the output. The first three failures trip the breaker, so the fourth screen gets "unavailable" after 0 seconds, without a call. Thirty seconds later the breaker goes half-open and lets the phone app's call through as a trial; SkyCast is still down, so it opens again. Thirty seconds after that, the wall panel's trial works and the breaker closes. Over the whole outage, SkyCast was called 5 times instead of once per screen per request.

Why a field, and not State's classes? With three states that each do one small thing, a state field and an if per state read clearly, and the diagram sits beside the code. If each state grew its own rules (a half-open state that lets five trial calls through and counts how many work), give each state its own class, exactly as Pass 2 did for packages.

Retry and the breaker, together

The hub uses both: retry for hiccups, the breaker for outages. You can put them either way round, as long as you know which you chose. With the retry inside the breaker, the breaker counts a briefing as failed only after its three tries gave up (the exercise does this). Libraries usually put the retry outside, so each try counts as a failure, and they stop retrying the moment the breaker says it's open: retrying a breaker that's open is asking a closed door again.

How it got its name

Electricians named the switch; software borrowed it. The idea was written up for programs in a 2007 book about keeping software running in production, Release It!, and became the Circuit Breaker pattern. Like Retry, it's not in the original 1994 book: it belongs to programs that talk to other programs across a network.

The modern version: a library does it for you

At work you'll rarely write the class above; every language has a library with the same three states and more settings (failure rates, time windows, how many trial calls). The class you built is what those settings mean. Here's SkyCast's breaker in each (shown, not run here; versions checked on 2026-11-01):

// .NET: Polly 8.9.0 (dotnet add package Polly.Core --version 8.9.0)
using Polly;
using Polly.CircuitBreaker;

ResiliencePipeline<string> pipeline = new ResiliencePipelineBuilder<string>()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<string>
    {
        FailureRatio = 1.0,                          // open when every call in the window failed...
        MinimumThroughput = 3,                       // ...and there were at least 3 of them
        SamplingDuration = TimeSpan.FromSeconds(60), // the window: the last 60 s
        BreakDuration = TimeSpan.FromSeconds(30),    // stay open 30 s, then let a trial call through
    })
    .Build();

string weather;
try
{
    weather = pipeline.Execute(() => skyCast.Weather());
}
catch (BrokenCircuitException)
{
    weather = "unavailable"; // open: SkyCast wasn't called
}
catch (SkyCastDownException)
{
    weather = "unavailable"; // closed or half-open: the call failed
}
// Java: Resilience4j 2.4.0 (Maven: io.github.resilience4j:resilience4j-circuitbreaker:2.4.0)
import io.github.resilience4j.circuitbreaker.CallNotPermittedException;
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import java.time.Duration;
import java.util.function.Supplier;

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
        .slidingWindowSize(3)                             // look at the last 3 calls
        .minimumNumberOfCalls(3)
        .failureRateThreshold(100)                        // open when all 3 failed
        .waitDurationInOpenState(Duration.ofSeconds(30))  // stay open 30 s...
        .permittedNumberOfCallsInHalfOpenState(1)         // ...then let one trial call through
        .build();
CircuitBreaker breaker = CircuitBreaker.of("skycast", config);
Supplier<String> guarded = CircuitBreaker.decorateSupplier(breaker, skyCast::weather);

String weather;
try {
    weather = guarded.get();
} catch (CallNotPermittedException open) {
    weather = "unavailable"; // open: SkyCast wasn't called
} catch (SkyCastDown failed) {
    weather = "unavailable"; // closed or half-open: the call failed
}
# Python: pybreaker 1.4.1 (pip install pybreaker==1.4.1); tenacity, from the last lesson, does retries only
import pybreaker

skycast_breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30)  # open after 3 failures, for 30 s

try:
    weather = skycast_breaker.call(skycast.weather)
except pybreaker.CircuitBreakerError:
    weather = "unavailable"  # open (or this failure tripped it)
except SkyCastDown:
    weather = "unavailable"  # closed or half-open: the call failed
// JavaScript: cockatiel 4.0.0 (npm install cockatiel@4.0.0)
import { BrokenCircuitError, ConsecutiveBreaker, circuitBreaker, handleType } from "cockatiel";

const breaker = circuitBreaker(handleType(SkyCastDown), {
  halfOpenAfter: 30_000,              // stay open 30 s, then let a trial call through
  breaker: new ConsecutiveBreaker(3), // open after 3 failures in a row
});

let weather;
try {
  weather = await breaker.execute(() => skyCast.weather());
} catch (error) {
  if (!(error instanceof BrokenCircuitError) && !(error instanceof SkyCastDown)) throw error;
  weather = "unavailable"; // open (BrokenCircuitError), or the call failed (SkyCastDown)
}
// TypeScript: cockatiel 4.0.0 (npm install cockatiel@4.0.0)
import { BrokenCircuitError, ConsecutiveBreaker, circuitBreaker, handleType } from "cockatiel";

const breaker = circuitBreaker(handleType(SkyCastDown), {
  halfOpenAfter: 30_000,              // stay open 30 s, then let a trial call through
  breaker: new ConsecutiveBreaker(3), // open after 3 failures in a row
});

let weather: string;
try {
  weather = await breaker.execute(() => skyCast.weather());
} catch (error) {
  if (!(error instanceof BrokenCircuitError) && !(error instanceof SkyCastDown)) throw error;
  weather = "unavailable"; // open (BrokenCircuitError), or the call failed (SkyCastDown)
}

Each library throws its own "the breaker is open" error (BrokenCircuitException, CallNotPermittedException, CircuitBreakerError, BrokenCircuitError) instead of taking a fallback value, so the caller catches it. Polly and Resilience4j also put retry, breaker and timeout into one pipeline, in the order you choose.

Where you've already used it

  • The breaker box in your house, and the reset button on a bathroom outlet.
  • ASP.NET Core's AddStandardResilienceHandler() puts a retry and a circuit breaker on every call an HttpClient makes, and Spring Cloud Circuit Breaker does the same in Spring.
  • A phone's "app not responding" box, which stops waiting on a stuck app and offers to close it, is the same instinct: stop waiting on something that has shown it won't answer.

When not to use it

  • When the other system is inside your own program. A method call doesn't go down for an hour; a breaker around it is ceremony.
  • When there's no sensible fallback and the caller must know. Paying for an order can't quietly fall back to "unavailable"; it should fail clearly and ask the customer to try again later. A breaker still helps by failing fast, but the error must reach someone.
  • When calls are rare. If the hub asks once an hour, "three failures in a row" takes three hours to notice. A breaker pays off when many calls go to the same service.

The cost: one more moving part with its own settings (how many failures, how long open), and a breaker with the wrong settings can trip on a few slow calls and turn a small problem into "unavailable" for everyone.

How often you'll meet it: now and then (our estimate: in any app that calls other services, usually as a library setting, but rarely written by hand).

Check yourself

2 questions to try on the lesson page.

So what

Retry gives a hiccup a second chance; a circuit breaker notices an outage and stops knocking. Closed, open, half-open: three states, a few rules, and every screen in the house stops freezing on a service that isn't there.

Key terms

Cheat sheets