Publish/Subscribe: Observer Across the Network

When the front door has to tell Maya's phone, which is on another network and sometimes offline, Observer's list breaks down: the door must know the phone, wait for it, and loses the message when it's away; publish/subscribe puts a message board with topics in the middle, so senders post, readers subscribe, and neither knows the other.

Freelancer · Software Development · OOP · Updated

By

After this lesson you can

  • Connect senders and readers through a message board with topics, so neither knows the other
  • Tell Observer from publish/subscribe by what each one is for
  • Spot the smells that point to publish/subscribe: listeners on other computers, readers that go offline, a sender waiting on each one

Key takeaways

  • The smell: a sender that holds listeners on other computers, waits on each one, and loses messages when one is offline.
  • The cause: Observer joins the sender to each listener directly, which works only inside one program, with every listener present.
  • The fix: a message board (a broker) in the middle with topics; senders publish to a topic, readers subscribe to topics, and the board keeps each reader's messages until it's back.
  • Observer is one program telling its own listeners; publish/subscribe is many programs talking through a board, each knowing only the topic.
  • Smart homes really work this way: devices publish to topics on a broker, often with a protocol called MQTT.

Before this: Observer the Modern Way: Events and Listeners.

The last lesson's listeners were all in one program: the activity log and the door screen live inside the hub's app. Now Maya wants an alert on her phone whenever the front door unlocks while she's at work. Her phone isn't in the hub's program; it's a different computer, somewhere on the phone network, and sometimes it's in a tunnel with no signal. Observer's list can't stretch that far. The fix has a name that says how it works: publish/subscribe.

The symptom: the door has to know, and wait for, a phone

Here's Observer as we have it, with the phone added as one more listener. To keep the example in one program, the phone is a stand-in with an online switch; a real one would be across the network:

FrontDoor door = new FrontDoor();
Reader log = new Reader("Log");
Reader phone = new Reader("Maya's phone");
door.Subscribe(log);
door.Subscribe(phone);

door.Unlock("Sam");
Console.WriteLine("Maya's phone goes into a tunnel");
phone.Online = false;
door.Unlock("the guest");
Console.WriteLine("Maya's phone is back");
phone.Online = true;

class OfflineException(string message) : Exception(message) { }

// Stands in for a reader; Maya's phone would really be on another computer.
class Reader(string name)
{
    public string Name { get; } = name;
    public bool Online { get; set; } = true;

    public void Receive(string message)
    {
        if (!Online)
        {
            throw new OfflineException(quot;{Name} is offline");
        }
        Console.WriteLine(quot;{Name}: {message}");
    }
}

// Observer, as in Pass 1: the door keeps its listeners and calls each one.
class FrontDoor
{
    private readonly List<Reader> listeners = new List<Reader>();
    public void Subscribe(Reader listener) => listeners.Add(listener);

    public void Unlock(string who)
    {
        foreach (Reader listener in listeners)
        {
            try
            {
                listener.Receive(quot;front door unlocked by {who}");
            }
            catch (OfflineException error)
            {
                Console.WriteLine(quot;Lost: {error.Message}");
            }
        }
    }
}
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Lost: Maya's phone is offline
Maya's phone is back
class OfflineException extends RuntimeException {
    OfflineException(String message) { super(message); }
}

// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
    final String name;
    boolean online = true;
    Reader(String name) { this.name = name; }

    void receive(String message) {
        if (!online) {
            throw new OfflineException(name + " is offline");
        }
        IO.println(name + ": " + message);
    }
}

// Observer, as in Pass 1: the door keeps its listeners and calls each one.
class FrontDoor {
    private final List<Reader> listeners = new ArrayList<>();
    void subscribe(Reader listener) { listeners.add(listener); }

    void unlock(String who) {
        for (Reader listener : listeners) {
            try {
                listener.receive("front door unlocked by " + who);
            } catch (OfflineException error) {
                IO.println("Lost: " + error.getMessage());
            }
        }
    }
}

void main() {
    FrontDoor door = new FrontDoor();
    Reader log = new Reader("Log");
    Reader phone = new Reader("Maya's phone");
    door.subscribe(log);
    door.subscribe(phone);

    door.unlock("Sam");
    IO.println("Maya's phone goes into a tunnel");
    phone.online = false;
    door.unlock("the guest");
    IO.println("Maya's phone is back");
    phone.online = true;
}
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Lost: Maya's phone is offline
Maya's phone is back
class Reader:
    """Stands in for a reader; Maya's phone would really be on another computer."""

    def __init__(self, name):
        self.name = name
        self.online = True

    def receive(self, message):
        if not self.online:
            raise ConnectionError(f"{self.name} is offline")
        print(f"{self.name}: {message}")


class FrontDoor:
    """Observer, as in Pass 1: the door keeps its listeners and calls each one."""

    def __init__(self):
        self.listeners = []

    def subscribe(self, listener):
        self.listeners.append(listener)

    def unlock(self, who):
        for listener in self.listeners:
            try:
                listener.receive(f"front door unlocked by {who}")
            except ConnectionError as error:
                print(f"Lost: {error}")


door = FrontDoor()
log = Reader("Log")
phone = Reader("Maya's phone")
door.subscribe(log)
door.subscribe(phone)

door.unlock("Sam")
print("Maya's phone goes into a tunnel")
phone.online = False
door.unlock("the guest")
print("Maya's phone is back")
phone.online = True
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Lost: Maya's phone is offline
Maya's phone is back
class OfflineError extends Error {}

// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
  online = true;
  constructor(name) { this.name = name; }
  receive(message) {
    if (!this.online) throw new OfflineError(`${this.name} is offline`);
    console.log(`${this.name}: ${message}`);
  }
}

// Observer, as in Pass 1: the door keeps its listeners and calls each one.
class FrontDoor {
  #listeners = [];
  subscribe(listener) { this.#listeners.push(listener); }
  unlock(who) {
    for (const listener of this.#listeners) {
      try {
        listener.receive(`front door unlocked by ${who}`);
      } catch (error) {
        if (!(error instanceof OfflineError)) throw error;
        console.log(`Lost: ${error.message}`);
      }
    }
  }
}

const door = new FrontDoor();
const log = new Reader("Log");
const phone = new Reader("Maya's phone");
door.subscribe(log);
door.subscribe(phone);

door.unlock("Sam");
console.log("Maya's phone goes into a tunnel");
phone.online = false;
door.unlock("the guest");
console.log("Maya's phone is back");
phone.online = true;
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Lost: Maya's phone is offline
Maya's phone is back
class OfflineError extends Error {}

// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
  name: string;
  online = true;
  constructor(name: string) { this.name = name; }
  receive(message: string): void {
    if (!this.online) throw new OfflineError(`${this.name} is offline`);
    console.log(`${this.name}: ${message}`);
  }
}

// Observer, as in Pass 1: the door keeps its listeners and calls each one.
class FrontDoor {
  #listeners: Reader[] = [];
  subscribe(listener: Reader): void { this.#listeners.push(listener); }
  unlock(who: string): void {
    for (const listener of this.#listeners) {
      try {
        listener.receive(`front door unlocked by ${who}`);
      } catch (error) {
        if (!(error instanceof OfflineError)) throw error;
        console.log(`Lost: ${error.message}`);
      }
    }
  }
}

const door = new FrontDoor();
const log = new Reader("Log");
const phone = new Reader("Maya's phone");
door.subscribe(log);
door.subscribe(phone);

door.unlock("Sam");
console.log("Maya's phone goes into a tunnel");
phone.online = false;
door.unlock("the guest");
console.log("Maya's phone is back");
phone.online = true;
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Lost: Maya's phone is offline
Maya's phone is back

The guest unlocked the door while Maya was in the tunnel, and she never found out: the message was lost. And the code shows more trouble. The door, a lock, now catches network errors. It holds a direct line to Maya's phone, so it has to know how to reach it. If the phone answers slowly, the door waits, and so does the activity log behind it. When Sam's watch wants the same alerts, the door changes again.

  • The symptom: a sender that holds listeners on other computers, waits on each one in turn, and loses messages when one is away.
  • The cause: Observer joins the sender straight to each listener. That works inside one program, where every listener is right there and answers at once; across a network, none of that is true.

What would you try?

Have the door keep a list of missed messages for each listener. Now a lock is running a post office, for every listener, forever.

Have the phone ask the door every minute "anything new?" The door has to be reachable from the internet, every phone checks constantly, and alerts arrive up to a minute late.

Send a text message instead. That works for one alert to one phone. The log, the watch and next year's tablet each need their own way in, all known to the door.

What we want is a place in the middle that senders post to and readers collect from, which keeps a reader's messages while it's away, so senders and readers never have to know about each other.

The fix: a message board in the middle

A community bulletin board works this way. Someone pins a note under "Lost and found" or "Rides"; whoever cares about those headings reads them, today or next week. The person who pinned the note doesn't know who'll read it, and readers don't know who wrote it. Each only knows the board and the heading.

A community bulletin board in a building's lobby with three headings across the top, "Lost and found", "Rides" and "For sale"; one person is pinning a note under "Rides" while another, at the other end of the board, reads the notes under "Lost and found"; they aren't looking at each other

In code, the board is a message broker, and the headings are topics. A sender publishes a message to a topic ("front-door"); a reader subscribes to the topics it cares about. The broker hands each message to every subscriber of its topic, and keeps a reader's messages in its own mailbox while the reader is offline.

Publish/subscribe's roles: the front door and the SkyCast feed publish to a message board with two topics, front-door and weather. It delivers front-door messages to the activity log and Maya's phone (a mailbox waits while the phone is offline) and weather to the kitchen display

The classic version: a broker with topics and mailboxes

The door now knows only the board. The board keeps each reader's mailbox; when the phone comes back, it collects what's waiting:

MessageBoard board = new MessageBoard();
Reader log = new Reader("Log");
Reader phone = new Reader("Maya's phone");
Reader display = new Reader("Kitchen display");
board.Subscribe("front-door", log);
board.Subscribe("front-door", phone);
board.Subscribe("weather", display);
FrontDoor door = new FrontDoor(board);

door.Unlock("Sam");
Console.WriteLine("Maya's phone goes into a tunnel");
phone.Online = false;
door.Unlock("the guest");
Console.WriteLine("Maya's phone is back");
phone.Online = true;
board.Deliver(phone); // the phone reconnects and asks for what's waiting
board.Publish("weather", "72F, sunny"); // SkyCast's feed knows only the board too

// The board in the middle: senders publish to a topic; each reader has its own mailbox.
class MessageBoard
{
    private readonly Dictionary<string, List<Reader>> readers = new Dictionary<string, List<Reader>>();
    private readonly Dictionary<Reader, Queue<string>> mailboxes = new Dictionary<Reader, Queue<string>>();

    public void Subscribe(string topic, Reader reader)
    {
        if (!readers.ContainsKey(topic))
        {
            readers[topic] = new List<Reader>();
        }
        readers[topic].Add(reader);
        mailboxes.TryAdd(reader, new Queue<string>());
    }

    public void Publish(string topic, string message)
    {
        foreach (Reader reader in readers.GetValueOrDefault(topic, new List<Reader>()))
        {
            mailboxes[reader].Enqueue(message);
            Deliver(reader);
        }
    }

    // Hands a reader everything waiting for it, if it's online.
    public void Deliver(Reader reader)
    {
        while (reader.Online && mailboxes[reader].Count > 0)
        {
            reader.Receive(mailboxes[reader].Dequeue());
        }
    }
}

// Stands in for a reader; Maya's phone would really be on another computer.
class Reader(string name)
{
    public string Name { get; } = name;
    public bool Online { get; set; } = true;
    public void Receive(string message) => Console.WriteLine(quot;{Name}: {message}");
}

// The door knows only the board.
class FrontDoor(MessageBoard board)
{
    public void Unlock(string who) => board.Publish("front-door", quot;front door unlocked by {who}");
}
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Maya's phone is back
Maya's phone: front door unlocked by the guest
Kitchen display: 72F, sunny
// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
    final String name;
    boolean online = true;
    Reader(String name) { this.name = name; }
    void receive(String message) { IO.println(name + ": " + message); }
}

// The board in the middle: senders publish to a topic; each reader has its own mailbox.
class MessageBoard {
    private final Map<String, List<Reader>> readers = new HashMap<>();
    private final Map<Reader, ArrayDeque<String>> mailboxes = new HashMap<>();

    void subscribe(String topic, Reader reader) {
        readers.computeIfAbsent(topic, key -> new ArrayList<>()).add(reader);
        mailboxes.putIfAbsent(reader, new ArrayDeque<>());
    }

    void publish(String topic, String message) {
        for (Reader reader : readers.getOrDefault(topic, List.of())) {
            mailboxes.get(reader).add(message);
            deliver(reader);
        }
    }

    // Hands a reader everything waiting for it, if it's online.
    void deliver(Reader reader) {
        while (reader.online && !mailboxes.get(reader).isEmpty()) {
            reader.receive(mailboxes.get(reader).poll());
        }
    }
}

// The door knows only the board.
class FrontDoor {
    private final MessageBoard board;
    FrontDoor(MessageBoard board) { this.board = board; }
    void unlock(String who) { board.publish("front-door", "front door unlocked by " + who); }
}

void main() {
    MessageBoard board = new MessageBoard();
    Reader log = new Reader("Log");
    Reader phone = new Reader("Maya's phone");
    Reader display = new Reader("Kitchen display");
    board.subscribe("front-door", log);
    board.subscribe("front-door", phone);
    board.subscribe("weather", display);
    FrontDoor door = new FrontDoor(board);

    door.unlock("Sam");
    IO.println("Maya's phone goes into a tunnel");
    phone.online = false;
    door.unlock("the guest");
    IO.println("Maya's phone is back");
    phone.online = true;
    board.deliver(phone); // the phone reconnects and asks for what's waiting
    board.publish("weather", "72F, sunny"); // SkyCast's feed knows only the board too
}
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Maya's phone is back
Maya's phone: front door unlocked by the guest
Kitchen display: 72F, sunny
class Reader:
    """Stands in for a reader; Maya's phone would really be on another computer."""

    def __init__(self, name):
        self.name = name
        self.online = True

    def receive(self, message):
        print(f"{self.name}: {message}")


class MessageBoard:
    """The board in the middle: senders publish to a topic; each reader has its own mailbox."""

    def __init__(self):
        self.readers = {}  # topic -> the readers subscribed to it
        self.mailboxes = {}  # reader -> the messages waiting for it

    def subscribe(self, topic, reader):
        self.readers.setdefault(topic, []).append(reader)
        self.mailboxes.setdefault(reader, [])

    def publish(self, topic, message):
        for reader in self.readers.get(topic, []):
            self.mailboxes[reader].append(message)
            self.deliver(reader)

    def deliver(self, reader):
        """Hands a reader everything waiting for it, if it's online."""
        while reader.online and self.mailboxes[reader]:
            reader.receive(self.mailboxes[reader].pop(0))


class FrontDoor:
    """The door knows only the board."""

    def __init__(self, board):
        self.board = board

    def unlock(self, who):
        self.board.publish("front-door", f"front door unlocked by {who}")


board = MessageBoard()
log = Reader("Log")
phone = Reader("Maya's phone")
display = Reader("Kitchen display")
board.subscribe("front-door", log)
board.subscribe("front-door", phone)
board.subscribe("weather", display)
door = FrontDoor(board)

door.unlock("Sam")
print("Maya's phone goes into a tunnel")
phone.online = False
door.unlock("the guest")
print("Maya's phone is back")
phone.online = True
board.deliver(phone)  # the phone reconnects and asks for what's waiting
board.publish("weather", "72F, sunny")  # SkyCast's feed knows only the board too
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Maya's phone is back
Maya's phone: front door unlocked by the guest
Kitchen display: 72F, sunny
// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
  online = true;
  constructor(name) { this.name = name; }
  receive(message) { console.log(`${this.name}: ${message}`); }
}

// The board in the middle: senders publish to a topic; each reader has its own mailbox.
class MessageBoard {
  #readers = new Map(); // topic -> the readers subscribed to it
  #mailboxes = new Map(); // reader -> the messages waiting for it

  subscribe(topic, reader) {
    this.#readers.set(topic, [...(this.#readers.get(topic) ?? []), reader]);
    if (!this.#mailboxes.has(reader)) this.#mailboxes.set(reader, []);
  }

  publish(topic, message) {
    for (const reader of this.#readers.get(topic) ?? []) {
      this.#mailboxes.get(reader).push(message);
      this.deliver(reader);
    }
  }

  // Hands a reader everything waiting for it, if it's online.
  deliver(reader) {
    const mailbox = this.#mailboxes.get(reader);
    while (reader.online && mailbox.length > 0) reader.receive(mailbox.shift());
  }
}

// The door knows only the board.
class FrontDoor {
  #board;
  constructor(board) { this.#board = board; }
  unlock(who) { this.#board.publish("front-door", `front door unlocked by ${who}`); }
}

const board = new MessageBoard();
const log = new Reader("Log");
const phone = new Reader("Maya's phone");
const display = new Reader("Kitchen display");
board.subscribe("front-door", log);
board.subscribe("front-door", phone);
board.subscribe("weather", display);
const door = new FrontDoor(board);

door.unlock("Sam");
console.log("Maya's phone goes into a tunnel");
phone.online = false;
door.unlock("the guest");
console.log("Maya's phone is back");
phone.online = true;
board.deliver(phone); // the phone reconnects and asks for what's waiting
board.publish("weather", "72F, sunny"); // SkyCast's feed knows only the board too
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Maya's phone is back
Maya's phone: front door unlocked by the guest
Kitchen display: 72F, sunny
// Stands in for a reader; Maya's phone would really be on another computer.
class Reader {
  name: string;
  online = true;
  constructor(name: string) { this.name = name; }
  receive(message: string): void { console.log(`${this.name}: ${message}`); }
}

// The board in the middle: senders publish to a topic; each reader has its own mailbox.
class MessageBoard {
  #readers = new Map<string, Reader[]>(); // topic -> the readers subscribed to it
  #mailboxes = new Map<Reader, string[]>(); // reader -> the messages waiting for it

  subscribe(topic: string, reader: Reader): void {
    this.#readers.set(topic, [...(this.#readers.get(topic) ?? []), reader]);
    if (!this.#mailboxes.has(reader)) this.#mailboxes.set(reader, []);
  }

  publish(topic: string, message: string): void {
    for (const reader of this.#readers.get(topic) ?? []) {
      this.#mailboxes.get(reader)!.push(message);
      this.deliver(reader);
    }
  }

  // Hands a reader everything waiting for it, if it's online.
  deliver(reader: Reader): void {
    const mailbox = this.#mailboxes.get(reader)!;
    while (reader.online && mailbox.length > 0) reader.receive(mailbox.shift()!);
  }
}

// The door knows only the board.
class FrontDoor {
  #board: MessageBoard;
  constructor(board: MessageBoard) { this.#board = board; }
  unlock(who: string): void { this.#board.publish("front-door", `front door unlocked by ${who}`); }
}

const board = new MessageBoard();
const log = new Reader("Log");
const phone = new Reader("Maya's phone");
const display = new Reader("Kitchen display");
board.subscribe("front-door", log);
board.subscribe("front-door", phone);
board.subscribe("weather", display);
const door = new FrontDoor(board);

door.unlock("Sam");
console.log("Maya's phone goes into a tunnel");
phone.online = false;
door.unlock("the guest");
console.log("Maya's phone is back");
phone.online = true;
board.deliver(phone); // the phone reconnects and asks for what's waiting
board.publish("weather", "72F, sunny"); // SkyCast's feed knows only the board too
Log: front door unlocked by Sam
Maya's phone: front door unlocked by Sam
Maya's phone goes into a tunnel
Log: front door unlocked by the guest
Maya's phone is back
Maya's phone: front door unlocked by the guest
Kitchen display: 72F, sunny

Maya now hears about the guest, a little late, when she's out of the tunnel. The door doesn't catch network errors or know any phone; it publishes to "front-door" and is done. The weather goes only to the kitchen display, because that's the only reader of the "weather" topic: the log never sees it. Sam's watch next year is one more subscribe, with no change to the door.

The real thing: a broker in its own program

The board here lives in one program so the example can run. A real broker is its own program (a server), and the hub, the phones and the watches all connect to it over the network. Smart homes very often use MQTT, a small messaging protocol made for devices, where topics look like paths, home/front-door or home/weather, and a reader can subscribe to home/# to hear everything in the house. Web apps use brokers such as RabbitMQ, Apache Kafka, Redis and Azure Service Bus. The idea is the one you built: publish to a topic, subscribe to topics, and let the broker keep the mail.

How it got its name

The two verbs are the whole pattern: senders publish, readers subscribe, so it's publish/subscribe, often shortened to pub/sub. It isn't in the original 1994 book, which is about one program; it comes from systems where many programs on many computers need to hear about the same events, and it's sometimes described as Observer across the network. Here's how the two compare:

Pattern What it's for Strengths Weaknesses Pick it when
Observer One object tells a list of listeners, in the same program, when it changes Simple and direct; nothing extra to run; listeners hear at once The subject holds every listener; a slow or broken listener holds up the subject; every listener must be in the program and present The subject and its listeners live in one program, such as a screen and its buttons
Publish/subscribe Senders post messages to topics on a board in the middle, and readers subscribe to topics Senders and readers never know each other; readers can be on other computers, many, or offline for a while; a new reader changes no sender A broker to run and look after; messages arrive a little later; harder to see who reacts to what Senders and readers are in different programs or on different computers, or come and go

The quickest way to tell them apart: with Observer, the listener's name is in the subject's list. With publish/subscribe, the only name both sides share is the topic.

Where you've already used it

  • Notifications on your phone: an app's server publishes, and the phone's notification service keeps the message until your phone is back online.
  • Smart-home hubs and sensors, very often through MQTT, as above.
  • Following a hashtag or a channel: you subscribe to the topic, not to each person who posts.

When not to use it

  • When everyone is in one program. A broker to tell the door screen about the door is a lot of machinery; that's an event, the last lesson.
  • When the sender needs an answer. Publishing is "here's what happened", not a question. Asking SkyCast for the weather is a call, not a message.
  • When order and timing must be exact. Messages through a broker arrive a little later, and from different senders they can arrive in a different order than they were sent.

The cost: one more running system that can itself go down (brokers are usually run as several copies for that reason), and a harder question when something goes wrong: who was subscribed to what?

How often you'll meet it: now and then (our estimate: in any system made of several programs or many devices; rarely inside one small app).

Check yourself

2 questions to try on the lesson page.

So what

Observer joins a sender to each listener, and that's perfect inside one program. When listeners live on other computers, come and go, or go offline, put a board in the middle: senders publish to topics, readers subscribe, and the broker keeps the mail until it can be delivered.

Key terms

Cheat sheets