Newsletter Service HLD: Fan Out a Million Emails Without Sending Twice

"Design a newsletter service — Substack, Mailchimp, the 'weekly digest' every product sends." It sounds like a CRUD app with a Send button. It isn't. The Send button is where it gets hard: one click has to reach millions of inboxes, each person exactly once, while the network drops, a provider throttles you, and someone unsubscribes mid-send. Get the fan-out wrong and you either drop people or — far worse — mail them the same thing five times and land on a spam blocklist.

So the whole design is one question wearing a few hats: how do you turn one campaign into millions of independent, retryable, deduplicated sends? Everything else — subscribers, templates, stats — is the easy CRUD around that core.

Let's start nowhere near a computer

Picture a print shop mailing a paper newsletter to a few hundred thousand readers. There's a master list of addresses, a stack of clerks stuffing envelopes, and the postal service that only accepts so many sacks an hour. Three details make the shop run, and they're the entire system:

• A "no junk mail" register — anyone who said stop is checked before an envelope is ever addressed. Mail them anyway and the post office fines you.
• A "return to sender" pile — addresses that bounced. They go straight onto the no-junk register; keep mailing dead addresses and the postal service starts trusting you less.
• A ticked checklist — as each envelope is sealed, the clerk crosses that name off. If the power cuts out and the shift restarts, nobody re-stuffs an envelope for a crossed-off name. That checklist is the difference between "everyone got one" and "everyone got three."

Swap the nouns and you have the system: master list → subscribers, no-junk register → suppression list, return-to-sender → bounce handling, the postal rate → the email provider's rate limit, the clerks → a pool of send workers, and that ticked checklist → the send log, the single most important table in the design.

Where this exact shape shows up

• Mailchimp / SendGrid / Substack — campaign sends are this design; "deliverability" is the no-junk-register and rate-limit discipline taken seriously.
• Any "notifications" service — push, SMS, and email all fan out one event to many recipients with per-recipient retry and dedup. Same skeleton, different last mile.
• The thread pool you've already built — the send workers are a thread pool draining a queue; this article is what happens when its tasks own a recipient and must survive a crash.
• The rate limiter — every worker passes through one before calling the provider.

Step 1 — Functional requirements (sentences first)

Say what it must do, in plain sentences, before drawing a single box.

• A visitor can subscribe (with double opt-in — a confirm click, so nobody signs up someone else) and unsubscribe in one click.
• An author can create a campaign (subject, templated body) and send it to a list, now or scheduled.
• The system delivers the campaign to every eligible subscriber exactly once, skipping anyone unsubscribed or suppressed at send time.
• Hard bounces and spam complaints are recorded and suppressed automatically.
• Per campaign, it tracks opens and clicks and reports stats (sent, delivered, bounced, opened, clicked).

That third sentence is the whole problem. "Exactly once to millions, honoring an unsubscribe that might land one second before we mail you" is not CRUD — it's a distributed-systems promise, and the rest of the design exists to keep it.

Step 2 — Non-functional requirements

What matters isn't just what it does but how well — and for a sender, the non-functional requirements are the design.

• Scalability / throughput. A campaign can target tens of millions; the send must fan out across many workers and finish in a sane window, not a week.
• Reliability (no lost, no doubled). Every accepted recipient is mailed despite crashes and redeliveries — at-least-once delivery plus dedup = effectively exactly-once per recipient.
• Deliverability. Respect the provider's rate limit, retry transient failures, and suppress hard bounces — or your sending reputation tanks and nothing arrives.
• Low control-plane latency. Creating a campaign or subscribing returns instantly; the heavy lifting (the send) is asynchronous.
• Durability. A queued campaign survives a process restart; no recipient is silently dropped.
• Compliance. Unsubscribe is honored promptly and one-click (CAN-SPAM / GDPR).

Listing them is the easy half. The design only earns them if it fulfills them — here's the contract, each requirement and the mechanism that keeps it:

Every trade-off later is chosen to keep one of these.

Step 3 — Nouns and the data model

The nouns fall out of the sentences: Subscriber, Campaign, the send log entry, a suppression record, and a tracking event.

Which datastore — and why it isn't a default. The control plane — subscribers, campaigns, the send log, suppression — wants a relational database (Postgres), and not out of habit. The send log needs a UNIQUE(campaign_id, subscriber_id) constraint to make "did we already queue this person?" a database-enforced fact rather than a hopeful check; that unique index is the dedup, and only a transactional store gives it to you for free. Subscribers need queryable segments ("everyone in list 7 who's still subscribed"). All of that is relational's home turf.

The tracking events are the opposite animal: a firehose of opens and clicks, append-only, that can dwarf every other write by orders of magnitude. Forcing them into Postgres would drown the primary. They belong in a stream (Kafka) feeding a columnar store (ClickHouse) for analytics, with Redis counters for the live per-campaign numbers. Right tool, right shape: relational for the truth you must not corrupt, a stream + OLAP for the volume you only aggregate.

The cast, in code:

package dev.fiveyear.newsletter;
 
/** One recipient's delivery state for a campaign — the idempotency unit. */
enum SendState { QUEUED, SENT, FAILED, SUPPRESSED }
 
/** A person on a list. `subscribed` flips to false on one-click unsubscribe. */
record Subscriber(long id, String email, boolean subscribed) {}
 
/** The provider's verdict for one send attempt. */
enum SendResult { ACCEPTED, HARD_BOUNCE }
 
/** A retryable error (provider 5xx, throttle, timeout) — never a bounce. */
class TransientFailure extends RuntimeException {
    TransientFailure(String message) { super(message); }
}
 
/** The email provider (SES/SendGrid). Throws TransientFailure on retryable errors. */
interface MailProvider {
    SendResult send(String to, String subject, String body);
}

Step 4 — The send log is the idempotency key

Here's the load-bearing idea. The send log records, for every (campaign, subscriber) pair, what's happened to it. Its unique key turns a dangerous operation — "enqueue this recipient" — into a safe, repeatable one: claim the slot atomically, and only act if you're the one who claimed it.

package dev.fiveyear.newsletter;
 
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
 
/**
 * The send log answers one question: "did this recipient already get this
 * campaign?" Its unique key (campaignId, subscriberId) is what makes the whole
 * pipeline idempotent — a crashed orchestrator or a redelivered queue message
 * can replay without anyone getting the email twice.
 */
public class SendLog {
    private record Key(String campaignId, long subscriberId) {}
 
    private final ConcurrentMap<Key, SendState> states = new ConcurrentHashMap<>();
 
    /** Atomically claim a slot. Returns true only the first time — that's the dedup. */
    public boolean enqueueOnce(String campaignId, long subscriberId) {
        return states.putIfAbsent(new Key(campaignId, subscriberId), SendState.QUEUED) == null;
    }
 
    public void mark(String campaignId, long subscriberId, SendState state) {
        states.put(new Key(campaignId, subscriberId), state);
    }
 
    public SendState state(String campaignId, long subscriberId) {
        return states.get(new Key(campaignId, subscriberId));
    }
 
    public long count(SendState state) {
        return states.values().stream().filter(v -> v == state).count();
    }
}

The ConcurrentHashMap here stands in for a database row with a unique index: putIfAbsent returning null is exactly what INSERT ... ON CONFLICT DO NOTHING tells you in Postgres — you created the row, so you enqueue the task; anyone else replaying the same fan-out gets a non-null back and does nothing. Same atomic claim, whether it's one box or a hundred.

The suppression list is the no-junk register — checked before every send:

package dev.fiveyear.newsletter;
 
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
 
/**
 * Hard bounces and spam complaints land here. Checked before every send —
 * mailing a suppressed address again is how you get your sending IP blacklisted.
 */
public class SuppressionList {
    private final Set<Long> blocked = ConcurrentHashMap.newKeySet();
 
    public void suppress(long subscriberId) { blocked.add(subscriberId); }
 
    public boolean isSuppressed(long subscriberId) { return blocked.contains(subscriberId); }
}

Step 5 — Verbs become APIs (the API design)

The verbs from Step 1 become endpoints. The one rule worth stating out loud: the control plane only enqueues — it never blocks on the actual sending. POST /campaigns/{id}/send returns the moment fan-out is durably scheduled, keeping the latency NFR.

POST /campaigns/{id}/send is itself idempotent: the campaign moves DRAFT → SENDING under a transaction, so a double-clicked Send (or a retried API call) doesn't start two fan-outs.

Step 6 — The core algorithm: fan out without double-sending

Two halves, split exactly along the latency/throughput line. The orchestrator (control plane) walks the list and enqueues each eligible recipient once. The workers (data plane) drain the queue and call the provider.

package dev.fiveyear.newsletter;
 
import java.util.List;
import java.util.function.Consumer;
 
/**
 * The control-plane half of a send: walk the target list and hand each eligible
 * recipient to the queue exactly once. Re-runnable after a crash — `enqueueOnce`
 * skips anyone already claimed, so a resumed fan-out only enqueues the remainder.
 */
public class CampaignOrchestrator {
    private final SendLog log;
    private final SuppressionList suppression;
 
    public CampaignOrchestrator(SendLog log, SuppressionList suppression) {
        this.log = log;
        this.suppression = suppression;
    }
 
    /**
     * In production `recipients` is a keyset-paginated scan of the list, not one
     * giant in-memory batch. `enqueue` drops a task on the durable queue.
     */
    public int fanOut(String campaignId, List<Subscriber> recipients, Consumer<Subscriber> enqueue) {
        int queued = 0;
        for (Subscriber s : recipients) {
            if (!s.subscribed() || suppression.isSuppressed(s.id())) {
                continue; // opt-out and suppression honored before a task is ever made
            }
            if (log.enqueueOnce(campaignId, s.id())) {
                enqueue.accept(s);
                queued++;
            }
        }
        return queued;
    }
}

package dev.fiveyear.newsletter;
 
/**
 * The data-plane half: pull a task, render, and hand the email to the provider
 * under a rate limit, with bounded retry. At-least-once delivery here plus the
 * send log's dedup upstream is what gives effectively exactly-once per recipient.
 */
public class SendWorker {
    private final MailProvider provider;
    private final SuppressionList suppression;
    private final SendLog log;
    private final int maxAttempts;
 
    public SendWorker(MailProvider provider, SuppressionList suppression, SendLog log, int maxAttempts) {
        this.provider = provider;
        this.suppression = suppression;
        this.log = log;
        this.maxAttempts = maxAttempts;
    }
 
    public void handle(String campaignId, Subscriber s, String subject, String body) {
        if (suppression.isSuppressed(s.id())) {            // a bounce may have arrived after fan-out
            log.mark(campaignId, s.id(), SendState.SUPPRESSED);
            return;
        }
        if (log.state(campaignId, s.id()) == SendState.SENT) {
            return;                                         // redelivered message — already done
        }
        for (int attempt = 1; attempt <= maxAttempts; attempt++) {
            try {
                SendResult result = provider.send(s.email(), subject, body);
                if (result == SendResult.HARD_BOUNCE) {
                    suppression.suppress(s.id());
                    log.mark(campaignId, s.id(), SendState.SUPPRESSED);
                } else {
                    log.mark(campaignId, s.id(), SendState.SENT);
                }
                return;
            } catch (TransientFailure transient_) {
                if (attempt == maxAttempts) {
                    log.mark(campaignId, s.id(), SendState.FAILED); // to the dead-letter sweep
                }
                // otherwise: exponential backoff with jitter, then retry
            }
        }
    }
}

Three guards earn the reliability NFR, and each is one line:

• enqueueOnce — a crashed orchestrator re-runs and re-enqueues only the unclaimed; no recipient is queued twice.
• the SENT check — a durable queue delivers at least once, so the same task can arrive twice; the worker that sees SENT does nothing.
• suppression at send time — an unsubscribe or bounce that lands after fan-out but before the worker runs is still honored.

And the worker doesn't get to sprint. Every provider caps your rate; outrun it and you're throttled or blocklisted. A token bucket paces the calls:

package dev.fiveyear.newsletter;
 
/**
 * A token bucket pacing calls to one provider/sending-IP at its allowed rate.
 * The bound is deliverability, not politeness: outrun the limit and the provider
 * throttles or blacklists you.
 */
public class RateLimiter {
    private final double ratePerSec;
    private final double capacity;
    private double tokens;
    private long lastNanos;
 
    public RateLimiter(double ratePerSec, double burst) {
        this.ratePerSec = ratePerSec;
        this.capacity = burst;
        this.tokens = burst;
        this.lastNanos = System.nanoTime();
    }
 
    public synchronized boolean tryAcquire() {
        long now = System.nanoTime();
        tokens = Math.min(capacity, tokens + (now - lastNanos) / 1e9 * ratePerSec);
        lastNanos = now;
        if (tokens >= 1.0) {
            tokens -= 1.0;
            return true;
        }
        return false;
    }
}

Step 7 — Tracking opens and clicks

Opens and clicks come back as plain HTTP hits, so each link has to say who clicked which campaign — without trusting the URL, which anyone can edit. The answer is a signed token: the open pixel and click link carry campaign:subscriber plus an HMAC, and the tracking endpoint verifies the signature before recording anything.

package dev.fiveyear.newsletter;
 
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Optional;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
 
/**
 * Open pixels and click links carry a signed token, not a raw subscriber id.
 * The HMAC lets the tracking endpoint trust who clicked without exposing — or
 * letting anyone forge — the (campaign, subscriber) pair.
 */
public class TrackingToken {
    private final byte[] key;
 
    public TrackingToken(byte[] key) { this.key = key.clone(); }
 
    public String sign(String campaignId, long subscriberId) {
        String payload = campaignId + ":" + subscriberId;
        return payload + ":" + hmac(payload);
    }
 
    /** Returns the trusted "campaignId:subscriberId" payload, or empty if forged. */
    public Optional<String> verify(String token) {
        int cut = token.lastIndexOf(':');
        if (cut < 0) return Optional.empty();
        String payload = token.substring(0, cut);
        String signature = token.substring(cut + 1);
        return constantTimeEquals(signature, hmac(payload)) ? Optional.of(payload) : Optional.empty();
    }
 
    private String hmac(String payload) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(key, "HmacSHA256"));
            byte[] out = mac.doFinal(payload.getBytes(StandardCharsets.UTF_8));
            return Base64.getUrlEncoder().withoutPadding().encodeToString(out);
        } catch (Exception e) {
            throw new IllegalStateException("HMAC failed", e);
        }
    }
 
    private static boolean constantTimeEquals(String a, String b) {
        if (a.length() != b.length()) return false;
        int diff = 0;
        for (int i = 0; i < a.length(); i++) diff |= a.charAt(i) ^ b.charAt(i);
        return diff == 0;
    }
}

A verified hit becomes an event on the stream; the stream feeds ClickHouse for analysis and bumps a Redis counter for the live number. The click endpoint records, then 302s to the real URL so the reader never notices the detour.

Step 8 — Trade-offs (each one keeping an NFR)

The last column is the discipline: every choice keeps one of the promises from Step 2.

The complete implementation

The classes above are the implementation. Here's the driver that exercises them end to end — idempotent fan-out, crash replay, retry-then-suppress, no double-send on redelivery, token verification, and the rate-limit burst:

package dev.fiveyear.newsletter;
 
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
 
public class Main {
    public static void main(String[] args) {
        SendLog log = new SendLog();
        SuppressionList suppression = new SuppressionList();
        CampaignOrchestrator orch = new CampaignOrchestrator(log, suppression);
 
        List<Subscriber> list = List.of(
            new Subscriber(1, "a@x.com", true),
            new Subscriber(2, "b@x.com", false),   // unsubscribed → skipped
            new Subscriber(3, "c@x.com", true),
            new Subscriber(4, "d@x.com", true));
        suppression.suppress(4);                    // prior hard bounce → skipped
 
        List<Subscriber> queue = new ArrayList<>();
        int first = orch.fanOut("camp1", list, queue::add);
        assertTrue(first == 2, "fan-out enqueues only eligible (1 and 3)");
 
        // crash + replay: orchestrator re-runs, must NOT re-enqueue
        int replay = orch.fanOut("camp1", list, queue::add);
        assertTrue(replay == 0, "idempotent replay enqueues nothing");
        assertTrue(queue.size() == 2, "no duplicate tasks after replay");
 
        // a flaky provider: 2 transient failures then accept; subscriber 3 hard-bounces
        MailProvider provider = new MailProvider() {
            int hits = 0;
            public SendResult send(String to, String subject, String body) {
                if (to.equals("c@x.com")) return SendResult.HARD_BOUNCE;
                if (++hits < 3) throw new TransientFailure("provider 503");
                return SendResult.ACCEPTED;
            }
        };
        SendWorker worker = new SendWorker(provider, suppression, log, 5);
        for (Subscriber s : queue) worker.handle("camp1", s, "Hi", "Body");
 
        assertTrue(log.state("camp1", 1) == SendState.SENT, "sub 1 sent after retries");
        assertTrue(log.state("camp1", 3) == SendState.SUPPRESSED, "sub 3 suppressed on hard bounce");
        assertTrue(suppression.isSuppressed(3), "hard bounce added to suppression");
        assertTrue(log.count(SendState.SENT) == 1, "exactly one SENT");
 
        // redelivered queue message for an already-SENT recipient → no second send
        worker.handle("camp1", list.get(0), "Hi", "Body");
        assertTrue(log.count(SendState.SENT) == 1, "redelivery does not double-send");
 
        // tracking token round-trips and rejects tampering
        TrackingToken tok = new TrackingToken("super-secret-key".getBytes());
        String t = tok.sign("camp1", 1);
        assertTrue(tok.verify(t).equals(Optional.of("camp1:1")), "valid token verifies");
        assertTrue(tok.verify(t + "x").isEmpty(), "tampered token rejected");
 
        // rate limiter admits the burst then throttles
        RateLimiter rl = new RateLimiter(1, 3);
        int admitted = 0;
        for (int i = 0; i < 10; i++) if (rl.tryAcquire()) admitted++;
        assertTrue(admitted == 3, "burst of 3 admitted, rest throttled");
 
        System.out.println("ALL NEWSLETTER ASSERTIONS PASSED");
    }
 
    static void assertTrue(boolean cond, String msg) {
        if (!cond) throw new AssertionError(msg);
    }
}

Step 9 — Only now, the boxes

With the core settled, the architecture is just those responsibilities given homes — drawn last, on purpose, so each box exists to serve a requirement rather than the requirements being bent to fit a diagram.

The split to notice: the control plane (API, Postgres, Stats API) stays small and fast; the data plane (orchestrator → durable queue → workers → provider) absorbs the volume; and the tracking plane (webhooks and /o /c → stream → ClickHouse + Redis) runs alongside without touching the send path.

Step 10 — Scaling the design, one bottleneck at a time

Start with one Postgres and one worker, then add machinery only where load actually lands.

• Reads climb first (list previews, stats dashboards) → add a cache and read replicas; the primary keeps the writes.
• Subscribers grow to 100M+ → keep them relational but shard by list/tenant; segment queries stay local to a shard.
• The events firehose dwarfs everything → never put opens/clicks in Postgres; stream to ClickHouse, live numbers in Redis. This is the load that breaks naive designs.
• One giant campaign is a throughput hotspot → partition the queue and run many workers; the orchestrator keyset-paginates the list so the fan-out itself parallelizes.
• The provider becomes the ceiling → multiple sending IPs / providers, each with its own rate bucket; warm new IPs gradually.

Step 11 — When a piece fails: designing for failure

A sender lives or dies on what it does when something breaks mid-campaign. Sort every component into one of three failure shapes and the answers write themselves.

• The orchestrator dies mid fan-out → it's resumable, not restartable: re-run reads the send log and enqueues only the unclaimed. Source-of-truth survives; the crash costs a replay, not a re-send.
• A worker crashes → the durable queue redelivers the task; the SENT guard makes the retry a no-op if it already went out. Nothing lost, nothing doubled.
• The provider is down or throttling → back off and retry; fail over to a secondary provider; the queue buffers in the meantime, so durability holds and the send merely slows.
• The tracking pipeline falls over → it's an optimization, not the truth. Sending is unaffected; open/click stats lag and catch up when the stream drains. Degrade, don't block.
• A provider webhook is lost → reconcile with the provider's event API on a schedule; never assume a single callback is the only record.

The interview corner

• "How do you guarantee exactly-once?" You don't — across a network it's impossible. You do at-least-once delivery + idempotent dedup via the send log's unique key. Say that distinction out loud; it's the whole interview.
• "A campaign is half-sent and the orchestrator crashes." Resume from the send log — enqueueOnce re-enqueues only the unclaimed. Idempotent by construction.
• "Someone unsubscribes one second before the send reaches them." Suppression and subscribed are checked at fan-out and again in the worker; the late opt-out still wins.
• "Where do opens and clicks go?" Not the primary database — a stream into ClickHouse plus Redis counters. The events firehose is the load that decides your storage.
• "Why one task per recipient, not per campaign?" Blast radius: a failure retries one person, not ten million. The per-recipient send log is what makes that affordable.
• "How do you protect sender reputation?" Per-provider rate limiting, immediate hard-bounce suppression, and warming sending IPs — deliverability is a first-class NFR, not an afterthought.

Where to go from here

• New to this format? Start with the rookie's guide to HLD for the method this article follows, beat by beat.
• The send workers are a thread pool; the per-provider pacing is a rate limiter. Both are worth building by hand once.
• For the same fan-out-to-millions problem with a read-time twist, see Twitter's timeline — fan-out on write vs. read is the sibling decision to this one.