Keeping a copy in sync

Most apps keep their own copy of the stories for their tickers. Webhooks arrive more than once and out of order (see Receiving webhooks), and you will usually handle several at once. Simple code goes wrong in ways that are hard to spot:

• A late story.updated overwrites a newer one.
• A story merges away and you delete it; then a late story.created, or a slow API response that read the story before it closed, saves it again.
• Two handlers both check "is this newer?", both see yes, and write in the wrong order.

Checking first and writing second leaves a gap between the two steps, and deduplicating event_ids doesn't close it. Processing every change to a story strictly one at a time would, but it is easy to get wrong across processes. Simpler: make each write a single statement that checks and writes at once. The code on this page is from the reference consumer in the newsfilter repository (examples/consumer), whose tests deliver events late, twice and concurrently, with slow and stale API reads, and check the copy ends up matching the API.

The tableCopy Anchor Link

Keep one row per story, and keep it when the story closes: the row is what stops a closed story coming back.

create table stories (
  id text primary key,
  ticker text,
  version integer not null,
  closed boolean not null default false,
  story jsonb            -- the snapshot; null once closed
);

The two writesCopy Anchor Link

Save a snapshot only if it is newer than the row and the row isn't closed. Postgres checks the condition and writes in one step, even when two handlers race for the same story:

insert into stories (id, ticker, version, closed, story)
values ($1, $2, $3, false, $4)
on conflict (id) do update
  set ticker = excluded.ticker, version = excluded.version,
    story = excluded.story
  where not stories.closed and stories.version < excluded.version

Close a story for good. version is the closing version from story.closed, or 0 when all you know is that the API answered 404:

insert into stories (id, version, closed, story)
values ($1, $2, true, null)
on conflict (id) do update
  set closed = true, story = null,
    version = greatest(stories.version, excluded.version)

With these two writes, applying an event twice changes nothing, and the order events arrive in doesn't matter. Your copy's correctness no longer depends on deduplicating event_ids.

Getting the snapshotCopy Anchor Link

Either way below uses the same two writes. Pick one.

Apply the snapshot in the event. No API calls:

async function applySnapshot(event: Event) {
  switch (event.type) {
    case 'story.created':
    case 'story.updated':
    case 'story.became_strong':
      return saveStory(event.data.story);
    case 'story.closed':
      await closeStory(event.data.story_id, event.data.version);
      if (event.data.merged_into) await saveStory(event.data.merged_into);
      return;
  }
}

Or fetch the story when notified. Each event means "these stories changed"; ask the API for their current state. A 404 means closed:

async function fetchOnNotify(event: Event) {
  for (const id of storiesOf(event)) {
    const story = await getStory(id); // null on 404
    if (story) await saveStory(story);
    else await closeStory(id, null);
  }
}

storiesOf is the story in the event: data.story.id, data.article.story_id for article.shortlisted, and for story.closed both data.story_id and the surviving story. A response that is slow because it read the story before a change can't undo that change: an older version fails the version check, and a closed row stays closed.

Fetching costs one API call per event and always gets the newest state, which can save you applying several intermediate versions. Applying snapshots needs no calls.

Merge chains need nothing special. If A merges into B and B then merges into C, A and B are closed rows and C holds their articles, whatever order the events arrive in.

NotificationsCopy Anchor Link

Writes that are safe to repeat are not enough for side effects that must happen once, such as notifying a user. Claim them with a single insert:

create table claimed (key text primary key);

insert into claimed (key) values ('strong:' || $1)  -- the story id
on conflict do nothing
returning key

Notify only if a row came back. Claim before notifying: if your process dies in between, the user misses one notification rather than getting it twice. Skip notifications for events whose origin is backfill.

Catch-upCopy Anchor Link

GET /v1/events returns every event for your tickers from the last 30 days, in the order they were published. Use it after downtime, on start-up, and every few minutes as a safety net under webhooks.

let cursor = await loadCursor(); // null the first time
for (;;) {
  let page;
  try {
    page = await listEvents(cursor);
  } catch (err) {
    if (!(err instanceof CursorExpiredError)) throw err; // 410
    await reloadEverything();
    cursor = null;
    continue;
  }
  for (const event of page.data) await onEvent(event); // the same handler
  cursor = page.next_cursor;
  await saveCursor(cursor); // only after the page is applied
  if (!page.has_more) break;
}

• Use the same handler for webhooks and catch-up. Seeing an event both ways is harmless.
• Save the cursor after applying the page, so a crash replays it instead of skipping it.
• Call at least once every 30 days. An older cursor answers 410: reload in full.

Adding a tickerCopy Anchor Link

Webhooks carry events for a ticker from the moment you subscribe to it, and catch-up doesn't replay what came before your cursor. When you add a ticker, load its current stories once with GET /v1/stories; events keep it up to date from there.

Reloading in fullCopy Anchor Link

To rebuild your copy, for example after a 410 or lost webhooks:

1. For each ticker, list its stories with GET /v1/stories and save each one with the snapshot write.
2. For each story your copy still has open that the listing didn't return, fetch it: save it if it exists (it was created during the reload), close it on 404.
3. Clear the catch-up cursor and catch up from the start of the window.

Keep everything else while you do it. Closed rows stop a reload, or a replayed event, from bringing a story back, and claimed notifications stop users being notified twice. Webhooks can keep arriving during the reload: every write is the same safe write.