Building a feed consumer

Use a feed when your application needs new data shortly after it is published. Instead of repeatedly querying a historical list endpoint, a feed consumer asks for items published after its last saved position.

Each consumer follows the same cycle: request a page, process every item, save the returned cursor, fetch any pages already waiting, then pause before checking again.

Available feeds

Chadwin currently provides two real-time feeds:

• Insider transaction feed: each event contains one newly published Form 4 or Form 5 transaction row
• Institutional filing feed: each event identifies one newly processed Form 13F filing

Start and resume a consumer

Each response includes next_cursor, which marks how far your consumer has progressed through the feed. After processing the page successfully, save this value and send it as cursor on the next request.

For the first request, choose whether to inspect recent events or begin with events published from now on:

• New events only: send start=now. The response establishes a position at the current end of the feed and may contain no items
• Recent events: omit both start and cursor. The response returns the most recent events matching your filters, up to limit, in processing order and establishes a position at the current end of the feed

After the first request, always resume with the stored cursor. Cursor values are opaque: store them exactly as returned without parsing or modifying them.

A cursor records a position in the feed, but it does not record the endpoint or filters used to produce it. Store that configuration with the cursor and reuse it on every request. To consume a different set of events, start a separate consumer with its own cursor; changing filters on an existing cursor will not replay events that were previously excluded.

Process each page

The item collection differs by feed, but every response also returns next_cursor and has_more. Handle the response as one unit:

1. Process every item in the endpoint's collection
2. Use feed_event_id to prevent a retry from creating the same downstream effect twice
3. Save next_cursor only after the entire page succeeds
4. If has_more is true, request the next page immediately
5. If has_more is false, wait 60 seconds before polling again

Filters are applied before the page limit. When has_more is true, another page of matching events is already available after next_cursor.

TypeScript example

Keep the endpoint-specific HTTP request in fetchPage and normalize its item collection to items. This keeps the polling loop independent of the endpoint's response shape.

type FeedItem = { feed_event_id: string };

type FeedPage<T extends FeedItem> = {
  items: T[];
  next_cursor: string;
  has_more: boolean;
};

async function pollFeed<T extends FeedItem>() {
  let cursor = await loadCursor();

  while (true) {
    try {
      const page: FeedPage<T> = await fetchPage(cursor);

      for (const item of page.items) {
        await processOnce(item.feed_event_id, item);
      }

      await saveCursor(page.next_cursor);
      cursor = page.next_cursor;

      if (!page.has_more) await sleep(60_000);
    } catch (error) {
      await waitBeforeRetry(error);
    }
  }
}

processOnce should record the event ID together with the downstream work it creates. If processing fails, the cursor remains unchanged and the page can be attempted again safely.

Errors and recovery

On a network error, 429, 5xx, or local processing failure, keep the last saved cursor and retry with backoff. Honor Retry-After when the API provides it.

If the API rejects a stored cursor, start=now establishes a new future-only position. Doing so abandons any items between the rejected cursor and the new position.