Pagination

Endpoints that return lists page with a cursor. The pattern is the same everywhere: send a request without a cursor to get the first page, then send the cursor from that response back to get the next one.

cursor = None
all_posts = []

while True:
    body = {"unique_id": "nasa"}
    if cursor:
        body["cursor"] = cursor

    page = post("tiktok-user-favorites-v4", body)["data"]
    all_posts.extend(page["posts"])

    if not page.get("has_more"):
        break
    cursor = page["cursor"]

Stop on the flag, not on an empty page

Use has_more to decide whether to continue. An empty page in the middle of a feed is normal: platforms filter as they page, so a page can come back with nothing while later pages still hold results.

Stopping the first time you see an empty list will silently truncate your data. Stopping when has_more is false will not.

Guard the loop with a maximum page count anyway. It costs one line and it means a platform bug cannot turn into an unbounded spend.

Cursors are opaque

A cursor is a token to hand back, not a value to read or construct. The format differs by platform and changes without notice. Do not parse one, increment one, or build one yourself.

Cursor lifetime

Most cursors keep working for a long time, so you can store one and resume a feed hours later.

X search cursors are the exception. They go stale after roughly thirty minutes. If you are paging deeply through search results there, page through in one run rather than persisting a cursor for later. Profile cursors on X do not have this problem.

Page size

Where an endpoint takes a count, it is a request rather than a guarantee. Platforms cap page sizes, and a value above the cap is quietly reduced rather than rejected. Read the length of what came back instead of assuming you got what you asked for.

Asking for smaller pages does not save you anything. The cost of a call is mostly in fetching and parsing the page, not in how many rows you keep, so ten pages of ten cost considerably more than one page of a hundred.

Deduplicate on the way in

Live feeds shift while you page them. A post can appear on two consecutive pages if something was published between your calls.

Key your results on post_id as you collect them. It is one set membership check and it removes a whole class of double-counting bug.