Realtime API News: Build a Live News Feed by Polling the GNews API (Python, JS, cURL)

October 12, 2026 · GNews Team

Realtime API news with GNews works by polling, not push: GNews is a REST API with no websocket or webhook, and on paid plans new articles become available in real time, so you call the search endpoint newest-first on a schedule and ask only for what was published since your last call. This tutorial builds that loop in Python, JavaScript and cURL: a from watermark, deduplication by article id, a polling interval sized to your plan's quota, and handling for 429 and 403 errors. One caveat up front: the Free plan serves articles with a 12-hour delay, so use it to build the loop and a paid plan to run it live.

What "realtime API news" means with GNews: polling, not push

GNews is a REST API: every request is a GET to https://gnews.io/api/v4/search or https://gnews.io/api/v4/top-headlines, and the GNews documentation describes no websocket, streaming endpoint, webhook or other push channel. Real-time API news with GNews therefore has two halves. The first is on the GNews side: on the Essential, Business and Enterprise plans, newly published articles become available in real time, while the Free plan serves them with a 12-hour delay (article availability). The second is on your side: your app polls search sorted by publishedAt, asks only for articles published since its last poll with the from parameter, drops the ones it has already seen by id, and keeps the polling interval inside the plan's daily quota and per-second limit. End-to-end freshness is the time an article takes to reach the index plus your polling interval. GNews does not publish an indexing latency figure, so measure freshness on your own queries instead of assuming one.

What you'll build

A small poller that turns one GNews query into a news API feed your app can read. After a baseline poll, each poll asks only for articles newer than a watermark, skips the ones already delivered, pages through overflows and backs off when rate-limited. New articles land in new_articles.jsonl (one per line) for a dashboard, alerting job or bot to read; a state file survives restarts, so the script runs as a loop or from cron with --once.

The demo query is the exact phrase "data breach", a typical security-monitoring topic. For a refresher on what a news API returns, see News API: what it returns, how to choose one and how to call it.

Prerequisites

A GNews API key. The free plan is enough to build and test the loop; a paid plan is what makes the data real-time (pricing).
Python 3.9+ (standard library only), Node.js 18+ (built-in fetch) and curl.
Your key in an environment variable, so it never lands in your code: export GNEWS_API_KEY="YOUR_API_KEY".

Step 1: Make the first poll with `search`, newest first

The first poll has no watermark yet: it asks for the latest matching articles. sortby=publishedAt is already the documented default for search (search endpoint reference); setting it explicitly makes the intent visible in your code.

Python

import json, os
from urllib.parse import urlencode
from urllib.request import urlopen

params = {
    "q": '"data breach"',
    "lang": "en",
    "max": 5,
    "sortby": "publishedAt",
    "apikey": os.environ["GNEWS_API_KEY"],
}
with urlopen("https://gnews.io/api/v4/search?" + urlencode(params), timeout=30) as r:
    data = json.load(r)

print(data["totalArticles"], "matching articles")
for a in data["articles"]:
    print(a["publishedAt"], a["source"]["name"], a["title"])

JavaScript (Node.js 18+, save as `first-poll.mjs`)

const params = new URLSearchParams({
  q: '"data breach"',
  lang: "en",
  max: "5",
  sortby: "publishedAt",
  apikey: process.env.GNEWS_API_KEY,
});
const res = await fetch(`https://gnews.io/api/v4/search?${params}`);
const data = await res.json();

console.log(data.totalArticles, "matching articles");
for (const a of data.articles) console.log(a.publishedAt, a.source.name, a.title);

cURL

curl -s --get "https://gnews.io/api/v4/search" \
  --data-urlencode 'q="data breach"' \
  -d lang=en -d max=5 -d sortby=publishedAt \
  -d "apikey=$GNEWS_API_KEY"

This request returned five articles, newest first, out of 6,343 matches (see Example response below).

Why poll `search` and not `top-headlines`

top-headlines returns a category's articles in an order the docs describe as "based on the Google News ranking", with no sortby parameter (top-headlines endpoint reference). That makes it a ranking, not a timeline: good for a "trending in tech" widget, like this request that returned five US technology headlines:

curl -s "https://gnews.io/api/v4/top-headlines?category=technology&lang=en&country=us&max=5&apikey=$GNEWS_API_KEY"

An incremental news API feed needs a reliable newest-first order and a query you control, which is what search with sortby=publishedAt provides.

Step 2: Fetch only news API updates since your last poll

Use the newest `publishedAt` you have seen as `from`

Send the newest publishedAt you have received (your watermark) as from on the next poll, in ISO 8601 UTC such as 2026-10-11T13:08:56Z. publishedAt is always UTC, so there is no timezone conversion. (to closes the window at the other end; a live loop does not need it.)

`from` is inclusive, so deduplicate by `id`

Per the docs, from means "greater than or equal to", so the article on the watermark comes back. Deduplicate on articles[].id, the article's unique identifier, keeping the last 1,000 seen ids.

Overlap the window by a few minutes

Set from five minutes before the watermark. This is defensive, not documented GNews behavior: an article indexed after a later-published one would otherwise be skipped. Deduplication absorbs the repeats.

Page through a poll that comes back full

If a poll returns exactly max articles, request page=2, 3 and so on until a page comes back short. Pagination stops at 1,000 articles and each page costs a request, so the code caps it at five. That cap is the one known hole: if more than MAX × MAX_PAGES new articles (25 with the defaults) arrive between two polls, the oldest of them are skipped. Poll more often or raise max if your query is that busy. Never paginate the first poll: without from, "the rest" is the whole archive.

The watermark pattern in one paragraph. To fetch only news API updates since your last poll, store two things: the newest publishedAt you have received (the watermark) and the ids of the articles you have already delivered. Each poll calls search with sortby=publishedAt and from set to the watermark minus a few minutes, then drops every article whose id is already in the seen set. Because from is inclusive and the window overlaps, a poll usually returns at least one article you already have; that is expected and costs nothing once deduplication runs. If a poll returns a full page, request the next pages until one comes back short, within a small cap that protects your daily quota (anything beyond that cap is skipped, so size max and the interval to your query's volume). When the response is empty, nothing new matched and the watermark stays where it is. Persist both values to disk, so a restart neither re-emits old articles nor opens a gap in the feed.

Python: the full poller

Save as poller.py, then run python3 poller.py for a long-running loop or python3 poller.py --once for a single poll (from cron, for instance).

import json, os, sys, time
from datetime import datetime, timedelta, timezone
from urllib.error import HTTPError
from urllib.parse import urlencode
from urllib.request import urlopen

API_KEY = os.environ["GNEWS_API_KEY"]
QUERY = '"data breach"'
MAX = 5            # up to your plan's cap: 10 Free, 25 Essential, 50 Business, 100 Enterprise
MAX_PAGES = 5      # each extra page costs one request from the daily quota
INTERVAL = 900     # seconds between polls, see Step 3
OVERLAP = timedelta(minutes=5)
STATE_FILE = "poller_state.json"
OUT_FILE = "new_articles.jsonl"

class QuotaReached(Exception):
    pass

def get(params):
    url = "https://gnews.io/api/v4/search?" + urlencode({**params, "apikey": API_KEY})
    for attempt in range(5):
        try:
            with urlopen(url, timeout=30) as r:
                return json.load(r)
        except HTTPError as e:
            body = e.read().decode()
            if e.code == 403:
                raise QuotaReached(body)
            if e.code == 503:
                time.sleep(180)
            elif e.code in (429, 500):
                time.sleep(2 ** attempt)  # 1, 2, 4, 8, 16 s
            else:
                sys.exit(f"GNews error {e.code}: {body}")
    raise RuntimeError("GNews still failing after 5 attempts")

def poll(state):
    params = {"q": QUERY, "lang": "en", "max": MAX, "sortby": "publishedAt"}
    pages = 1  # first run: take one page as the baseline, do not page through the archive
    if state["watermark"]:
        since = datetime.fromisoformat(state["watermark"].replace("Z", "+00:00")) - OVERLAP
        params["from"] = since.strftime("%Y-%m-%dT%H:%M:%SZ")
        pages = MAX_PAGES

    articles = []
    for page in range(1, pages + 1):
        if page > 1:
            params["page"] = page
        batch = get(params)["articles"]
        articles += batch
        if len(batch) < MAX:
            break

    seen = set(state["seen"])
    new = {a["id"]: a for a in articles if a["id"] not in seen}
    for a in articles:
        if not state["watermark"] or a["publishedAt"] > state["watermark"]:
            state["watermark"] = a["publishedAt"]
    state["seen"] = (state["seen"] + list(new))[-1000:]
    return list(new.values())

def load_state():
    try:
        with open(STATE_FILE) as f:
            return json.load(f)
    except FileNotFoundError:
        return {"watermark": None, "seen": []}

def seconds_until_midnight_utc():
    now = datetime.now(timezone.utc)
    midnight = (now + timedelta(days=1)).replace(hour=0, minute=0, second=0, microsecond=0)
    return (midnight - now).total_seconds()

def main():
    once = "--once" in sys.argv
    state = load_state()
    while True:
        wait = INTERVAL
        try:
            new = poll(state)
            with open(OUT_FILE, "a") as out:
                for a in reversed(new):  # oldest first
                    print(a["publishedAt"], a["source"]["name"], a["title"], sep="  ")
                    out.write(json.dumps(a) + "\n")
            with open(STATE_FILE, "w") as f:
                json.dump(state, f)
            print(f"{len(new)} new, watermark {state['watermark']}")
        except QuotaReached as e:
            print("Daily quota reached, pausing until 00:00 UTC:", e)
            wait = seconds_until_midnight_utc()
        except (OSError, RuntimeError) as e:
            print("Poll failed, retrying next cycle:", e)
        if once:
            break
        time.sleep(wait)

if __name__ == "__main__":
    main()

We ran it twice with --once on 2026-10-12 at 06:33 UTC. The first run made the baseline poll and printed five articles, oldest first; the second sent from=2026-10-11T13:03:56Z and delivered nothing twice:

$ python3 poller.py --once
2026-10-10T22:02:16Z  EUROPE SAYS  ASOS Confirms Data Breach After Rogue App Alert
2026-10-10T23:30:13Z  Fox News  What scammers can really do with your name, phone number and address
2026-10-11T00:00:00Z  Around Osceola  NEWS BRIEFS - New Florida DL/ID program; Potential Park Place data breach; new Valencia/FAMU academic agreement
2026-10-11T02:25:07Z  New York Post  Cash App users could receive up to $2,500 in data breach $15M settlement
2026-10-11T13:08:56Z  EUROPE SAYS  Cash App users could receive up to $2,500 in data breach $15M settlement
5 new, watermark 2026-10-11T13:08:56Z
$ python3 poller.py --once
0 new, watermark 2026-10-11T13:08:56Z

JavaScript: the full poller

Same logic for Node.js 18+. Save as poller.mjs and run node poller.mjs (add --once for a single poll).

import { readFileSync, writeFileSync, appendFileSync } from "node:fs";
import { setTimeout as sleep } from "node:timers/promises";

const API_KEY = process.env.GNEWS_API_KEY;
const QUERY = '"data breach"';
const MAX = 5;          // up to your plan's cap: 10 Free, 25 Essential, 50 Business, 100 Enterprise
const MAX_PAGES = 5;    // each extra page costs one request from the daily quota
const INTERVAL = 900;   // seconds between polls, see Step 3
const OVERLAP_MS = 5 * 60 * 1000;
const STATE_FILE = "poller_state.json";
const OUT_FILE = "new_articles.jsonl";

class QuotaReached extends Error {}

async function get(params) {
  const url = "https://gnews.io/api/v4/search?" + new URLSearchParams({ ...params, apikey: API_KEY });
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(url);
    if (res.ok) return res.json();
    const body = await res.text();
    if (res.status === 403) throw new QuotaReached(body);
    if (res.status === 503) await sleep(180_000);
    else if (res.status === 429 || res.status === 500) await sleep(2 ** attempt * 1000);
    else {
      console.error(`GNews error ${res.status}: ${body}`);
      process.exit(1);
    }
  }
  throw new Error("GNews still failing after 5 attempts");
}

async function poll(state) {
  const params = { q: QUERY, lang: "en", max: MAX, sortby: "publishedAt" };
  let pages = 1; // first run: take one page as the baseline, do not page through the archive
  if (state.watermark) {
    params.from = new Date(Date.parse(state.watermark) - OVERLAP_MS).toISOString();
    pages = MAX_PAGES;
  }

  const articles = [];
  for (let page = 1; page <= pages; page++) {
    if (page > 1) params.page = page;
    const batch = (await get(params)).articles;
    articles.push(...batch);
    if (batch.length < MAX) break;
  }

  const seen = new Set(state.seen);
  const fresh = new Map();
  for (const a of articles) if (!seen.has(a.id)) fresh.set(a.id, a);
  for (const a of articles) {
    if (!state.watermark || a.publishedAt > state.watermark) state.watermark = a.publishedAt;
  }
  state.seen = [...state.seen, ...fresh.keys()].slice(-1000);
  return [...fresh.values()];
}

function loadState() {
  try {
    return JSON.parse(readFileSync(STATE_FILE, "utf8"));
  } catch (e) {
    if (e.code !== "ENOENT") throw e;
    return { watermark: null, seen: [] };
  }
}

function secondsUntilMidnightUtc() {
  const now = new Date();
  return (Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate() + 1) - now) / 1000;
}

const once = process.argv.includes("--once");
const state = loadState();
while (true) {
  let wait = INTERVAL;
  try {
    const fresh = await poll(state);
    for (const a of fresh.reverse()) { // oldest first
      console.log(`${a.publishedAt}  ${a.source.name}  ${a.title}`);
      appendFileSync(OUT_FILE, JSON.stringify(a) + "\n");
    }
    writeFileSync(STATE_FILE, JSON.stringify(state));
    console.log(`${fresh.length} new, watermark ${state.watermark}`);
  } catch (e) {
    if (e instanceof QuotaReached) {
      console.log("Daily quota reached, pausing until 00:00 UTC:", e.message);
      wait = secondsUntilMidnightUtc();
    } else {
      console.log("Poll failed, retrying next cycle:", e.message);
    }
  }
  if (once) break;
  await sleep(wait * 1000);
}

cURL: one incremental poll

The second poll from the run above, by hand. Add -d page=2 when a response comes back full.

curl -s --get "https://gnews.io/api/v4/search" \
  --data-urlencode 'q="data breach"' \
  -d lang=en -d max=5 -d sortby=publishedAt \
  -d from=2026-10-11T13:03:56Z \
  -d "apikey=$GNEWS_API_KEY"

Step 3: Pick a polling interval your plan can afford

Interval = 86,400 seconds divided by requests per day, for one query with no retries or extra pages. Sources: pricing, article availability, error handling. The daily quota resets at 00:00 UTC.

Plan	Requests/day	Fastest steady interval, 1 query	Per-second limit	Freshness	Max articles/request
Free	100	~14.4 min (864 s)	1 req/s	12-hour delay	10
Essential	1,000	~86 s	10 req/s	Real-time	25
Business	5,000	~17 s	10 req/s	Real-time	50
Enterprise	25,000	~3.5 s	10 req/s	Real-time	100

Free-plan trap: a fixed window such as from = now - 15 minutes returns an empty articles array on Free, because of the 12-hour delay ("asking for a period outside your plan is not an error"). A watermark based on the last publishedAt received works on every plan.

How often should you poll a news API? Start from the daily quota, not from how fresh you would like the data to be. Divide 86,400 seconds by your plan's requests per day to get the fastest steady interval for one query: about 14.4 minutes on the GNews Free plan (100 requests), 86 seconds on Essential (1,000), 17 seconds on Business (5,000) and 3.5 seconds on Enterprise (25,000). Multiply that interval by the number of queries you poll, then leave a margin for extra pages and retries; 20 to 30 percent is a comfortable starting point. Stay under the per-second limit as well: 1 request per second on Free, 10 on paid plans. On the Free plan, articles arrive with a 12-hour delay anyway, so a 15-minute interval is plenty for building and testing.

Step 4: Handle 429, 403 and empty results without breaking the loop

The get() function in both pollers maps each documented status code (error handling) to one action. Error bodies are JSON with an errors key (for a bad parameter, an object such as {"errors": {"<param>": "<message>"}}) and get printed as is.

Status	Meaning per the docs	What the poller does
429	Too many requests per second (1 Free, 10 paid)	Backs off 1, 2, 4, 8, 16 s, then skips the cycle
403	Daily quota reached (or subscription expired)	Sleeps until 00:00 UTC, no retry loop
500	Server error, retry after a short delay	Same backoff as 429
503	Maintenance	Waits 3 minutes, then retries
400, 401	Bad parameter, missing or unknown key	Exits with the error: retrying cannot fix it
200, empty `articles`	Nothing new	Keeps the watermark, waits for the next poll

Network errors and exhausted retries are logged and the loop moves on; the watermark only advances after a successful poll, so a failed cycle never opens a gap.

Example response

The first poll from Step 1, fetched at 2026-10-12T06:31:37Z: two of the five articles, every field kept, long content strings shortened (marked ...).

{
  "totalArticles": 6343,
  "articles": [
    {
      "id": "9e96e7d4bee76dc4157c00f8d54b7f19",
      "title": "Cash App users could receive up to $2,500 in data breach $15M settlement",
      "description": "Eligible Cash App users could soon begin receiving $2,500 in their pockets as part of a whopping $15 million law suit settlement over a data breach.",
      "content": "Eligible Cash App users could soon begin receiving $2,500 in their pockets as part of a whopping $15 million law suit settlement over a data breach. ...",
      "url": "https://www.europesays.com/3304091/",
      "image": "https://www.europesays.com/wp-content/uploads/2026/10/1791724138.jpeg",
      "publishedAt": "2026-10-11T13:08:56Z",
      "lang": "en",
      "source": {
        "id": "ed23d65928efaec6669dd2212a773d9f",
        "name": "EUROPE SAYS",
        "url": "https://www.europesays.com",
        "country": "us"
      }
    },
    {
      "id": "ded8a22b147a78d29fc3e2d27742fcf3",
      "title": "Cash App users could receive up to $2,500 in data breach $15M settlement",
      "description": "Eligible Cash App users could receive up to $2,500 this month following a $15 million class-action settlement over an alleged 2021 data breach.",
      "content": "Eligible Cash App users could soon begin receiving $2,500 in their pockets as part of a whopping $15 million law suit settlement over a data breach. ...",
      "url": "https://nypost.com/2026/10/10/us-news/cash-app-users-could-receive-up-to-2500-in-data-breach-15m-settlement/",
      "image": "https://nypost.com/wp-content/uploads/sites/2/2026/10/AdobeStock_510707080_Editorial_Use_Only-copy.jpg?quality=75&strip=all&w=1200",
      "publishedAt": "2026-10-11T02:25:07Z",
      "lang": "en",
      "source": {
        "id": "95a96a1867c4dac6f720e9d08d4c1b96",
        "name": "New York Post",
        "url": "https://nypost.com",
        "country": "us"
      }
    }
  ]
}

The fields the loop relies on:

totalArticles: total matches (6,343), not the number returned (5). Do not use it to detect new articles.
articles[].id: unique identifier, the deduplication key.
articles[].publishedAt: always UTC, the watermark source.
articles[].url and source.name: what your app shows. source.country exists only on search, not on top-headlines.

Note the first two articles: the same Cash App story and title from two sources, with two different ids. Deduplicating by id stops one article from being delivered twice; it does not merge a story several outlets publish. If that matters, add a check on a normalized title.

About freshness: at 06:31:37 UTC, the newest matching article had a publishedAt of 13:08:56 UTC the day before, 17 hours and 22 minutes earlier. That gap is not an indexing latency measurement: it combines how recently a matching article was published and your plan's delay (12 hours on Free, real-time on paid plans).

FAQ

Does GNews have a websocket or streaming API for real-time API news? No. GNews is a REST API; its docs describe no websocket, streaming endpoint or webhook. For real-time API news, poll search newest-first with a from watermark, deduplicate by id, and use a paid plan, where articles are available in real time.

How fresh are GNews articles on the free plan? A 12-hour delay, over the last 30 days. Essential, Business and Enterprise are real-time, with history since 2020 (article availability, pricing). Comparing providers? See Top 5 news APIs in 2026.

How often should I poll a news API? Divide 86,400 seconds by your daily requests, then by the number of queries, and keep headroom for retries. For one query: about every 15 minutes on Free, every 86 seconds on Essential (table in Step 3).

How do I avoid duplicate articles when polling? from is inclusive, so the article on the watermark comes back on the next poll. Keep a set of seen id values and skip any article already in it, so your API feed never shows the same article twice. Overlapping the window by a few minutes then costs nothing.

What happens if I exceed the rate limit or the daily quota? Too many requests per second returns 429: back off exponentially. An exhausted daily quota returns 403: stop until the 00:00 UTC reset (error handling).

Next steps

Post each new article to a chat channel: pull headlines with GNews and post them to Slack or Discord.
Share them on Mastodon at a human pace: auto-post fresh articles with GNews and Mastodon.
Run the same loop on a ticker watchlist: track markets and tickers with GNews.
Tune the query itself (in, country, query operators) with the search endpoint reference.

Ready to build? Start building for free: create a key, run python3 poller.py --once, and watch the watermark move.