Data Scraping & Aggregation

A Multi-Source News Aggregation Platform Consolidating Nine Publishers Into One Ranked Feed

How Pfactorial Technologies built a news aggregation platform that scrapes, ranks, and serves headlines from nine publishers as one consolidated, searchable feed.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client wanted a single destination for national and international news without asking readers to check nine different publisher sites individually, each with its own layout, categorization, and update cadence.

A simple scraper bolted onto a page of links would not have solved this: the sources differ in markup, some only render content after JavaScript executes, none expose a shared schema or a trustworthy notion of "top story," and every source formats its publication timestamp differently - so freshness has to be computed, not just displayed.

Pfactorial built a scraping and normalization pipeline that consolidates headlines from nine publishers into one ranked, timestamped, category-filterable feed, with a live search layer for on-demand queries beyond the core categories.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to turn fragmented, unreliable third-party sources into one normalized, ranked feed - a repeatable pattern for any client facing multi-source data aggregation, not a one-off scraper.

THE CHALLENGE

Turning nine independently-run publisher sites into one consistent feed meant solving problems that don't show up until a source changes its layout or its notion of a top story.

1. Nine publishers, nine different site structures and update cadences

Each source uses its own markup, category taxonomy, and publishing rhythm, so a single scraper design could not cover all of them - some pages render fine with a simple HTTP request, others only reveal content after JavaScript executes.

2. Publisher layouts change without notice and silently break collection

Several sources have already gone dark for parts of their category set after site redesigns, and a scraper that fails silently just serves stale or empty sections instead of flagging the gap.

3. A fair ranking needs a common signal across incompatible sources

Each publisher's own notion of 'top story' isn't exposed consistently, so ranking has to be derived independently - from placement and image presence - rather than trusted from the source.

4. Freshness has to be computed, not just displayed

Every source reports publication time differently, so relative freshness ('3 hours ago,' '2 days ago') has to be calculated from a normalized timestamp at request time, not passed through from the source.

The real brief Not "scrape a few news sites into a list" but "normalize nine incompatible publishers into one consistently ranked, freshness-aware feed that keeps working when any one of them redesigns their site."

THE SOLUTION

Pfactorial built a two-stage pipeline - a scheduled collection layer that scrapes and normalizes nine publishers, and a serving layer that ranks, categorizes, and searches the normalized result - sitting behind a lightweight API for on-demand queries.

Figure 1 - Every publisher feeds the same normalization and ranking pipeline before a single serving layer renders the categorized, searchable result.

Architectural principles

Normalize once, at collection time, not at render time - Every publisher's idiosyncratic date format, category label, and markup is converted to one common schema the moment it's collected, so the serving layer never has to special-case a source.
Rank is computed, not trusted from the source - Placement and image presence - signals every publisher exposes in some form - drive a simple, consistent rank across all nine sources, instead of relying on each site's own definition of a top story.
Request first, browser-render only when required - Each scraper defaults to a lightweight HTTP request and only falls back to a full Playwright browser instance for sites that require JavaScript execution, keeping collection as fast as the source allows.
A source going dark degrades gracefully, it doesn't take down the feed - Several sources have already gone inactive after layout changes without collection halting for the other eight - inactive sources are a known, monitored state, not a crash.

CAPABILITIES DELIVERED

Each capability turns raw scraped output into something a reader can act on without visiting nine separate sites.

CAPABILITY	WHAT IT DOES
Ranked category feeds	Key Highlights, Business, Sports, and World News each surface the top five stories, sorted by rank then recency.
Full category browsing	An All News view spans every collected category - World, Local, Science, Fashion, and more - under the same rank-then-date sort.
Human-readable freshness	Publication timestamps are converted on the fly into moments-, minutes-, hours-, or days-ago labels rather than raw dates.
Live custom search	A dedicated search endpoint returns ten results for any on-demand query, independent of the fixed category set.
Nine-publisher collection	Inshorts, Hindustan Times, Firstpost, NDTV, Reuters, Economic Times, India Times, Indian Express, and Bing feed the same pipeline.
Source health tracking	Each source-category feed is tracked as active or inactive, so a publisher's layout change is a known, monitored gap rather than a silent one.

Figure 2 - The same normalization and ranking logic covers every category, from the home page highlights to a one-off search query.

Design note Rank is deliberately coarse - 1 or 2, not a continuous score - because placement and image presence are the only signals every publisher reliably exposes; a finer-grained score would imply precision the underlying data doesn't support.

ENGINEERING FOR SCALE AND RELIABILITY

Five decisions keep nine independent, frequently-changing scrapers operable by one team.

Request-based scraping by default, headless browser only on demand

Most sources are collected with a simple HTTP GET parsed as HTML; only sources that require JavaScript execution fall back to Playwright, which keeps the common case fast and the exception case reliable.

Append-to-file backup ahead of every database write

Each script writes its collected batch to a text file in append mode before pushing to the database, so a cancelled or failed run doesn't lose the work already done.

Per-source, per-category functions instead of one generic scraper

Each publisher gets its own extraction functions tailored to its markup, so a layout change on one site is a contained, fixable regression rather than a shared-code break affecting every source.

Timezone-aware conversion to ISO 8601 UTC at the point of collection

Each source's local or IST-formatted timestamp is converted to UTC ISO 8601 immediately during collection, so freshness calculations downstream never have to reason about a source's original timezone.

A standalone API for search, separate from the collection pipeline

The live-search Flask service reads independently of the scheduled scraping jobs, so an on-demand query never waits on or blocks the collection run.

DELIVERY APPROACH

The build proceeded publisher by publisher, establishing the shared schema early so each new source was additive rather than a rewrite.

1. Core schema & first source - establishing the common headline / image / datetime / category / summary / source schema against the first integrated publisher.

2. Request-based scrapers for the majority of sources - Inshorts, Hindustan Times, NDTV, Economic Times, and Indian Express integrated using direct HTTP requests and BeautifulSoup parsing.

3. Playwright fallback for JavaScript-rendered sources - a synchronous Playwright wrapper added for sources whose content only renders after script execution.

4. Ranking and category-serving layer - rank assignment at collection time, plus the home page and full-category views reading by rank and recency.

5. Live search API - a standalone Flask endpoint added for on-demand keyword queries against the same collected store.

RESULTS AND IMPACT

- Key outcomes from this engagement.

The platform is live, actively collecting from nine publishers across thirty-six source-category feeds, and serving ranked, categorized headlines plus on-demand search results to readers.

Because normalization and ranking happen once at collection time rather than being re-derived per view, adding a tenth publisher or reviving an inactive one is a contained scraper change, not a platform redesign.

What it enabled commercially

The client now operates a single news surface instead of maintaining nine separate integrations end-to-end, with source health tracked explicitly so maintenance effort goes to the publishers that actually need it.

WHY PFACTORIAL

This engagement reflects Pfactorial's experience building scraping and normalization pipelines that hold up against sources that change without notice - the same discipline of a common schema, computed rather than trusted signals, and per-source isolation applies to any multi-source data aggregation problem.

- Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with organisations that need fragmented, third-party data sources unified into one reliable, maintainable feed. If you're evaluating a content-aggregation or multi-source scraping platform, we're happy to give you an honest read on scope and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.

Result and Analysis

ENGAGEMENT SNAPSHOT

How Pfactorial Technologies built a news aggregation platform that scrapes, ranks, and serves headlines from nine publishers as one consolidated, searchable feed.