Data Scraping & Aggregation

A Multi-Source Movie Discovery Platform With a Recommendation Engine Rebuilt to Scale

How Pfactorial Technologies built a movie discovery platform for Indian regional cinema audiences, unifying eight scraped data sources behind a recommendation engine redesigned to keep scaling as the catalogue grew.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Pfactorial_Case_Study_CineFinder image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client needed a single place to answer "where can I watch this, and what else is like it" for Indian audiences, with an initial focus on Malayalam cinema - covering OTT releases, theatrical runs, upcoming titles, and full movies studios publish free on YouTube, none of which normally live in one place.

A naive approach breaks down at scale: each source renders and blocks differently, so one generic scraper can't cover all of them, and a similarity matrix computed once over the whole catalogue is quadratic in the number of titles - which stopped being viable as the dataset grew past the platform's first phase.

Pfactorial built a movie discovery platform with source-specific collectors feeding a one-way pipeline into a MySQL-backed Django API, and redesigned the recommendation engine around chunked TF-IDF and cosine-similarity scoring so it could keep scaling past the original in-memory approach - later rebuilding the frontend and widening the pipeline for a third product phase.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to keep a data-pipeline-heavy product working as its catalogue outgrows the architecture it launched with - redesigning the recommendation engine mid-flight rather than treating the original design as fixed.

THE CHALLENGE

Building one coherent catalogue out of eight differently-behaved external sources, and keeping recommendations fast as the dataset grew, meant solving problems a single scraper or a naive similarity matrix couldn't.

1. Every data source breaks differently

IMDb and TMDb tolerate Selenium; BookMyShow and TicketNew render late and need Playwright; Paytm exposes an API that's cheaper and more reliable to call directly than to scrape - each source needed its own collector, not one generic scraper.

2. An in-memory similarity matrix doesn't scale

Computing one similarity matrix over the entire catalogue is quadratic in the number of titles, which stopped being viable once the dataset grew past the platform's first phase.

3. Scraped data is inconsistent by nature

Genre and title strings differ in capitalisation and formatting between sources, so matching logic has to tolerate that rather than assume clean, uniform input.

4. A broken scraper shouldn't be able to take the product down

Nothing in the request path touches a scraper directly, so a blocked or broken source has to degrade data freshness, not availability, for users already on the site.

The real brief Not "scrape a few sites into a database" but "build a pipeline that keeps working, and a recommendation engine that keeps scaling, as the catalogue and the source list both grow."

THE SOLUTION

Pfactorial built a one-way data pipeline - source-specific scrapers into cleaned CSV, into MySQL, served by a Django REST API to a React client - with a chunked TF-IDF and cosine-similarity recommendation engine that replaced the original in-memory approach once it stopped scaling.

Pfactorial_Case_Study_CineFinder image 2

Figure 1 - Data flows one way, from source to screen; a broken collector degrades freshness, never availability.

Architectural principles

One-way data flow, request path never touches a scraper - Scrapers write to CSV, CSV loads to MySQL, and the API only ever reads from MySQL - so a blocked or broken source degrades freshness, not the site's availability.
A collector per source, not one generic scraper - IMDb and TMDb suit Selenium, BookMyShow and TicketNew need Playwright's late-render handling, and Paytm's API is called directly - matching the tool to how each source actually behaves.
Case-insensitive matching everywhere scraped data is queried - Title and genre searches use containment matching rather than exact matching, because scraped strings vary in capitalisation and formatting across sources.
Recommendations computed offline, served as a lookup - Similarity scoring happens in the pipeline, not at request time - the API reads precomputed scores out of the database, so no vectorisation runs while a user is waiting.

CAPABILITIES DELIVERED

Each capability draws on the same underlying pipeline and data model, from search to recommendations to theatre listings.

CAPABILITY	WHAT IT DOES
Movie & show detail pages	Poster, description, IMDb rating, genre, release date, cast with character names and headshots, embedded trailer, reviews, and platform links.
Multi-actor cast search	Search by combining several actor names in one query, following the cast-to-actor relation to return shared titles.
Genre and attribute filtering	Case-insensitive genre listings with filters across release year, IMDb rating, audience certificate, language, and show type.
Free movies on YouTube	A catalogue of full-length movies production houses publish free on YouTube, filterable by channel.
Theatre and upcoming releases	Now-showing titles filtered by city with booking links out to ticketing partners, plus upcoming theatre and OTT releases filterable by language.
Content-based recommendations	Chunked TF-IDF and cosine-similarity scoring surfaces similar titles, precomputed and served as a fast lookup.
User accounts and reviews	Sign-in and review submission, with reviews joined to the movie record and shown on the detail page.

Pfactorial_Case_Study_CineFinder image 3

Figure 2 - The same pipeline pattern covers search, filtering, and recommendations alike.

Design note Chunking the similarity computation and keeping only the top 25 scores per title, rather than persisting the full pairwise matrix, is the tradeoff that let the recommendation engine survive the catalogue outgrowing its phase-1 design - it costs some recommendation depth for a computation that stays bounded as titles are added.

ENGINEERING FOR SCALE AND RELIABILITY

Five decisions kept the pipeline resilient and the recommendation engine scalable as the product moved through three phases.

Source-specific collectors instead of one generic scraper

Selenium, Playwright, BeautifulSoup, and direct HTTP requests are each used where they fit best - Playwright for sources that render late, a direct API call for Paytm rather than scraping the same data off a page - because a single scraping strategy doesn't hold up across eight differently-behaved sources.

Chunked similarity scoring instead of one whole-catalogue matrix

The original phase-1 design held one similarity matrix over the entire catalogue in memory; phase 2 split the data into chunks, scored each separately, and merged results, because the whole-catalogue matrix is quadratic in the number of titles.

A one-way pipeline, not a live path from scraper to user

Nothing in the request path touches a scraper, so a blocked or broken source degrades data freshness rather than site availability - the API only ever reads from MySQL.

Google Generative AI for unstructured source text, not for everything

Structured fields are extracted with a Generative AI model only where a source gives prose instead of clean fields, such as upcoming-release descriptions - not used as a blanket extraction layer.

Case-insensitive containment matching as the default query pattern

Title and genre lookups use containment matching rather than exact match throughout, because scraped genre and title strings differ in capitalisation between sources and exact matching would silently drop results.

DELIVERY APPROACH

The platform shipped across three phases, with the second and third phases driven by the catalogue outgrowing the prior design.

1. Phase 1 - initial catalogue and matrix-based recommendations - launching with a single in-memory similarity matrix over the early catalogue.

2. Phase 2 - chunked recommendation engine - redesigning the similarity computation into chunks once the whole-catalogue matrix stopped being viable, merging and ranking scores per title.

3. Source collector build-out - standing up the eight source-specific collectors - Selenium, Playwright, BeautifulSoup, direct API - and the pandas cleaning and deduplication stage.

4. Data model and API - building the MySQL schema and Django REST API, including the movie, cast, actor, reviews, and watch-link relations.

5. Phase 3 - frontend rebuild - rebuilding the React client and widening the data pipeline under the platform's third-phase rebrand, alongside the policy and content work needed for ad-network approval.

RESULTS AND IMPACT

Pfactorial_Case_Study_CineFinder image 4

- Key outcomes from this engagement.

The platform now unifies eight independently-behaved data sources - OTT catalogues, IMDb, ticketing sites, and YouTube - into one searchable, filterable catalogue, with recommendations served from a chunked TF-IDF and cosine-similarity engine that replaced the original in-memory matrix once the dataset outgrew it.

Because the request path never touches a scraper, a blocked or broken source degrades data freshness rather than availability, and because matching is case-insensitive throughout, inconsistent scraped data doesn't silently break search or genre filtering for users.

What it enabled commercially

The client now has a data pipeline and recommendation architecture built to keep scaling as both the catalogue and the source list grow, rather than one sized only for the dataset it launched with - the redesign that got them through phase 2 is the same pattern that will carry the next order of magnitude of titles.

WHY PFACTORIAL

This engagement reflects Pfactorial's practice of building data-pipeline-heavy products that keep working as they scale - redesigning a core algorithm mid-flight when the original approach stops holding up, rather than treating architecture as fixed once shipped.

Pfactorial_Case_Study_CineFinder image 5

- Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with organisations building catalogue or discovery products stitched together from multiple external data sources. If you're scoping a platform that depends on scraping, aggregation, or a recommendation engine that needs to keep scaling, we're happy to talk through the architecture before anyone commits. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.