Recruiting & HR Tech

A No-API Data Collection Pipeline That Sources 29,000+ Lift Equipment Listings From Public Marketplaces

How Pfactorial Technologies proved that lift equipment listing data - specifications, pricing, images and dealer details - can be collected reliably from public marketplaces without a paid API, for an online equipment marketplace platform.

August 21, 2026

OUTLINE

ENGAGEMENT SNAPSHOT

Pfactorial_Case_Study_LiftsToday_Data_Sourcing image 1

Figure 1 - Key figures from this engagement, at a glance.

EXECUTIVE SUMMARY

Our client, an online equipment marketplace platform, needed to know whether lift equipment listing data - specifications, pricing, images and dealer details - could be sourced without a paid API, across forklifts, telehandlers, boom lifts, scissor lifts and attachments.

The real question was not whether one page could be scraped once; it was whether data from differently structured sites could be collected reliably, validated field by field, and normalized into one consistent structure the platform could actually import, without silent failures that leave the dataset subtly wrong.

Pfactorial assessed candidate public marketplaces on inventory volume, completeness and accessibility, then built dedicated collectors against the two strongest sources, validating every field before it is written and covering both collectors with an automated offline test suite.

Why this engagement is representative This engagement demonstrates Pfactorial's ability to replace an expensive or unavailable paid data API with a validated, self-built collection pipeline against public sources - proving the approach on live inventory rather than a scripted demo.

THE CHALLENGE

Proving out an API-free data source meant solving problems that don't show up until the pipeline has to run against real, differently structured sites.

1. No official API exists for the target category

Lift equipment listings had to be sourced by direct collection against public marketplace pages rather than a documented, stable API contract.

2. Every source structures its data differently

The collection approach had to transfer from a national aggregator with a full specification block to a smaller, differently constructed single-dealer site without changing the delivered structure.

3. A silently wrong record is worse than a missing one

Condition, availability, pricing and measurements all needed a second-signal check before being accepted, so the dataset stays consistent rather than plausible-looking but subtly incorrect.

4. The output has to be import-ready, not just collected

Every listing needed to land in one consistent 35-field structure regardless of source, with price-on-request and missing fields handled explicitly rather than as inconsistent nulls or zeros.

The real brief Not "scrape a page and see what comes back" but "prove a validated, import-ready data pipeline against real public sources, with every field checked before it's written and the approach ready to extend to a third source without touching anything downstream."

THE SOLUTION

Pfactorial evaluated candidate public marketplaces and built dedicated collectors against the two strongest sources, writing every record into one shared, validated output structure regardless of origin.

Pfactorial_Case_Study_LiftsToday_Data_Sourcing image 2

Figure 1 - Every record - regardless of which source it came from - passes through the same fetch, extract, validate and normalise stages before it's written.

Architectural principles

Validate before writing, not after - Condition, availability, pricing and measurements are each confirmed against a second signal on the page before a record is accepted, so silent failures don't reach the dataset.
One shared output structure, regardless of source - Every record carries the same 35 fields whether it came from the primary or secondary source, with missing values recorded as empty rather than omitted, keeping the structure constant for import.
No detail-page visits where the listing page already has everything - The primary source's listing page carries the complete record - full specification block and every photograph - so complete records come back in a single request rather than two.
Adding a source shouldn't touch anything downstream - Each source gets its own self-contained collector writing into the same shared structure, so a third source means one new collector, not a rebuild.

CAPABILITIES DELIVERED

Each capability moves the pipeline from a single scraped page to a validated, import-ready dataset.

CAPABILITY	WHAT IT DOES
Multi-source collection without paid APIs	Dedicated collectors against two public marketplaces, both accessible by direct HTTP request with no API key or subscription.
Full-fidelity field capture	Equipment details, technical specifications, pricing, images, category classification and dealer contact details captured for every listing.
Category mapping	Source categories mapped onto the platform's own category identifiers, so listings arrive ready to slot into the existing catalogue.
Field-level validation	Condition, availability, pricing and measurements each checked against a second signal before a record is written.
Consistent 35-field output	Every record, from either source, delivered in the same structure in both JSON and CSV, with specifications and image arrays preserved in JSON and flattened for review in CSV.
Offline automated verification	136 checks across both collectors confirm every field is interpreted correctly, without needing network access.

Pfactorial_Case_Study_LiftsToday_Data_Sourcing image 3

Figure 2 - The same collect-validate-normalise pipeline applies to every source; a third source adds one collector without changing anything downstream.

Design note The primary source's listing page carries the complete record for every machine, so no detail-page visit is needed at all - a design choice that turns a 24-listing category page into 24 complete records in a single request, rather than the 25 requests a detail-page-per-listing approach would cost.

ENGINEERING FOR SCALE AND RELIABILITY

Four decisions kept the pipeline lightweight and dependable rather than fragile.

Two dependencies, both long-established, instead of a heavier scraping framework

requests for HTTP retrieval and BeautifulSoup 4 for parsing kept the collectors simple to install and maintain, with no browser automation or paid service in the path.

Second-signal validation instead of trusting the first parsed value

Condition, availability and identifiers are each confirmed against a second indicator on the page before a record is written, catching the class of failure where collection continues but the data is subtly wrong.

Price-on-request recorded explicitly, not as a zero

Pricing ambiguity is preserved in the output rather than collapsed into a misleading numeric value that would corrupt downstream pricing logic.

An offline test suite covering both collectors

136 checks run without network access, so field-interpretation correctness can be verified in CI without depending on live site availability.

DELIVERY APPROACH

The proof of concept moved from source assessment to a validated, import-ready dataset in four phases.

1. Source assessment - evaluating candidate public marketplaces on inventory volume, data completeness and accessibility, and selecting two to build against.

2. Collector build - building a dedicated collector per source, each writing into one shared output structure.

3. Validation & normalisation - adding second-signal field validation and mapping source-specific values onto the platform's own category identifiers.

4. Verification & sample delivery - covering both collectors with a 136-check offline test suite and delivering live sample data - 24 telehandler and 10 forklift listings - in JSON and CSV.

RESULTS AND IMPACT

Pfactorial_Case_Study_LiftsToday_Data_Sourcing image 4

- Key outcomes from this engagement.

The proof of concept delivered working collection against two public marketplaces, with live sample data - 24 complete telehandler listings and 10 forklift listings - supplied in both JSON and CSV, covering prices from $23,000 to $144,950, 107 full-resolution images and 529 individual specification values across the telehandler sample alone.

Because every field is validated against a second signal before being written and both collectors are covered by a 136-check offline test suite, the dataset is delivered consistent and ready for import rather than requiring manual cleanup before it can be trusted.

What it enabled commercially

The platform now has a validated, no-licensing-cost path to the inventory data it needs, with a full category of 1,831 telehandler listings collectible in under three minutes, and a collection pattern that extends to further sources by adding one self-contained collector rather than re-architecting the pipeline.

WHY PFACTORIAL

This engagement sits within Pfactorial's applied data-engineering practice: proving out validated, cost-free alternatives to paid data APIs on live inventory, with the same rigor - field-level validation, offline test coverage, consistent output structure - that production data pipelines require.

Pfactorial_Case_Study_LiftsToday_Data_Sourcing image 5

- Service lines this engagement draws on.

Engagement enquiries Pfactorial Technologies works with marketplace and e-commerce platforms that need reliable data sourced without an expensive or unavailable paid API. If you're evaluating an alternative data-collection approach, we're happy to give you an honest read on scope and risk before anyone commits to anything. · pfactorial.ai

APPENDIX A - TECHNOLOGY STACK

The technology stack underpinning the system, grouped by the layer it serves.