How this data is collected, how often, and how far back it goes
The three questions you asked, answered directly and without hedging.
1 · How the data is collected
Collection
Every field on this site was read off a live Wayfair page
- Method
- HTTP collection of Wayfair's server-rendered category, search and product pages, parsed from the live DOM.
- Incognito by default
- Every run uses a fresh, cookie-clean, logged-out session — the programmatic equivalent of an incognito window. No account, no cart, no browsing history, so no personalisation bias.
- Geography
- Pinned to a fixed postcode (this sample: 10001, New York NY). Delivery promises and some pricing are location-dependent, so the postcode is held constant across days. Any number of postcodes can run in parallel.
- Sort state
- Default Recommended sort — what roughly 95% of Wayfair shoppers actually see. Any other sort or filter state can be tracked alongside.
- Depth
- This sample covers the first 3 result pages (~144 slots) per category and per keyword. Depth is configurable.
Measured vs. derived
The dataset keeps these strictly separate — no derived value is ever mixed into a measured column
Organic rank, on-page slot position, sponsored flag, title, brand, SKU, price, list price, discount, merchandising badge, star rating, cumulative review count, shipping promise, variant colours and sizes, image and product URL.
Review-count deltas, estimated units, rank bands, share of shelf, brand classification. Always computed from measured fields, always documented in the data dictionary.
One important distinction: we record both slot_position (what a shopper sees, ads included) and organic_rank (position among unpaid results). About 37.4% of slots in these categories are paid, so the two differ substantially. Organic rank is what measures merchandising strength.
2 · Refresh frequency
PLOTT DATA builds the collection to the requirement rather than fitting you to a fixed product schedule. If it is technically possible, we can run it.
| Cadence | Best for | Notes |
|---|---|---|
| Dailyrecommended | Rank tracking, price and promo monitoring, share of shelf | One snapshot per category per day at a fixed hour, so day-over-day moves are clean |
| Intraday (2–24×/day) | Promo and Way Day monitoring, ad-slot churn, price-war response | Rank and price genuinely move within a day; hourly resolves it |
| Weekly | Velocity, assortment drift, competitive entry and exit | Aggregated from the daily series — you keep the daily grain underneath |
| Real-time / on-demand | Launch days, incident checks, ad-hoc category pulls | Triggered via API, results in minutes |
This sample ships daily grain with a weekly roll-up, because that is the combination that actually supports rank↔sales inference: daily catches the movement, weekly is the window over which review counts accumulate enough to be a usable demand signal.
3 · How far back history goes
Wayfair does not publish rank history, and it cannot be reconstructed retroactively by anyone — including us. A rank time series begins the day collection begins. We would rather say that plainly than sell you a backfill that does not exist.
Full cross-section, immediately usable
Every ranked listing with position, price, rating and cumulative review count. Share of shelf, ad density and the review entry bar per category are all computable from a single snapshot — every observed finding on this site came from one day.
First true movement
Rank changes, price-change events, new entrants and listings that drop out become visible.
Velocity turns reliable
Review deltas accumulate enough signal to estimate units, and rank↔velocity relationships hold up statistically.
Seasonality and promo cycles
Way Day and holiday effects, sustained competitor trajectories, assortment strategy shifts.
One genuine head start
Cumulative review_count is a stock that Wayfair displays today and that encodes years of past demand. So from the very first snapshot you already get a lifetime-to-date demand proxy for every SKU on the shelf — you simply cannot slice when those reviews arrived until measurement starts. Every day of collection converts more of that stock into measurable flow. The rank-band analysis on this site is built entirely from that day-one stock, and it is already actionable.
Coverage notes
Stated up front, because you will find them in the data.
Client-side rendered categories
Cat Trees, Perches & Scratchers (c1874142) streams its result grid after hydration, so plain HTML collection returns ~17 of ~144 slots.
The keyword surface covers the same demand fully — 7 pet queries are included at full depth, including the four cat-tree head terms. In production this category is collected via the rendered-browser path with residential egress.
Sponsored slots have no organic rank
By design — a paid placement has no merchandising position.
Filter is_sponsored = False for merchandising analysis; use slot_position for true share of voice including advertising.
Review counts are cumulative
And can occasionally decrease when Wayfair removes reviews.
Treat negative new_reviews as zero when estimating units. The effect is small but real, and we surface it rather than smoothing it away.
Compliance
Collection covers publicly accessible pages only — the same pages any shopper can load without signing in. No account access, no authentication is circumvented, no personal data is collected, and no Wayfair partner-portal data is touched. Requests are rate-limited and spread to avoid placing meaningful load on the origin. Delivery formats: CSV, JSON/JSONL, Parquet, direct-to-warehouse (Snowflake, BigQuery, Redshift), S3/GCS drop, or REST API.