videorecommendationscraping

Crawling Vertical-First Video Platforms: Metadata, Thumbnails and Content Discovery for AI Microdramas

UUnknown

2026-03-07

11 min read

Practical playbook for scraping mobile-first vertical video: extract thumbnails, metadata and recommendation signals for microdrama models.

Hook: Why scraping mobile-first vertical video platforms is suddenly urgent (and hard)

If you build recommendation models for microdramas or need scalable signals from mobile-first vertical video platforms, you've likely hit three blockers: platforms are API‑obscured, anti‑bot systems are aggressive in 2026, and the useful signals (thumbnails, watch percent, recommendation reasons) are often ephemeral. This playbook gives a compact, technical, and legal-first approach to extract clean metadata, high‑quality thumbnails and recommended‑feed signals at scale — using Scrapy, Playwright, Puppeteer, Selenium and lightweight HTTP clients, plus webhook ingestion patterns for downstream training pipelines.

The 2026 context: what's changed since 2024–25

Two quick trends shape strategies in 2026:

Vertical video platforms (Holywater and peers) scaled AI-driven microdramas and mobile-first feeds. Funding rounds in late 2025/early 2026 (e.g., Holywater’s growth capital) accelerated personalization and server-side ranking logic.
Anti-abuse tech evolved: platforms now combine device attestation (PlayIntegrity/DeviceCheck), behavioral fingerprinting, ephemeral signed URLs, and ML detectors that flag non-human scroll patterns and synthetic browsing.

High-level approach (inverted pyramid): what you should do first

Reverse-engineer the surface — find the usable endpoints (app API / web GraphQL) and the tokens required.
Decide capture mode — API-first (preferred), headful browser emulation, or hybrid (headless to obtain tokens + HTTP client to call APIs).
Collect signals in a structured schema: video metadata, thumbnails, feed exposure (position, reason), watch metrics (view, completion, rewatch), and social/engagement counts.
Pipeline & webhook the scraped output into storage, feature stores, and training pipelines with idempotency and signing.

Step 1 — Finding the real API: techniques that work in 2026

Most mobile‑first platforms expose richer APIs to their apps than to the web. Common payloads are JSON, GraphQL or protobuf over HTTP/2. Here are practical techniques to find those endpoints safely:

Use a dynamic proxy (mitmproxy, Burp) on a rooted/emulator device to capture app traffic. Many apps still use TLS pinning — bypass with Frida or patch the binary (for research on your own systems or with permission).
Instrument a headful browser (Playwright/Chromium) to capture XHRs while emulating a mobile UA. Playwright’s network hooks let you log GraphQL operations and response shapes.
Search for mobile SDKs inside APKs. Messaging and ad SDKs often leak endpoints and header names; unzip and grep the binary for endpoints, header keys, and signature routines.

Common artifacts to look for

Headers: x-client-version, x-device-id, x-platform, x-os, x-app-build, Authorization
Signed tokens / nonces: x-signature, x-timestamp, signature query param
GraphQL operation names: feedQuery, watchEvent, recommendationReasons
Protobuf/NDJSON or paginated JSON shape with feed items

Step 2 — Token strategies: obtain tokens without headless slowness

Goals: be efficient (avoid rendering full UI for every request), but produce tokens the platform accepts. Use a hybrid model: a small fleet of headful Playwright sessions produce short‑lived tokens (or cookies), which are reused by HTTP clients for bulk calls.

Headful to headless handoff: run Playwright one-per-proxy to log Authorization tokens and client headers. Rotate sessions every X minutes to mimic natural clients.
Emulate mobile device: use Playwright device descriptors (iPhone/Pixel) and enable touch events and client hints (DPR, viewport). Many servers expect those headers.
Replayable signature generation: where signatures are HMAC of payload+timestamp, extract secret lookup or key derivation routines from the app binary and replicate them server-side (only for compliant internal integrations). If signature requires device state or hardware keys (attestation), use a fleet of real devices or Android emulators with PlayIntegrity attestation bypass strategies — but consider legal risk.

Step 3 — Data model: what to collect for recommendation models

The goal is to capture both content metadata and exposure signals that feed ranking models. Keep the schema strict and immutable once training starts.

Example JSON schema (simplified)
{
  "video_id": "str",
  "title": "str",
  "description": "str",
  "uploader_id": "str",
  "duration_ms": 120000,
  "published_at": "iso8601",
  "tags": ["microdrama","romcom"],
  "thumbnail_urls": {"low":"url","hd":"url"},
  "captions_url": "url.vtt",
  "engagement": {"views":12345,"likes":456,"comments":12},
  "watch_metrics": {"avg_watch_pct":0.68,"complete_rate":0.42,"rewatch_rate":0.05},
  "feed_exposures": [
    {"session_id":"s1","position":3,"served_reason":"similar_artist","timestamp":"iso"}
  ]
}

Key signals to prioritize

Feed exposure: feed_id, position, served_reason, timestamp — crucial for causal modeling.
Watch metrics: watch percent, time-to-first-drop, rewatch — prefer platform-provided metrics where available.
Thumbnail details: CDN paths, variants (poster, first frame, generated thumbnails), hash and dominant colors.
Recommendation graph edges: “also viewed”, “because you watched” lists emitted by the platform.

Step 4 — Crawling stacks: patterns & code snippets

Use the right tool for the task: Scrapy for high‑throughput JSON crawling, Playwright/Puppeteer for dynamic behavior and token harvest, Selenium only where legacy browsers or real extension interactions are required.

Scrapy: high-throughput API collector (bulk feed items)

# scrapy spider (simplified)
import scrapy
import json

class FeedSpider(scrapy.Spider):
    name = 'feed'
    start_urls = ['https://api.example.com/v1/feed']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, headers={
                'Authorization': 'Bearer ' + self.settings.get('TOKEN'),
                'User-Agent': 'okhttp/4.9.3'
            })

    def parse(self, response):
        data = json.loads(response.text)
        for item in data['items']:
            yield {
                'video_id': item['id'],
                'title': item.get('title'),
                'thumbnail': item['thumbs']['hd']
            }
        # follow pagination
        if data.get('next'):
            yield scrapy.Request(data['next'], callback=self.parse)

Playwright: emulate mobile feed and capture exposure events

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    iphone = p.devices['iPhone 12']
    browser = p.chromium.launch(headless=False)
    context = browser.new_context(**iphone)
    page = context.new_page()
    page.goto('https://m.example.com')

    # intercept feed GraphQL requests
    page.on('request', lambda req: print('REQ', req.url) if 'feed' in req.url else None)

    # simulate natural swipe scrolling
    for _ in range(30):
        page.keyboard.press('PageDown')
        page.wait_for_timeout(400 + random.randint(0,300))

    browser.close()

Puppeteer: capture thumbnails and render first frame

const puppeteer = require('puppeteer-core');
const devices = require('puppeteer/DeviceDescriptors');
(async () => {
  const browser = await puppeteer.launch({args:['--no-sandbox']});
  const page = await browser.newPage();
  await page.emulate(devices['iPhone X']);
  await page.goto('https://m.example.com/watch/VIDEO_ID');

  // Wait for player and capture poster
  await page.waitForSelector('video');
  const poster = await page.$eval('video', v => v.getAttribute('poster'));
  console.log('poster', poster);

  // capture thumbnail screenshot
  const thumbnailBuffer = await page.screenshot({clip:{x:0,y:0,width:720,height:1280}});
  // upload buffer to S3 or store
  await browser.close();
})();

HTTPX/Requests for efficient bulk calls with tokens

import httpx

client = httpx.Client(http2=True, headers={'User-Agent':'okhttp/4.9.3'})
client.headers.update({'Authorization':'Bearer ' + token})
resp = client.get('https://api.example.com/v1/videos/bulk?ids=1,2,3')
print(resp.json())

Step 5 — Dealing with anti-bot tech (practical mitigations)

Anti-abuse in 2026 is multi‑layered. You don't need to defeat it — you need to behave like a real client at scale.

Rotate IPs + proxies: residential or mobile proxies per session. Keep session affinity for tokens.
Behavioral realism: variable scroll timing, touch events, randomized read times. Emulate drop-off and short sessions.
Device fingerprints: use device descriptors (DPR, UA, screen, timezone) that match your proxy geography. Playwright supports precise descriptors.
Exponential backoff: when you see 429 or mitigations page, back off and rotate token/proxy. Log failures with full stack for post-mortem.
Respect attestation: if a platform uses attestation tokens you cannot legitimately obtain, consider partnering or using public datasets — do not bypass hardware-based checks unlawfully.

Step 6 — Thumbnail strategy: get the best, normalize and store

Thumbnails are the anchor for microdrama previews and cataloging. Platforms often serve multiple poster sizes; prefer CDN canonical image endpoints or generate your own first-frame thumbnails when necessary.

Prefer direct thumbnail URL returned in API; if only poster=tokenized URL exists, use your headful session to fetch a durable CDN path.
If no thumbnail is available, render first frame from the video stream using ffmpeg and store multiple sizes (64x, 320x, 720x) and compute pHash and dominant color.
Store thumbnails with stable naming (video_id + variant + sha256) in object storage and include CDN caching headers.

# ffmpeg extract first frame
ffmpeg -i video.mp4 -vf "select=gte(n\,1)" -vframes 1 -q:v 2 thumb.jpg

Step 7 — Capturing recommendation signals and feed reasons

Modern feeds attach a served_reason (e.g., "because_you_watched", "trending_in_region", "similar_creator"). Capturing those is high-value for candidate generation and counterfactual evaluation.

When intercepting feed API responses, store operationName or reason field for each item.
Capture feed position and session context — do not drop these fields. They enable position bias correction in models.
Collect both direct graph edges (related_videos arrays) and implicit edges derived from sequential exposures in the same session.

Step 8 — Ingestion: webhooks, signing, and idempotency

Use a webhook-based ingestion to stream scraped items into your feature store or queue. Sign payloads for integrity and include dedupe keys.

# example webhook sender (Python)
import hmac, hashlib, json, requests

WEBHOOK_URL = 'https://ingest.example.com/receive'
SECRET = b'supersecret'

payload = json.dumps(scraped_item).encode()
sig = hmac.new(SECRET, payload, hashlib.sha256).hexdigest()
resp = requests.post(WEBHOOK_URL, data=payload, headers={'X-Signature':sig, 'Content-Type':'application/json'})
print(resp.status_code)

On the receiver side, verify signatures, de‑duplicate by video_id + source, and store raw and normalized records separately for reproducibility.

Step 9 — Feature engineering: from raw signals to model-ready features

Convert scraped watch_metrics and feed exposures into derived features:

Normalized view velocity (views per hour since publish)
Position bias adjusted engagement (CTR adjusted by expected position)
Thumbnail attractiveness score (pHash similarity to top thumbnails, color contrast)
Temporal features: time-of-day/week performance for microdramas

Compliance and ethics: short checklist (must-read)

Platforms tightened policies in late 2025 — always verify Terms of Service and consult legal before programmatic scraping of authenticated endpoints or internal APIs.

Follow robots.txt for public web endpoints.
Do not scrape private user data or PII without consent.
If you reverse-engineer an app for tokens, ensure you have authorization or operate inside allowed research boundaries.
Prefer partnerships or data licensing for production-grade, high-volume needs.

Operationalizing at scale: orchestration and monitoring

Build these primitives into an orchestration layer: token manager, proxy pool, headful session allocator, and a deduping ingestion endpoint. Monitor these signals:

Token failures per minute (spike indicates signature drift)
429/403 rate per proxy
Freshness lag: time from publish to ingestion
Data quality: missing thumbnails, abnormal watch metrics

Case study (short): harvesting recommendation reasons for microdrama discovery

We instrumented Playwright to emulate 200 mobile sessions across regions, harvesting feed responses and storing served_reason fields. Using position and session context, we derived a graph of co‑exposure edges. Integrating those edges into candidate generation increased recall of “next episode” suggestions by 18% in offline evaluation while reducing false positives via position bias correction.

Advanced tips & future predictions (2026+)

Expect more platforms to adopt ephemeral, per-session signed thumbnails — fallback to headful capture and caching is critical.
On-device attestation will push teams to build real device farms rather than pure emulators for robust token generation.
Graph learning models will benefit from richer served_reason labels; prioritize capturing any textual reason or opCode in GraphQL responses.
Privacy-preserving signals (DP-sanitized aggregates) will become available via platform partnerships — pursue strategic data agreements for enterprise-grade pipelines.

Quick checklist before you run a crawl

Confirm ToS/legal clearance for the endpoints you target.
Start with a small pilot: token harvesting + 100 sessions.
Validate thumbnail quality and caption extraction for sample videos.
Establish webhook ingestion and test idempotency.
Monitor and cap cost: headful sessions are expensive; use hybrid mode.

Actionable takeaways

Hybrid is the pragmatic default: headful sessions for token generation + HTTP clients for bulk pulls.
Capture served_reason and position: these unlock much higher-value candidate features for microdrama recommender systems.
Thumbnail pipeline matters: prefer platform CDN images, but be ready to render and compute pHash and colors for model features.
Instrument resilience: rotate proxies, emulate device fingerprints, and monitor token validity constantly.

Resources & tools

Playwright & Puppeteer — for mobile emulation and dynamic capture
Scrapy — for high-throughput API collection
mitmproxy & Frida — for reverse engineering mobile apps
ffmpeg & imagehash — for thumbnail extraction and features
Object storage + CDN — for durable thumbnail hosting and serving

Final thoughts & call to action

The vertical video era (exemplified by companies like Holywater in early 2026) makes rich content and recommendation signals both more valuable and harder to collect. Use a hybrid crawler architecture, prioritize served_reason and position signals, and build robust thumbnail pipelines. Above all, balance engineering with legal and ethical guardrails: where possible, opt for partnerships and licensed data for production systems.

Ready to operationalize this playbook? Share your platform target and constraints and we’ll outline a customised crawl blueprint, including token workflows, proxy sizing, and a webhook schema for your feature store.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.