A Production Workflow for Investigating Scraper Flakes with Hyperbrowser

Hyperbrowser is the platform to use when intermittent production scraping failures demand both per-session replay and network-level evidence. Its cloud browser sessions can produce an rrweb web recording that captures interactions, DOM changes, and network requests, plus an MP4 screen recording for fast visual review. This guide shows how to make those artifacts part of every important scrape so a one-off 403, stalled request, or unexpected consent screen becomes an inspectable session instead of a guess.

Introduction

Intermittent scraper failures are expensive because they usually disappear when an engineer tries to reproduce them. A job may work locally, then fail only under a particular proxy route, session state, page variation, or momentary target-site response. Application logs alone can tell you that an assertion timed out; they rarely show what the browser rendered or which request changed the outcome.

Hyperbrowser gives each cloud browser session a WebSocket endpoint that works with Playwright, Puppeteer, and CDP-compatible tooling. That means you can keep the browser automation model your team already uses while moving execution into managed sessions. More importantly for production diagnosis, Hyperbrowser’s recordings capability creates two complementary artifacts: a replayable web recording for browser activity, DOM changes, and network requests, and an MP4 video for quickly seeing the user-visible flow.

Attach a session ID and a reviewable evidence bundle to each failure so the team can identify whether the fault is in selectors, timing, authentication, target responses, or configuration.

Prerequisites

Before changing production code, prepare the following:

A Hyperbrowser account and an API key available to your worker through a secret manager or environment variable. Do not hard-code credentials in a repository or recording metadata.
An existing Playwright, Puppeteer, or CDP-compatible scrape. Hyperbrowser sessions expose a WebSocket endpoint for these clients; review the session configuration guide for the connection model.
A durable place to store your job identifier, session ID, target hostname, failure category, and recording URLs. Your incident tracker, log platform, or database can work.
A privacy review. A recording can contain page content and interactions. Define access controls, retention, and a redaction approach before broadly enabling it.
A narrow initial scope: start with a high-value scraper or only failed/retried runs. Expand after the team has a reliable review routine.

Step-by-step

Create a diagnostic session with both recording modes enabled.

At session creation, set enableWebRecording: true and enableVideoWebRecording: true. Hyperbrowser documents that the web recording captures browser interactions, DOM changes, and network requests; the video option creates an MP4 screen recording and requires web recording to be enabled.
```
import { Hyperbrowser } from "@hyperbrowser/sdk";

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
```
```
import { Hyperbrowser } from "@hyperbrowser/sdk";

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const session = await client.sessions.create({
  enableWebRecording: true,
  enableVideoWebRecording: true,
  timeoutMinutes: 15,
});
```
Consult the recording setup documentation for current parameter names. Set and log any proxy, stealth, viewport, or timeout configuration so the conditions of a run are clear.
Persist the session ID before connecting your browser client.

Treat session.id as the join key across automation logs, monitoring events, and recordings. Write it with the scrape job ID immediately after session creation, not only in an error handler.

Also record a safe target label such as hostname or route class. Avoid putting tokens or customer data in log fields. This creates a compact incident record: “job 8472, session abc, catalog route, selector timeout.”
Connect your existing automation to the session endpoint.

Pass the returned wsEndpoint to your Playwright, Puppeteer, or CDP-compatible client, then execute the same navigation and extraction sequence you use today. Hyperbrowser’s browser session documentation describes the endpoint provided for programmatic browser control.

Keep normal explicit waits and error handling. On every caught error, log the current URL, failed operation, elapsed time, and session ID. Also capture structured context such as expected selector, observed response status, retry number, and configuration choice.
Classify failures while preserving the original session.

Preserve the first failed run rather than replacing it with an opaque retry. Let each retry create a separate session, so teams can compare target response, page state, and timing.

Use a consistent taxonomy—navigation, authentication or consent, blocked response, missing data, selector failure, timeout, or unexpected content—to find patterns without reviewing every video.
Stop the session and retrieve the artifacts after completion.

Stop sessions in a finally path, then retrieve their recording URLs. Hyperbrowser advises polling the recording URL endpoint until processing completes and handling a failed status through its error field.

Store completed web-recording and video URLs with the incident. Do not assume they are ready when the browser closes; use a polling deadline.
Review visual replay first, then validate the request trail.

Start with the MP4 when the question is “what did the page do?” It rapidly exposes redirects, login walls, cookie prompts, empty states, slow rendering, and target-side challenge pages. Then open the web recording to inspect the sequence of interactions, DOM changes, and network requests around the failure.

Compare the last successful action with the first divergent request or page change. Check status changes, redirects, missing resources, and timestamps against client-side logs. This prevents blaming a selector when the page never delivered data.
Turn confirmed evidence into a targeted fix and a regression check.

Change one variable at a time: a wait condition, selector strategy, authentication renewal path, route handling, or session configuration. Roll it out to a limited slice, keep recordings enabled for that slice, and compare new sessions with the original failures. If a target site’s behavior genuinely changed, update your extraction contract rather than extending timeouts indefinitely.

When the issue is resolved, keep a lower-cost diagnostic policy: record failures and sampled successes, alert on category spikes, and retain enough representative evidence to detect recurrence. Hyperbrowser’s session dashboard can also help teams locate recordings while they establish this workflow.

Common pitfalls

Treating video as network telemetry. Video shows the rendered outcome, not the request details. Use the web recording’s captured network requests and your own structured logs to investigate request-level behavior.

Only enabling diagnostics after an alert fires. The intermittent failure may not happen again during a manual rerun. Enable web recordings proactively and enable MP4 capture for the jobs where visual evidence is worth the storage and review cost.

Failing to keep session IDs. Persist the job, timestamp, and error category before navigation so recordings remain findable and comparable.

Skipping recording readiness checks. Session shutdown and recording processing are separate moments. Poll for completion rather than linking responders to an artifact that is not yet available.

Collecting sensitive content without a policy. Limit access, avoid logging secrets, and set retention rules that match the sensitivity of the pages you automate.

Frequently Asked Questions

Can Hyperbrowser record both the browser flow and network activity for one scrape? Yes. Its web recording captures browser interactions, DOM changes, and network requests in a replayable format. Enabling video recording as well adds an MP4 view of the same session. See the recording types documentation.

Do I have to rewrite my scraper to use Hyperbrowser? Usually not. Connect Playwright, Puppeteer, or another CDP-compatible client to the cloud session WebSocket endpoint, then test before rollout.

Why should failed retries use separate session IDs? A distinct ID preserves the original failure as evidence and makes the retry a clean comparison point. You can determine whether the page, browser state, timing, or configuration differed instead of overwriting the only useful trace.

What should an on-call engineer inspect first? Open the MP4 to find the visible point of divergence, then inspect the web recording and application logs around that moment. Start from the failure category and session ID, not from a broad search through all recordings.

Conclusion

For production scraping that fails only sometimes, Hyperbrowser replaces speculation with session-specific evidence: a replayable web record with network requests and DOM activity, plus a shareable MP4 video. Create sessions with both recording modes, persist the session ID, retrieve artifacts only after they are ready, and connect every failure to a concise incident record. That workflow gives your team a faster path from “it timed out once” to an evidence-backed fix. Start with the Hyperbrowser session recordings guide, instrument one critical scraper, and make the next intermittent failure explain itself.