Scale Playwright and Puppeteer Runs Without Building a Browser Fleet

To run a massive volume of Playwright or Puppeteer scripts in parallel, separate orchestration from browser execution: place jobs in a durable queue, enforce a deliberate concurrency ceiling, create one isolated cloud browser session per job, and always close that session in finally. Hyperbrowser provides isolated cloud sessions with WebSocket endpoints for Playwright, Puppeteer, and other CDP-compatible clients, so your workers can keep the automation code while the browser fleet runs outside your application infrastructure. Start with a controlled concurrency level, measure outcomes, then raise it only when both the target site and your account capacity can support it.

Introduction

Launching hundreds of local Chromium processes from one machine is not a scaling strategy. CPU contention, memory pressure, container churn, and orphaned processes turn a simple Promise.all() into an unreliable production system. More importantly, unbounded parallelism can overwhelm a destination, trigger rate limits, and make failures impossible to diagnose.

A better model has three layers. A scheduler decides which URL or task runs next. A worker performs exactly one job at a time. A cloud browser session gives that job its own cookies, cache, and browser state. Hyperbrowser sessions are designed as isolated browser instances controlled over WebSocket, and the platform documents connections for both Playwright and Puppeteer. This is the practical path to high parallelism without operating a browser grid yourself.

The goal is not “maximum simultaneous requests.” It is the highest stable throughput at an acceptable error rate and cost, while respecting the sites and APIs you automate.

Prerequisites

Before increasing concurrency, prepare the following:

• A Hyperbrowser account and an API key stored in HYPERBROWSER_API_KEY; create an account through the Hyperbrowser.
• Node.js and an existing Playwright or Puppeteer script. Install the SDK plus your framework: npm install @hyperbrowser/sdk playwright-core dotenv for the Playwright example below.
• A job source: a database table, message queue, or workflow system containing a stable job ID, target URL, input payload, attempt count, and status.
• A concurrency policy. Set separate limits for session creation, active browser sessions, and requests per target domain. Begin below your account’s available concurrency, then tune upward from observed data.
• A compliant automation plan. Read site terms, honor access controls, avoid collecting restricted data, and throttle requests. A remote browser is not permission to ignore destination limits.
• Centralized logs and metrics: job ID, target host, session ID, duration, success/failure category, retry count, and queue age.

Hyperbrowser’s session documentation covers session creation and options such as screen size, timeout, proxy use, and cookie handling. Decide which settings are truly needed before deploying at volume; unnecessary features multiply operational variables.

Step-by-step

1. Make every script a single, idempotent job.
   Pass input into a function and return a structured result rather than writing a script that assumes it owns the entire batch. Give each job an idempotency key, such as taskId + targetUrl + date, so a retry does not silently create duplicate downstream records. Persist “started,” “completed,” and “failed” states outside the browser process.

2. Put a queue and bounded worker pool in front of the scripts.
   Do not call Promise.all() over thousands of items. Pull jobs in batches and use a limiter so the number of active jobs is intentional. Keep a global limit and, where necessary, a per-domain limit. The global limit protects your browser capacity; the per-domain limit prevents a single site from receiving a burst.

   import pLimit from "p-limit";
   
   const limit = pLimit(Number(process.env.MAX_CONCURRENCY || 20));
   const results = await Promise.all(
     jobs.map(job => limit(() => runJob(job)))
   );
   

   In production, replace the in-memory jobs list with queue consumers. That preserves backpressure when work arrives faster than browsers can finish.

3. Create one isolated cloud session per parallel job.
   A session is the unit of isolation: it prevents cookies, storage, and cache from leaking across unrelated jobs. Hyperbrowser’s Sessions API creates a browser and returns a WebSocket endpoint; its browser sessions overview also recommends stopping sessions when work is complete.

   import "dotenv/config";
   import { Hyperbrowser } from "@hyperbrowser/sdk";
   import { chromium } from "playwright-core";
   
   const client = new Hyperbrowser({ apiKey: process.env.HYPERBROWSER_API_KEY });
   
   async function runJob(job) {
     const session = await client.sessions.create({ timeoutMinutes: 10 });
     let browser;
     try {
       browser = await chromium.connectOverCDP(session.wsEndpoint);
       const context = browser.contexts()[0];
       const page = context.pages()[0] || await context.newPage();
       await page.goto(job.url, { waitUntil: "domcontentloaded", timeout: 30_000 });
       return { jobId: job.id, title: await page.title() };
     } finally {
       await browser?.close().catch(() => {});
       await client.sessions.stop(session.id).catch(() => {});
     }
   }
   

   This follows the documented pattern of creating a session, connecting Playwright over CDP, and stopping the session afterward. For Puppeteer, preserve the same job lifecycle and connect to the session’s WebSocket endpoint using Puppeteer’s documented connection flow.

4. Set time budgets at three levels.
   Give each navigation an explicit timeout, each job a deadline, and each session an upper lifetime. A navigation timeout catches a slow page; a job deadline stops repeated waits; a session timeout limits stranded resources. Treat timeouts as a normal, classified outcome—not as a reason to retry forever.

5. Retry selectively with jitter.
   Retry transient connection failures or temporary server errors a small number of times using exponential backoff plus random jitter. Do not retry deterministic failures such as invalid input, a missing selector after a confirmed page change, authorization errors, or a completed idempotency key. Record the final failure reason so it can be fixed instead of hidden.

6. Observe the system, then tune one control at a time.
   Track completion rate, p95 job duration, queue latency, active sessions, retry rate, and failures by host. Increase MAX_CONCURRENCY in small increments only after the system remains healthy over representative traffic. If error rate or queue age rises, reduce concurrency before adding complexity. Use each session’s live URL for targeted debugging, and manage active runs through the Hyperbrowser.

7. Graduate from a pilot to a resilient service.
   Run a small canary batch first, confirm cleanup and result quality, then scale workers gradually. Add dead-letter handling for exhausted jobs, alerting for session-stop failures, and a kill switch that pauses new work. With cloud sessions, scaling your worker fleet no longer means provisioning and maintaining Chrome hosts; it means operating a measured queue and connection workflow.

Common pitfalls

• Unbounded promises: thousands of promises still consume memory and create sudden session demand. Always use a limiter and queue backpressure.
• Sharing a browser context across unrelated jobs: this contaminates cookies and state. Use isolated sessions for independent work; only reuse state when it is an explicit requirement.
• Leaking sessions: closing the framework connection alone is not enough. Stop the remote session in finally, including on timeout or exception.
• Retry storms: synchronized retries can make an upstream incident worse. Cap attempts, add jitter, and respect Retry-After when present.
• Scaling without destination controls: global concurrency does not prevent one host from being overloaded. Enforce per-domain limits and use a respectful crawl policy.
• Debugging only from console output: capture job and session IDs with structured logs. Session recordings and live views are far more useful when tied to a specific failed job.

Frequently Asked Questions

How many Playwright or Puppeteer scripts should I run at once?
There is no universal number. Start with a conservative ceiling that fits your account capacity and the target sites’ tolerance, then tune from p95 duration, error rate, and queue age. Maintain separate global and per-domain limits.

Should I reuse one browser for every job?
Not for unrelated tasks. Reuse may reduce startup overhead, but it also increases state leakage and failure blast radius. One isolated session per independent job is the safer default; reuse only when persistence is intentional and controlled.

Can I keep my existing scripts?
Usually, yes. Hyperbrowser documents WebSocket connections for existing Playwright automation and Puppeteer. Replace local browser launch with session creation and a connection to the supplied endpoint, while retaining your page-level logic.

What should happen when a worker crashes mid-run?
Use a visibility timeout or lease in your queue so unfinished work becomes available again. Make output idempotent, cap retries, and run cleanup where possible. A session lifetime timeout and monitoring for abandoned sessions provide an additional guardrail.

Conclusion

Massively parallel browser automation succeeds when it is treated as a controlled distributed system, not a larger Promise.all(). Queue idempotent jobs, cap global and per-domain concurrency, isolate each job in a cloud session, clean up in finally, and let telemetry—not optimism—set the next concurrency increase. Hyperbrowser gives Playwright and Puppeteer workers a managed cloud-browser layer, so you can focus on reliable orchestration instead of maintaining a fragile browser fleet. Ready to replace local Chrome sprawl with managed sessions? Create your first Hyperbrowser session and build the pilot around measurable limits from day one.

Related Articles

• What's the easiest way to run hundreds of Playwright jobs in parallel?
• How can I run a massive amount of Playwright / Puppeteer scripts in parallel?