Inside a Job

When your spider runs, the platform hands it everything it needs to reach the rest of the platform — through environment variables. Your spider reads them with os.environ; there’s nothing to install or configure.

What’s in the environment

Using the proxy

Route requests through the proxy gateway with the injected credentials. In Scrapy, set the proxy per request and add the auth header — a minimal downloader middleware:

import base64, os
 
class ProxyMiddleware:
    HOST = os.environ.get("INSIGHT_PROXY_HOST", "insightproxy.insightscrap.com")
    PORTS = {"datacenter": 60000, "residential": 61000, "dynamic": 62000}
 
    def __init__(self):
        self.user = os.environ["INSIGHT_PROXY_USERNAME"]
        self.password = os.environ["INSIGHT_PROXY_PASSWORD"]
 
    def process_request(self, request, spider):
        tier = request.meta.get("proxy_type", "datacenter")
        country = request.meta.get("proxy_country")   # e.g. "mx"
        session = request.meta.get("proxy_session")   # sticky token, optional
 
        user = self.user
        if session:
            user += f"-session-{session}"
        if country:
            user += f"-{country}"
 
        request.meta["proxy"] = f"http://{self.HOST}:{self.PORTS[tier]}"
        token = base64.b64encode(f"{user}:{self.password}".encode()).decode()
        request.headers["Proxy-Authorization"] = f"Basic {token}"

The full proxy contract — tiers, geo-targeting, sticky sessions, error codes — is on the Proxies page.

Using a managed browser

Never launch a browser inside a job — connect to a managed browser over the injected WebSocket URL (the access token is already embedded):

from playwright.sync_api import sync_playwright
import os
 
ws = os.environ["BROWSER_WS_URL_PLAYWRIGHT"]
 
with sync_playwright() as p:
    browser = p.chromium.connect(ws)
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()

Each engine has its own variable and connect call — see Managed Browsers. Managed browsers already egress through a proxy, so you don’t set one on the browser.

Using personas

When your spider needs working identities and their inboxes, call the crawler API with the injected ISBACKEND_INTERNAL_URL and BROWSER_TOKEN:

import os, urllib.request, json
 
base = os.environ["ISBACKEND_INTERNAL_URL"].rstrip("/")
token = os.environ["BROWSER_TOKEN"]
req = urllib.request.Request(
    f"{base}/api/v1/crawler/personas",
    headers={"Authorization": f"Bearer {token}"},
)
personas = json.load(urllib.request.urlopen(req, timeout=30))["personas"]

The full persona + inbox API is on the Personas page.

Reading your schedules & run metrics

Every job can read back your org’s schedules and run history with metrics — the same data the dashboard shows — using the injected INSIGHT_METRICS_API_URL and INSIGHT_METRICS_API_TOKEN. This is how you build a metrics-export spider that ships run stats to your own warehouse or dashboard:

import os, urllib.request, urllib.parse, json
 
base = os.environ["INSIGHT_METRICS_API_URL"].rstrip("/")
token = os.environ["INSIGHT_METRICS_API_TOKEN"]
 
def get(path, **params):
    url = base + path + (("?" + urllib.parse.urlencode(params)) if params else "")
    req = urllib.request.Request(url, headers={"Authorization": f"Bearer {token}"})
    return json.load(urllib.request.urlopen(req, timeout=30))
 
# The next 30 scheduled spiders (soonest first):
schedules = get("/api/v1/client/schedules", sort="next_run_at", dir="asc", limit=30)["schedules"]
 
# The last 20 runs with their metrics:
runs = get("/api/v1/client/jobs", limit=20)["jobs"]
for r in runs:
    print(r["spider"], r["status"], r["items_scraped"], r["error_count"])

The token is scoped to your organization, so you only ever see your own schedules and runs. The full endpoint reference is on the Schedules & Runs API page.

Using the database add-on

With the postgres add-on enabled, connect with the injected DSN — it’s already scoped to your organization:

import os, psycopg2
 
conn = psycopg2.connect(os.environ["POSTGRES_URL"])

Getting your data out

Your scraped items are yours to route. The platform runs the spider and captures its logs and stats, but it does not store your items for you — you send them to a destination you control, configured in your project’s settings.py (deployed with your code):

• A Scrapy feed (FEEDS) to your own S3/GCS bucket, or
• A custom item pipeline writing to your database or API, or
• The postgres add-on database.

# settings.py — write items to your own bucket as they're scraped
FEEDS = {
    "s3://your-bucket/%(name)s/%(time)s.jsonl": {"format": "jsonlines"},
}

Isolation & resources

Each job runs in its own hardened container. What that means for your spider:

• Read-only filesystem — only /tmp is writable. /tmp is a modest in-memory scratch area (on the order of a few hundred MB) and it counts toward the job’s memory. If you buffer large feeds locally before upload, stream them or keep batches small.
• Memory, CPU, and process limits apply per job — write spiders that stream rather than hold everything in memory.
• No shared state between jobs — each run starts clean and its container is removed when it finishes. Don’t rely on anything written in a previous run; persist to your database or feed instead.
• Runtime is bounded by your spider. Use CLOSESPIDER_TIMEOUT / CLOSESPIDER_ITEMCOUNT to cap a run deterministically.