Crawlers
Run your Scrapy spiders on the platform. You bring a Scrapy
project with a Dockerfile; the platform builds it into an image and runs each
spider as an isolated container — on demand or on a schedule. Inside a run,
your proxy, browser, and persona access is handed to the spider automatically,
and you watch every job’s logs, item counts, and status from the dashboard.
You never provision a machine, install Scrapy anywhere, or babysit a queue. You push code, press run (or set a schedule), and read your data out the other side.
Everything here lives under the Crawlers tab in the dashboard. It’s a thin
layer over standard Scrapy — if it runs with scrapy crawl locally, it runs
here. There’s no framework to adopt and no SDK to install.
The pieces
| Concept | What it is |
|---|---|
| Project | A named home for one Scrapy codebase. You deploy code into it; the platform builds it. |
| Build / Version | Each deploy of a project produces a build. A successful build becomes a version you can run. |
| Spider | A spider inside your project, discovered automatically from the built image (scrapy list). |
| Job | One run of one spider, in its own container. Has live logs, item/error counts, and a final status. |
| Schedule | A cron rule that fires jobs for you on a recurring basis. |
| Alert | A rule that watches job metrics (errors, item counts, runtime) and notifies you — or stops a bad run. |
How it flows
- Create a project in the Projects tab.
- Deploy your code — connect a deploy key and push your Scrapy project (with a
Dockerfile). The platform builds it into an image automatically. See Deploying a Project. - Run a spider — from the Run Spider tab, or set a Schedule. See Running & Scheduling.
- Inside the run, your spider reads the environment the platform injects to reach proxies, managed browsers, personas, and an optional database. See Inside a Job.
- Write your data to your own destination (a Scrapy feed or item pipeline), and watch the job’s logs and metrics — with optional alerts on failures. See Monitoring & Alerts.
The dashboard at a glance
Under Crawlers you’ll find:
- Overview — recent activity and health at a glance.
- Jobs — every run, with filters, live logs, and progress charts.
- Run Spider — start a run: pick a project, version, and spider; pass arguments and settings.
- Schedules — recurring runs on a cron, with a point-and-click builder or raw cron.
- Projects — your projects, build status, versions, and which add-ons each one uses.
- Deploy — deploy keys, the ready-made CI workflow, API keys, and secrets.
- Monitoring & Alerts — live running-job views and threshold-based alerting.
New here? Start with Deploying a Project — it walks a bare Scrapy project all the way to a first successful build.