Monitoring & Alerts

Every run is observable end to end — status, live logs, and metrics parsed straight from Scrapy — and you can have the platform watch those metrics for you and act when something looks wrong.

Jobs

The Jobs tab lists every run, filterable by project, spider, status, tags, or free-text search. A job moves through these states:

Open any job to see its detail.

Live logs

Each job streams its logs in real time — follow a run as it happens, or read the full log after the fact, filtered by level. Logs are retained per job so you can go back to a previous run.

Metrics & progress

The platform parses your Scrapy log as it runs and captures, with no configuration on your part:

• Items scraped, pages crawled, errors and warnings
• HTTP response-code breakdown and retry counts
• The finish reason and full Scrapy stats
• A progress chart — items and pages over the life of the run

The Overview and Monitoring tabs roll this up: recent activity, currently running jobs, and items-per-job while runs are in flight.

Alerts

In the Alerts tab, define rules that watch job metrics and notify you — or intervene — automatically.

A rule is a metric, an operator, and a threshold:

• Metrics: error_count, warning_count, items_scraped, pages_scraped, runtime_seconds, retry_count.
• Operators: >, >=, <, <=, ==, !=.

For example: error_count > 50, or items_scraped < 1 to catch a run that silently produced nothing.

When a rule trips it can:

• Notify through a channel — Slack, email, or a webhook.
• Optionally stop or force-stop the offending job, so a runaway or broken run doesn’t keep burning time.

Rules support cooldowns and active windows (working hours / days, in your timezone) so you’re alerted when it matters and not flooded. A history of fired alerts is kept for review.