Storage

Every organization with the S3 Object Storage addon gets a private, S3-compatible bucket. Point any S3 tool at it — boto3 , rclone , the AWS CLI, or Scrapy’s built-in feed export. Your spiders can write to it with zero configuration, and you can browse and download everything from the dashboard.

Your data is encrypted at rest. Objects are encrypted before they ever reach the underlying storage, so only you can read them — the platform hands back plaintext transparently whenever you or your spiders read a file.

From your spiders (zero config)

When a job runs with the storage addon granted, the platform injects a ready-to-use credential into the job’s environment — no keys to manage, no setup:

Variable	What it is
`INSIGHT_STORAGE_ENDPOINT`	The S3 endpoint to connect to
`INSIGHT_STORAGE_BUCKET`	Your organization’s bucket name
`AWS_ENDPOINT_URL`	Same endpoint, under the standard AWS name
`AWS_ACCESS_KEY_ID`	Your access key
`AWS_SECRET_ACCESS_KEY`	Your secret key

The AWS_* names are what boto3 and Scrapy already look for, so most tools work with no code at all.

Scrapy — write results straight to the bucket

Point FEEDS at an s3:// URL built from the injected bucket name:


# settings.py
import os
 
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
AWS_ENDPOINT_URL = os.environ["AWS_ENDPOINT_URL"]      # Scrapy >= 2.6
 
FEEDS = {
    f"s3://{os.environ['INSIGHT_STORAGE_BUCKET']}/runs/%(name)s/%(time)s.jsonl": {
        "format": "jsonlines",
    }
}

Every run now lands under runs/<spider>/<timestamp>.jsonl in your bucket.

boto3 — read and write directly


import os
import boto3
from botocore.config import Config
 
s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["INSIGHT_STORAGE_ENDPOINT"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    # Path-style addressing is required (see Rules below).
    config=Config(s3={"addressing_style": "path"}, request_checksum_calculation="when_required"),
)
bucket = os.environ["INSIGHT_STORAGE_BUCKET"]
 
s3.put_object(Bucket=bucket, Key="runs/demo/result.json", Body=b'{"ok": true}')
obj = s3.get_object(Bucket=bucket, Key="runs/demo/result.json")
print(obj["Body"].read())

Browsing files in the dashboard

Open Storage in the dashboard sidebar to browse your bucket: navigate folders, see file sizes and timestamps, and download any file with one click. Downloads stream straight from storage, decrypted — you never handle keys in the browser.

Accessing your bucket from outside

To reach your bucket from your own machines (a laptop, a server, CI), reveal your credentials in the dashboard under Proxy Users → Storage Access (S3). That panel gives you the endpoint, bucket, and keys, plus a ready-to-paste rclone config.


# rclone.conf
[insight]
type = s3
provider = Other
access_key_id = <your access key>
secret_access_key = <your secret key>
endpoint = <endpoint from Storage Access>
force_path_style = true
no_check_bucket = true


rclone ls insight:<your-bucket>
rclone copy ./report.csv insight:<your-bucket>/uploads/
rclone copy insight:<your-bucket>/runs/ ./downloaded-runs/

Rules

⚠️

Use path-style addressing. Set force_path_style = true (rclone) or addressing_style: "path" (boto3). Virtual-host style (bucket.endpoint) is not supported.

Pack your output. Storage is optimized for run outputs and exports, not millions of tiny objects. Write compressed, batched files (one jsonl/csv per run) rather than one object per item — it’s faster and cheaper.
Large uploads: avoid multipart. In rclone the config above is enough; in boto3, raise multipart_threshold (via TransferConfig) so files upload in a single request.
Metered by size. Storage is billed on how much you keep, measured daily. Delete data you no longer need.
Not a hot store. Best for archival and exports; expect archival, not key-value, latencies.