Skip to content
Export and delivery

Export scraped data automatically to Amazon S3

Attach an AWS S3 bucket to your batch so every completed run uploads its results there automatically — including how to create the bucket, the IAM policy, and the access keys.

If your data pipeline already reads from S3, you can skip the download-and-upload step entirely. Attach a bucket to your batch and lobstr.io writes the results there itself every time a run completes.

Most of the work is on the AWS side: a bucket, a narrow permissions policy, and a dedicated user whose keys you paste into lobstr.io. The lobstr.io part is three fields.

Before you start

Part 1 — Set up AWS

  1. Create an S3 bucket

    Open the S3 console and click Create bucket. Set the region using the dropdown in the top-right corner before you start.

    Enter a globally unique bucket name using only lowercase letters, numbers, and hyphens.

    Create an S3 bucket with a globally unique name in the AWS console

    Leave everything else at its defaults — Bucket type on General purpose, Bucket namespace on Global, and Block all public access switched on. Click Create bucket.

    Avoid dots in the bucket name. They're technically allowed, but they break TLS certificate matching for virtual-hosted-style requests and cause problems that are annoying to diagnose later.

  2. Create a permissions policy

    Go to IAM → Policies → Create policy. Switch the editor to the JSON tab and replace the contents with this, substituting your own bucket name:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "s3:PutObject",
          "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
      ]
    }
    Paste the S3 PutObject policy into the IAM JSON policy editor

    Click Next, name it lobstr-delivery-write, and click Create policy.

    This grants permission to write files to that one bucket and nothing else — it can't read, can't delete, and can't touch any other bucket in your account.

  3. Create a dedicated IAM user

    In IAM, go to Users and click Create user. Name it lobstr-delivery and leave the console-access checkbox unticked — this user authenticates with keys only.

    Create an IAM user without AWS Management Console access

    On the next screen, choose Attach policies directly, tick lobstr-delivery-write, then click through and create the user.

    Attach the lobstr-delivery-write policy directly to the new IAM user
  4. Generate access keys

    Open the new user and click Create access key in the Access key 1 panel.

    Create access key from the IAM user summary panel

    Choose Application running outside AWS, skip the description tag, and click Create access key.

    Select Application running outside AWS as the access key use case

    Copy both the Access key and the Secret access key now.

    Retrieve the access key and secret access key on the final screen

    The secret access key is shown only on this screen and can't be retrieved afterwards. Copy it now, or download the .csv. If you lose it, you'll have to create a new access key and deactivate the old one.

Part 2 — Connect it to your batch

  1. Add S3 as a delivery method

    In your lobstr.io dashboard, open the batch, go to its Delivery settings, and expand the Amazon s3 section.

    Enter the bucket name in Bucket, then paste the access key and secret key into their fields.

    Fill in the bucket, AWS access key, and AWS secret key in the Amazon S3 delivery settings

    Enter the bucket name on its own — my-bucket, not the full s3://my-bucket URL.

  2. Test the connection

    Click Test Amazon S3 before saving. A pass confirms the credentials and permissions are correct.

    If it fails with an access-denied error, the bucket name in the policy ARN doesn't match the bucket name in the form.

  3. Enable and save

    Tick the checkbox beside Amazon s3 to activate the integration, then click Save.

    Filled-in credentials with the box left unticked means deliveries won't fire. The fields and the checkbox are separate — you need both.

  4. Run the batch and verify

    Click Launch. When the run completes, open the bucket in the S3 console — the results file appears there within about a minute.

    Files land at the root of the bucket, named after the batch plus the run's UTC timestamp:

    Google Maps Reviews Scraper (12)_2026-09-03T16:03:06.899628+00:00.csv
    The results CSV delivered to the root of the S3 bucket after a run completes

    Every subsequent run drops a new file automatically, including scheduled ones. Nothing is overwritten — the timestamp keeps each run's file distinct.

Troubleshooting

Access denied

The bucket name in the policy ARN doesn't match the bucket you entered in lobstr.io. Open the policy in IAM and check the Resource line — it needs your real bucket name and must end in /*.

No such bucket

Either the bucket name is mistyped, or the bucket is in a different region than the one you're looking at. Bucket names are global, so a typo can also mean you're pointing at a bucket that isn't yours.

Invalid credentials

Usually the secret key was copied with trailing whitespace. Re-paste both values carefully. If the secret is lost, create a new access key in IAM and deactivate the old one.

The test passes but no files appear

Check the Amazon s3 checkbox is ticked and that you clicked Save. A successful test doesn't activate the delivery on its own.

Things to know

  • Delivery applies to future runs only. Runs that finished before you set this up aren't uploaded retroactively. To backfill, download past runs from the Runs table and upload the files yourself.
  • S3 delivery is per batch. Configure it separately on each batch you want writing to a bucket.
  • You can combine delivery channels. Amazon S3, Google Sheets, SFTP, email, and webhooks each have their own checkbox in the Delivery panel and can all be active at the same time.
  • Rotate the key if it's ever exposed. Create a new access key in IAM, paste it into the Delivery panel, save, then deactivate the old key. The narrow policy means the worst case is writes to one bucket — which is exactly why the dedicated user is worth the extra two minutes.