Batches: collect data on a whole list, now or every day
A lobstr.io batch runs a scraper on your list of inputs, by hand or on a schedule, and delivers the results as CSV, to Google Sheets or to S3. Create, set up, rename.
A batch runs a scraper on a list of inputs (search URLs, places, profiles, keywords) and collects the results for you: by hand when you launch it, or on a schedule. No code.
A batch is made of:
- A scraper: what to collect, on which site.
- Tasks: your list of inputs, one task each.
- Settings: extra data, which results to keep, its slots.
- A schedule, if it runs on repeat.
- A delivery, if its results go to Google Sheets, S3 or email.
Each time it runs, it makes a run, with its own results.
For one place, one listing or one page of results right now, an operation is quicker: one call, the records back in seconds. A batch is for lists and schedules. In the API v1, a batch is called a squid.
Create a batch
- Click New batch
In the app, click Batches in the sidebar, then New batch.
- Pick a scraper
Search a site or the data you need, and click the scraper. The batch is created and opens on its settings.
- Add your tasks and settings
Add your tasks, choose your settings, then click Save and launch, or Save to launch it later.
From an operation's page in the store, In bulk, every day creates a batch for the same data.
Two things can stop a batch from launching:
- The scraper needs a paid plan. Some scrapers are marked so in the store: see What Free doesn't include.
- Every slot is busy. See When every slot is busy.
Rename a batch
lobstr.io names a batch when you create it. To rename it, open the batch, click Settings, click its name at the top of the page, type the new one, and click Save.
Batch names are unique: a name another of your batches uses is refused.
Its runs and results
The batch's page lists its runs, newest first, with their status, results and credits. See Runs.
Results can be downloaded for 7 days on Free, 45 days on Starter and Pro, and 90 days on Team: see Download results.