Skip to content
Batches

Re-running a batch: duplicates, credits and only new results

Each run of a lobstr.io batch processes every task again. What Skip duplicates removes, what is billed again, and how to collect only new results.

Each run of a batch processes its whole task list again. So when you launch a batch again, by hand or on a schedule, results from earlier runs can come back. Here is what is billed again, and how to get only what's new.

Skip duplicates

Skip duplicates is in the batch's settings, under Advanced settings:

Option What it skips
Off Nothing
This run A result returned by several tasks of one run appears once in that run
This batch Also what this batch collected in earlier runs
All my batches Also what any of your batches of the same scraper collected

This batch and All my batches are there on the scrapers that support them, such as Google Maps. Only results you can still download count: 7 days on Free, 45 on Starter and Pro, 90 on Team. Skipped results are not billed.

What is billed again

A result that comes back in a later run, without This batch or All my batches, is a new result for that run: its price is billed again. lobstr.io fetches it again from the site on every run, which costs real resources: proxies, compute, captcha solving.

Most extras work differently: once this batch has run an extra on a result, it skips it on later runs.

In the same run In a later run of the batch
The result Billed once Billed again, unless skipped
An extra already run on it by this batch (most extras) Billed once Not run, not billed

Example: Facebook Marketplace with Get Listing Details on. A new listing costs 1 credit, plus 4 for its details. A listing the batch already collected costs 1 credit when it comes back: its details aren't fetched or billed again.

A run's Credits tab shows what it spent on results and on each extra.

Only new results

Newer than

Many scrapers have a Newer than setting: only results from the last hours, days or weeks, or since a date. On a schedule, match it to the interval: the last 24 hours for a batch that runs every day, the last 7 days for a weekly one.

On Facebook Marketplace, which doesn't sort listings by date, Newer than skips older listings one by one instead of stopping the run. Skipped listings aren't billed.

The site's own date filter

No Newer than on your scraper? If the site's search has a date filter, such as "listed in the last 24 hours", apply it on the site, copy the URL, and add it as a task. Then schedule the batch at the same interval.

The scraper's own skip option

A maximum of results

A batch's maximum of results counts results from earlier runs too. If the search returns the same results in the same order every time, a new run can reach the maximum with results you already have: combine it with Newer than or Skip duplicates.