Re-running a batch: duplicates, credits and only new results
Each run of a lobstr.io batch processes every task again. What Skip duplicates removes, what is billed again, and how to collect only new results.
Each run of a batch processes its whole task list again. So when you launch a batch again, by hand or on a schedule, results from earlier runs can come back. Here is what is billed again, and how to get only what's new.
Skip duplicates
Skip duplicates is in the batch's settings, under Advanced settings:
| Option | What it skips |
|---|---|
| Off | Nothing |
| This run | A result returned by several tasks of one run appears once in that run |
| This batch | Also what this batch collected in earlier runs |
| All my batches | Also what any of your batches of the same scraper collected |
This batch and All my batches are there on the scrapers that support them, such as Google Maps. Only results you can still download count: 7 days on Free, 45 on Starter and Pro, 90 on Team. Skipped results are not billed.
What is billed again
A result that comes back in a later run, without This batch or All my batches, is a new result for that run: its price is billed again. lobstr.io fetches it again from the site on every run, which costs real resources: proxies, compute, captcha solving.
Most extras work differently: once this batch has run an extra on a result, it skips it on later runs.
| In the same run | In a later run of the batch | |
|---|---|---|
| The result | Billed once | Billed again, unless skipped |
| An extra already run on it by this batch (most extras) | Billed once | Not run, not billed |
Example: Facebook Marketplace with Get Listing Details on. A new listing costs 1 credit, plus 4 for its details. A listing the batch already collected costs 1 credit when it comes back: its details aren't fetched or billed again.
A run's Credits tab shows what it spent on results and on each extra.
Only new results
Newer than
Many scrapers have a Newer than setting: only results from the last hours, days or weeks, or since a date. On a schedule, match it to the interval: the last 24 hours for a batch that runs every day, the last 7 days for a weekly one.
On Facebook Marketplace, which doesn't sort listings by date, Newer than skips older listings one by one instead of stopping the run. Skipped listings aren't billed.
The site's own date filter
No Newer than on your scraper? If the site's search has a date filter, such as "listed in the last 24 hours", apply it on the site, copy the URL, and add it as a task. Then schedule the batch at the same interval.
The scraper's own skip option
- Sales Navigator Leads Scraper: Skip Collected Leads
- LinkedIn Leads Scraper: Only New Leads
A maximum of results
A batch's maximum of results counts results from earlier runs too. If the search returns the same results in the same order every time, a new run can reach the maximum with results you already have: combine it with Newer than or Skip duplicates.