Skip to content
Batches

Tasks: the inputs that tell a batch what to scrape

Tasks are the inputs for a lobstr.io batch — URLs, queries, usernames, or structured params. Learn how tasks are added, deduplicated, and re-processed every run.

A task is a single unit of input for a batch. It's what you hand the scraper so it knows what to scrape — a URL to visit, a query to search, a username to look up, or a set of structured parameters. Each batch can hold up to 10,000 tasks, and every run you launch processes all of them.

What a task looks like

The accepted input type depends on the batch. Common shapes:

  • URL — a link to a page, list, or profile (e.g. https://www.realtor.com/realestateandhomes-search/California/pg-5).
  • Plain text query — a search term (e.g. restaurant).
  • Username — a handle or ID (e.g. a Twitter handle or YouTube channel).
  • Structured parameters — fields like Category, Country, Region, District, City (used by batches like Google Maps Leads Scraper).

Some batches only accept one shape. Others — like Google Maps Leads Scraper — accept both URLs and structured params. The Tasks section of the batch's settings always shows you what that specific batch expects.

How to add tasks

In the batch's settings, the Tasks section takes your inputs in one of its modes, depending on the scraper: By URL, By location (or By keyword), or By area for a whole country, region or city on Google Maps (see Search a whole area).

One at a time

Paste a URL, a query or a username and add it. Good for a few tasks.

From a file

For long lists, up to 10,000 tasks:

  1. Open Upload tasks

    In the Tasks section, click Upload file (or Upload). The window says which column your file needs, such as url, and offers a template to download.

  2. Build your file

    A .csv, .tsv or .txt file of up to 10 MB: the column's name in the first cell, then one task per row.

  3. Import it

    Drop the file in the window, check its columns, and import it. The import shows its progress in the Tasks section.

The column's name must match exactly, capitals included. With a wrong header, nothing is imported.

By location

On scrapers that take structured inputs (a category, a country, regions, cities), pick them in the form: every combination becomes one task.

Deduplication and validation

Two things happen automatically when you add tasks:

  • Deduplication — if the same task is already in the list, it's not added again. This prevents accidental double-processing when you upload overlapping CSVs.
  • Validation — tasks that don't match the batch's expected input format are skipped. If you paste a URL into a batch that expects a username, that task won't be added.

If an upload brings fewer tasks than your file has rows, the missing ones were either duplicates already in the list or invalid format.

When tasks won't add at all

Fewer tasks than expected is normal. Zero tasks added, or an upload that's rejected outright, usually comes down to one of these:

What you're seeing Cause Fix
Nothing imported from a valid-looking file The column's name doesn't match what the batch expects, character for character Copy it from the Upload tasks window, or start from its template
The file is rejected before any import happens lobstr.io couldn't read the file's character encoding Re-save as CSV UTF-8 from Excel, Google Sheets, or your editor
A large file is rejected It is over 10 MB Split it into smaller files and upload them one after another: tasks add up
Some rows added, others silently skipped Those rows were duplicates, or the wrong input type for this batch Check the format the batch expects — a URL won't be accepted where a username is required
The count stops growing The batch has reached 10,000 tasks Split the work across several batches

A run needs at least one task. Launching a batch with an empty task list won't start one — add tasks first, then launch.

The task list

The Tasks section lists the batch's tasks, 10 per page. You can:

  • Search tasks to find one.
  • Download the list as a CSV.
  • Tick tasks and remove them.
  • Remove all: the batch keeps its settings, schedule and results, but every task goes, done or not. There is no undo.

How tasks work with runs

This is the most important thing to understand about tasks:

Every run re-processes every task in the list. There's no queue that drains over time. If your batch has 500 tasks, each launch scrapes all 500 — not only "new" ones.

This model is intentional — it lets you scrape fresh, up-to-date data on a schedule without manually re-adding tasks. It also means:

  • To scrape something new, add new tasks to the list before the next launch.
  • Tasks you add while a run is in progress join that run: it checks the list again each time a scraper finishes a task. If the run has already finished, they wait for the next launch.
  • To stop scraping certain tasks, remove them from the list (or Remove all and start over).
  • If you want every run to pull fresh data for the same inputs (e.g. a weekly Google Maps refresh), just leave the list alone and schedule the batch.

Each task's results in a run

A run's Tasks tab shows what each task brought:

Column What it shows
Task The input: URL, query, username or parameters
Status Where the task is
Results Results it collected
Pages Pages the scraper went through
Ended When it finished
Reason Why it ended

When one task brings 0 results and the others hundreds, its Reason usually says why. A task that failed can be retried from its row.