Skip to content
Six Figure Jobs
True Talent Solutions Partners logo

True Talent Solutions Partners

Research Crawling Engineer

  • Remote · United States
  • Posted 4d ago
  • Full-time

Salary

$160K - $250K

Listed in the posting

Pays more than 69% of 15,869 six-figure listings in USD.

This job
$160k–$250k
Typical
$175k
Most pay
$145k–$215k
Apply on company site

Requirements

  • Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline ex
  • Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.
  • Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.
  • Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and dynamic sites.
  • Experience with headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
  • Experience with web data quality, benchmarking, cleaning, deduplication, and normalization; NLP pipelines or ML dataset curation are a plus.

Skills

Nice to have

Full job description

Description

About the Role

Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.

What You'll Do

  • Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.

  • Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.

  • Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.

  • Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.

  • Collaborate with research teams to shape datasets for research and model training.

  • Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.

  • Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

Location

Remote.

Requirements

What We're Looking For

  • Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.

  • Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.

  • Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.

  • Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and dynamic sites.

  • Experience with headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.

  • Experience with web data quality, benchmarking, cleaning, deduplication, and normalization; NLP pipelines or ML dataset curation are a plus.

  • Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.

Benefits

Compensation & Benefits

Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.

Apply on company site

Opens True Talent Solutions Partners’s own application page in a new tab.

Get jobs like this one by email

Research Crawling Engineer jobs · USA

Only new matching jobs, nothing else. Confirm by email; unsubscribe in one click.

Explore more

Salary

$160K - $250K

Apply