Skip to content
Six Figure Jobs
True Talent Solutions Partners logo

True Talent Solutions Partners

Machine Learning Infrastructure Engineer

Salary

$200K - $400K

Listed in the posting

Pays more than 93% of 173 machine learning engineer listings in USD.

This job
$200k–$400k
Typical
$210k
Most pay
$188k–$251k
Apply on company site

Requirements

  • Two to ten years of experience building large-scale infrastructure for pretrained or foundation models, including compute clusters, data infrastructure, inference, and distributed training.
  • Hands-on experience optimizing large-scale training, data, or inference workloads, and using distributed training frameworks such as FSDP or DeepSpeed.
  • Experience with low-level GPU performance analysis and debugging, including CUDA and JAX.
  • Familiarity with Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.
  • Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.
  • Comfort working across the ML lifecycle in a fast-paced environment, including analyzing experiments and ablation studies.

Benefits

  • Relocation

Skills

AI
AWS
Azure
Cloud Computing
Containerization
C++
Cuda
Deepspeed
Distributed Training
Docker

Nice to have

Full job description

Description

About the Role

Join an AI team building large-scale models for physical systems, with weather as an initial application. You will develop the distributed infrastructure needed to train and serve foundation models, helping advance AI that can predict outcomes and identify effective actions.

What You'll Do

  • Design, deploy, and maintain distributed machine learning training and inference clusters.

  • Build scalable end-to-end pipelines for petabyte-scale datasets and manage workloads across the machine learning lifecycle.

  • Research and evaluate training approaches, including parallelization methods and numerical precision trade-offs at different model scales.

  • Profile and debug low-level GPU operations to improve performance.

  • Follow relevant research and apply promising ideas to infrastructure and model workflows.

Location

On-site in San Francisco, California. Relocation support is provided.

Requirements

What We're Looking For

  • Two to ten years of experience building large-scale machine learning infrastructure for pretrained or foundation models.

  • Hands-on experience with compute clusters, data infrastructure, distributed training, and inference systems.

  • Expertise optimizing large-scale training, data, or inference workloads, including low-level GPU performance analysis.

  • Experience with distributed training frameworks such as FSDP or DeepSpeed; familiarity with CUDA, JAX, Python, C++, Linux, or NVIDIA hardware is valuable.

  • Knowledge of cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.

  • Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and comfort working across the machine learning lifecycle.

  • Familiarity with scalable task management and model serving, observability practices, and experiment analysis such as ablation studies.

  • A track record of working effectively in fast-paced environments and on time-sensitive projects.

Benefits

Compensation & Benefits

Base salary: $200,000 to $400,000 annually. Visa sponsorship is not available.

Apply on company site

Opens True Talent Solutions Partners’s own application page in a new tab.

Get jobs like this one by email

Machine Learning Engineer jobs · USA

Only new matching jobs, nothing else. Confirm by email; unsubscribe in one click.

Explore more

Salary

$200K - $400K

Apply