Skip to content
Six Figure Jobs
PsiQuantum logo

PsiQuantum

Staff Software Engineer (Observability/Site Reliability)

  • Palo Alto, United States
  • Onsite
  • Posted 3d ago
  • Full-time
  • Staff / Principal
  • 8+ yrs experience

Salary

$236K - $277K

Listed in the posting

Pays more than 86% of 1,439 software engineer listings in USD.

This job
$236k–$277k
Typical
$195k
Most pay
$164k–$228k
Apply on company site

Benefits

  • Generous PTO
  • 401(k) match
  • Parental leave
  • Retirement plan
  • Health insurance
  • Mental health

Skills

AI
API
Automation
AWS
CI/CD
Clickhouse
Cloud Computing
Distributed Systems
DNS
GitLab

Nice to have

Full job description

Description

  • The Observability team builds and runs the telemetry platform that lets PsiQuantum see how its quantum computing systems, and the infrastructure behind them, are behaving. We collect metrics, logs, and traces from on-premises Kubernetes clusters, cloud infrastructure, and the custom hardware that makes up a photonic quantum computer, including control electronics and cryostats. Engineering, operations, hardware, and research teams rely on this data to run the machine, catch problems early, and work out what went wrong when something fails
  • We are looking for a Staff Software Engineer to join the Observability team. You will wire up OpenTelemetry instrumentation and collection, run and scale Prometheus and Grafana, and turn raw signals into dashboards and alerts that people trust. A large part of the job is working with hardware teams: deciding what their devices should report, and getting that data off the device and into the platform reliably
  • This is a hands-on role with Staff-level scope. You will own major parts of the observability stack from design through operation, share the team’s on-call rotation, and set the conventions other teams follow when they instrument their own systems. You will combine hands-on delivery with technical leadership, sound judgment under ambiguity, and a strong bias toward simple, maintainable systems
  • Design, deploy, and operate the observability platform across on-premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces
  • Roll out OpenTelemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs and instrumentation in Python, Rust, and other languages
  • Run Prometheus in production, including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage
  • Manage Grafana dashboards, alerts, and data sources as code so they are versioned, reviewed, and reproducible across environments
  • Work with hardware teams to instrument custom devices, control electronics, cryostats, and other lab equipment; build the exporters and collectors that bring device telemetry into the platform, including from equipment that was never designed to be monitored
  • Define and publish telemetry standards - metric and label naming, cardinality budgets, structured logging, trace context propagation, and resource attributes - so data from different teams and systems can be correlated
  • Partner with service owners and operations teams to define SLOs and alerts that reflect real impact, reduce alert noise, and make every page actionable
  • Take part in the team’s on-call rotation: respond to incidents affecting the observability platform, help other teams diagnose production issues with telemetry, and lead blameless post-incident reviews through to completed fixes
  • Run the platform as a production service in its own right, with capacity planning, cost management, upgrades, backup and recovery, and monitoring of the monitoring stack itself, delivered through GitLab CI/CD and infrastructure as code
  • Use AI and LLM tooling to accelerate engineering and operational work responsibly - including coding assistants, query and dashboard generation, and incident analysis - while applying rigorous review, security, and validation practices
  • Work with the team’s technical lead on architecture and roadmap; lead design reviews, mentor engineers, document consequential decisions, and coordinate with software, hardware, research, infrastructure, and operations teams to turn their observability needs into a shared plan
  • Instrument and collect: Configure OTel Collectors and Prometheus scrape targets, write exporters, and help teams instrument new services and devices
  • Build and review: Write code and configuration, review merge requests for platform and dashboard changes, and simplify pipelines that are hard to understand or operate
  • Work with hardware teams: Learn how a new device or cryostat behaves, agree on what to measure and alert on, and get its data flowing
  • Watch and respond: Check platform health and alert quality, handle on-call pages, and help other teams debug with metrics, logs, and traces
  • Shape the platform: Work through storage, retention, cost, and tooling tradeoffs with the technical lead; unblock other engineers; keep near-term work aligned with the roadmap
  • Apply AI with judgment: Use LLM-assisted tools for queries, configuration, code, and documentation, then verify outputs to the standard expected for production infrastructure

Benefits

  • Financial & Retirement: 401(k) with company match, equity grants, and access to financial wellness tools and planning resources.
  • Health: Competitive health coverage for you and your dependents that go beyond your standard medical, dental & vision.
  • Wellness: $200 quarterly health reimbursement, 12 therapy + 6 coaching sessions with Modern Health for you and depndents, and annual membership to Care.com.
  • Time Off: Discretionary Time Off (DTO) for full-time employees in addition to federal holidays and an end-of-year company shut-down.
  • Family Leave: 12 weeks of paid parental leave with the support of a dedicated leave consultant.
  • Flexible WOrk: Work in office, hybrid, or remote depending on your team and location.
  • Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, dashboard design, and running both reliably at scale
  • Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing alerts people act on
  • 8+ years of professional software, infrastructure, or site reliability engineering experience, including significant time building and operating observability systems in production
  • Ability to work closely with hardware, electronics, and research teams across sites and time zones, in a fast-moving environment where requirements evolve as the quantum computer itself evolves
  • Proficiency with modern AI/LLM engineering tools, such as coding copilots and agents, model APIs, and retrieval workflows; a disciplined approach to privacy, security, hallucination risk, and generated-code review
  • Production experience with OpenTelemetry, including the Collector, OTLP, SDK and automatic instrumentation, and semantic conventions
  • Strong programming skills in Python and at least one of Go or Rust, with the ability to write production-quality exporters, Collector components, and automation rather than only configuring off-the-shelf tools
  • Solid knowledge of Linux and networking fundamentals, including processes, filesystems, resource limits, DNS, TCP/IP, and TLS, and experience debugging distributed systems from the host up
  • Staff-level technical leadership: leading design work, setting standards other teams adopt, mentoring engineers, influencing across teams, and communicating tradeoffs to technical and non-technical stakeholders
  • A Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or a closely related technical discipline is required
  • Hands-on experience running workloads on self-managed, on-premises Kubernetes and in the cloud, with containers, Helm, and infrastructure as code such as Terraform; practical experience with GitLab and GitLab CI is strongly preferred
  • Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse
  • Experience collecting telemetry from laboratory or industrial equipment over protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces, or through vendor SDKs
  • Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing
  • Experience with high-frequency or high-cardinality time-series data, including downsampling, aggregation, and data lifecycle management
  • Experience with eBPF-based observability, continuous profiling, or Kubernetes operators
  • Experience running observability across multiple sites and hybrid on-premises and cloud environments, including sites with constrained network connectivity
  • Experience with SLO tooling, synthetic monitoring, or incident management platforms such as PagerDuty
  • Familiarity with VAST Data, Kafka, and AWS
  • A background in physics or quantum computing is helpful but not required; curiosity, systems thinking, and motivation to learn the domain are essential
  • Contributions to OpenTelemetry, Prometheus, Grafana, or related open-source projects

Apply on company site

Opens PsiQuantum’s own application page in a new tab.

Get jobs like this one by email

Software Engineer jobs · USA

Only new matching jobs, nothing else. Confirm by email; unsubscribe in one click.

Explore more

Salary

$236K - $277K

Apply