Customer Reliability Engineer - AI Infrastructure
- Competitive salary
- San Francisco, California, United States
- Permanent
- Artificial Intelligence
Ready to take the next step in your career?
Join a fast-growing AI infrastructure company building the marketplace and platform through which the world’s fragmented GPU supply is bought, sold, and operated. It connects model builders, research labs, and inference providers with reliable capacity from neoclouds, hyperscalers, and independent data centres worldwide.
The organization is expanding its team and is looking to appoint a Support Engineer for a fully remote, EU-based role. The ideal candidate will be the engineer customers hear from when a node goes dark or a training job fails mid-run, owning each issue from first triage through to resolution and building tooling to stop it recurring. Prior GPU or HPC experience is not required.
Ready to take your expertise to the next level? Get in touch today!
Responsibilities:
- Own customer issues end to end, debugging at the Linux and Kubernetes layer across processes, networking, storage, scheduling and kernel logs.
- Diagnose GPU failures including driver and device plugin issues, XID errors, thermal throttling and multi node job failures.
- Take part in a 24/7 on call rotation, leading first response on alerts and outages and keeping customers clearly updated throughout.
- Determine whether faults sit with the provider, the platform or the customer, then open and chase tickets with compute providers to resolution.
- Build Python and Bash automation alongside monitoring for cluster health, GPU telemetry, dashboards and alerting.
- Write post incident reviews and turn them into runbooks, alerts and fixes, keeping customer documentation current.
Skills/Must Have:
- Strong hands on Linux troubleshooting from the command line across logs, processes, networking and disk.
- Working knowledge of Kubernetes including pods, nodes, deployments, services and scheduling, and how to investigate when they break.
- Proficiency in Python or Bash for automating repetitive operational tasks.
- Excellent written communication, with the ability to explain technical problems clearly to frustrated customers.
- Sound judgement under pressure, knowing what to check first, when to escalate and how to keep stakeholders informed.
- Willingness to join a 24/7 on call rotation, based in the EU and working fully remote.
- Experience with NVIDIA GPUs in production (CUDA, DCGM), InfiniBand/RoCE or NCCL, Slurm, Prometheus/Grafana, Terraform/Ansible/Helm, or a previous customer facing technical role is a strong plus.
Benefits:
- Meaningful equity
- Comprehensive healthcare
- Unlimited PTO
Salary:
- Competitive salary