Validation Engineer - AI Infrastructure

Are you looking for an exciting new opportunity?

Join a fast-scaling UK AI infrastructure owner-operator building sovereign, renewable-powered GPU compute environments. By converting legacy brownfield and energy sites into modern data center facilities, the organization delivers high-density bare-metal APIs, Kubernetes clusters, and low-carbon AI capacity to enterprise, healthcare, financial, and public-sector clients across the UK.

The company is seeking a Validation Engineer to join its Product team and own validation end-to-end across Product, Engineering, and Operations. The successful candidate will design and build automated preflight suites and performance benchmarks for multi-node GPU clusters, validating platform reliability, hardware deployments, and infrastructure releases prior to enterprise customer deployment.

If you would like to learn more about this opportunity, feel free to reach out and apply today!

Responsibilities:

• Design and build zero-dependency, Python-based preflight tools that validate multi-node distributed initialisation, master-to-worker rendezvous routing and GPU-to-CPU process affinity before large training workloads launch
• Write and maintain Infrastructure as Code to provision, configure, test and tear down complex bare-metal and containerised compute environments
• Build adaptive test orchestration harnesses that autonomously detect environment drift and analyse the impact of platform changes
• Automate cluster-level and high-performance storage benchmarking, aggregating results to establish node-to-node throughput limits
• Script low-level checks for server topology, PCIe link speeds, HBM memory status, secure boot parameters and firmware performance via BMC, IPMI or Redfish
• Embed validation suites into CI/CD pipelines, build observability pipelines that isolate slow or degrading nodes, and lead root-cause analysis on platform regressions and interconnect bottlenecks alongside Platform, Network and Data Centre teams

Skills/Must have:

• Proven, hands-on validation or benchmarking experience in enterprise-scale environments. This is a specialist validation role, not a platform or systems engineering role with some testing exposure.
• Strong Python (zero-dependency tooling, test harnesses, low-level system checks), Infrastructure as Code for bare-metal and containerised environments, and Kubernetes
• Validating multi-node distributed systems; hardware interfaces (BMC, IPMI, Redfish); CI/CD integration of validation suites; observability pipelines for diagnostics and hardware metrics
• A rigorous, evidence-led mindset; the confidence to hold the line on release readiness; and a collaborative approach to cross-functional root-cause analysis

Desirable Skills:

• Validation of infrastructure optimised for large-scale model training, including GPU-to-CPU affinity and rendezvous routing
• Deep understanding of high-throughput network fabrics, traffic patterns and congestion control in multi-node GPU clusters
• Experience building cluster-level and storage benchmarking suites to isolate throughput limits
• Experience designing intelligent test systems that detect environment drift autonomously

Benefits:

• Performance related bonus

Salary:

• £100,000 to £110,000+ base (subject to client sign-off, see note below)