Staff Engineer, Test Automation - San Mateo (R5792)

Shield AI
San Mateo, CA, US
Posted 1 months ago
Posted 21 October 2026
full-time

About the Role

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai https://shield.ai/. Follow Shield AI on LinkedIn https://www.linkedin.com/company/shield-ai, X https://x.com/shieldaitech, Instagram https://www.instagram.com/shield_ai, and YouTube https://www.youtube.com/@Shield_AI. 

 

JOB DESCRIPTION:

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai https://shield.ai/. Follow Shield AI on LinkedIn https://www.linkedin.com/company/shield-ai, X https://x.com/shieldaitech, Instagram https://www.instagram.com/shield_ai, and YouTube https://www.youtube.com/@Shield_AI.

The Hivemind Software Engineering Integration and Test team is seeking a Staff Automated Test Engineer to provide technical leadership for automated verification across our next-generation autonomy platform. You will define and evolve test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines that enable the Hivemind software ecosystem to create, test, and deploy resilient autonomy capabilities for unmanned aircraft and robotic platforms operating in complex, contested, and GPS-denied environments.

You will work across production flight code, machine learning models, simulation and synthetic environments, mission planning and orchestration systems, operator-facing Ground Control Station applications, telemetry and data pipelines, cloud-native developer infrastructure, and hardware-in-the-loop systems.

In this hands-on Staff role, you will lead complex, cross-functional initiatives, establish automation and verification standards, identify systemic quality risks, and develop scalable test infrastructure. The ideal candidate combines deep Python automation and distributed-systems testing experience with technical leadership and experience validating machine learning systems throughout the data, training, evaluation, deployment, and monitoring lifecycle.

What you'll do:

  • Own the technical strategy, architecture, and roadmap for automated testing, verification, and MLOps quality across the Hivemind ecosystem.

  • Design and maintain scalable test frameworks for autonomy software, backend services, APIs, operator-facing applications, and distributed hardware environments.

  • Lead functional, integration, regression, system, performance, reliability, and end-to-end testing across simulation, edge-compute, software-in-the-loop, and hardware-in-the-loop environments.

  • Build automated ML validation pipelines covering data quality, training reproducibility, model accuracy, robustness, regression, latency, resource utilization, and system integration.

  • Establish CI/CD and continuous training workflows that provide versioning and traceability for datasets, models, configurations, evaluation results, and deployment artifacts.

  • Develop scenario-based validation for autonomy models, including edge cases, degraded sensing or communications, distribution shifts, and representative mission conditions.

  • Create observability, analytics, and failure-triage capabilities for software behavior, model and data drift, inference health, test results, and production performance.

  • Build Python automation that improves test execution, parallelization, reporting, environment setup, experiment comparison, and developer productivity.

  • Create test harnesses, simulators, stubs, mocks, and synthetic data capabilities that improve system testability and coverage.

  • Collaborate with software, autonomy, machine learning, data, simulation, and systems engineers to define verification strategies and improve designs before implementation.

  • Develop and govern AI-assisted engineering workflows using coding agents and LLM-based tools for test generation, log analysis, debugging, and failure triage while maintaining security, reproducibility, and traceability.

Required qualifications:

  • Typically 8+ years of relevant experience in software engineering, test infrastructure, developer tooling, MLOps, systems integration, or systems verification, or an equivalent combination of experience and demonstrated impact.

  • 5+ years of experience building scalable automation frameworks or developer tooling in Python.

  • Demonstrated success designing test, CI/CD, or MLOps infrastructure used across multiple engineering teams.

  • Experience validating machine learning systems across the data, training, evaluation, packaging, deployment, and monitoring lifecycle.

  • Understanding of ML quality risks such as data leakage, training-serving skew, nondeterminism, distribution shift, drift, model regression, and statistical acceptance criteria.

  • Experience defining model-performance baselines, automated evaluation suites, release thresholds, and candidate-to-production comparison workflows.

  • Experience testing GPU-accelerated infrastructure and workloads, including GPU scheduling, allocation, utilization, and resource contention in Kubernetes environments.

  • Experience with performance benchmarking, profiling, and observability for GPU workloads, including identifying compute, memory, storage, networking, and data-loading bottlenecks.

  • Experience validating multi-tenant Kubernetes environments, including RBAC, resource quotas, workload isolation, and scheduling behavior.

  • Experience qualifying integrated hardware and software systems, including automated validation of compute, GPU, storage, networking, drivers, firmware, and deployed software configurations.

  • Strong system-design skills and experience testing distributed systems, backend services, APIs, and integrated hardware and software environments.

  • Experience developing integration and regression strategies for internally developed, third-party, open-source, and partner software, including dependency management, compatibility testing, and upgrades.

  • Strong understanding of asynchronous and concurrent Python programming for scalable automation and parallel test execution.

  • Experience with package and dependency management, reproducible environments, and build systems such as Conan, pip, setuptools, Poetry, Nix, or similar.

  • Experience with automated observability, log collection, analytics, reporting, and root-cause analysis in complex software, data, and infrastructure systems.

  • Experience working in Linux-based development environments.

Preferred qualifications:

  • Experience with model registries, experiment tracking, dataset or feature versioning, model serving, and automated artifact promotion using MLflow, Kubeflow, Weights & Biases, SageMaker, Vertex AI, or similar platforms.

  • Experience with GPU scheduling and orchestration platforms such as Run:ai, NVIDIA GPU Operator, KAI Scheduler, Kueue, Volcano, or similar technologies.

  • Experience with NVIDIA GPU infrastructure, including CUDA, drivers, container runtimes, Multi-Instance GPU, GPU fractionalization, and hardware/software compatibility testing.

  • Experience with GPU profiling and performance-analysis tools such as NVIDIA Nsight, PyTorch Profiler, or similar technologies.

  • Experience validating perception, planning, decision-making, reinforcement learning, or other autonomy models in simulation and on deployed systems.

  • Experience testing models on embedded or edge-compute platforms, including latency, memory, power, accelerator compatibility, quantization, and hardware-specific behavior.

  • Experience qualifying production servers or appliances, including hardware validation, burn-in, provisioning, firmware, networking, storage, and software-stack validation before deployment.

  • Experience with reliability, fault-injection, and recovery testing across distributed compute, storage, networking, and GPU infrastructure.

  • Experience validating reproducible installation, operation, upgrades, and rollback in cloud, on-premises, disconnected, or air-gapped environments.

  • Experience with containers, Kubernetes, cloud infrastructure, infrastructure as code, and reproducible test environments.

  • Proficiency with Go or TypeScript for automation tooling or UI test development.

  • Experience integrating Python with native C or C++ applications through bindings, wrappers, subprocess interfaces, or similar interoperability tooling.

  • Aerospace, robotics, autonomy, embedded systems, or safety-critical software experience.

  • Familiarity with software-in-the-loop, hardware-in-the-loop, requirements-based verification, configuration management, artifact traceability, or standards such as DO-178C and MIL-STD-882.

#LI-DS5

#LD

 

Full-time regular employee offer package:

Pay within range listed + Bonus + Benefits + Equity

 

Temporary employee offer package:

Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

 

Salary compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certifications, and specific work location. All offers are contingent on a cleared background and possible reference check. Military fellows and part-time employees are not eligible for benefits. Please speak to your talent acquisition representative for more information.