Principal Software Engineer, GPU Firmware and GPU System Software — CSP Engageme

NVIDIA·Remote(Canada, United States)

Software Development

WFA Digital Insight

As the demand for specialized tech talent grows, NVIDIA stands out with its commitment to innovation. With the rise of remote work, the need for skilled engineers who can manage complex systems at scale has increased by 25% in the past two years. Principal Software Engineers with a deep understanding of GPU architecture and firmware are in high demand, and this role offers a unique chance to work with key CSPs and drive systemic improvements. Before applying, candidates should be aware of the importance of customer obsession and the ability to simplify complex technologies for fleet-scale customers.

Job Description

About the Role

The Principal Software Engineer for GPU Firmware and GPU System Software is a critical role within NVIDIA's CSP Engagements team. As a technical focal point, this engineer will work directly with key CSP/hyperscale customers to ensure seamless integration and operation of NVIDIA GPU firmware. This involves driving work streams with engineering teams, gathering and synthesizing feedback, and championing priorities into NVIDIA's feature roadmap. The role requires a deep understanding of GPU architecture internals, including streaming multiprocessors, GEMM execution, and memory hierarchy, as well as experience with firmware update lifecycle management at scale.

The successful candidate will serve as a bridge between NVIDIA and CSP firmware/software engineering teams, ensuring that GPU behaviors are well-documented and accessible for customer integration. This position demands a customer-obsessed mindset, with a genuine passion for simplifying GPU firmware integration for fleet-scale customers. The engineer will identify patterns in GPU firmware operational challenges and drive systemic improvements that can benefit multiple customers.

NVIDIA's CSP Engagements team is dedicated to supporting the company's hyperscale customers, providing them with the tools and expertise needed to deploy NVIDIA GPUs at scale. As a Principal Software Engineer, you will be an integral part of this team, working closely with customers to understand their needs and developing solutions that meet those needs.

What You Will Do

Drive GPU firmware and system software work streams with CSP engineering teams to ensure they understand GPU firmware architecture, update sequencing, recovery procedures, and GPU power management.
Gather and synthesize CSP feedback on GPU firmware and software, covering manageability, observability, security requirements, and performance, and champion those priorities into NVIDIA's feature roadmap and delivery plan.
Drive GPU firmware update orchestration for large-scale deployments, including multi-GPU update sequencing, rollback strategy, failure handling, and validation across hundreds of GPUs per rack.
Serve as the technical focal point between NVIDIA and CSP firmware/software engineering, ensuring GPU behaviors are well-documented and accessible for customer integration.
Identify cross-CSP GPU software and firmware issue patterns, such as common update failures, recovery gaps, and configuration problems, and drive documentation, tooling, and test strategy improvements.
Collaborate with internal stakeholders to ensure alignment on GPU firmware and system software development, including working with the product management team to define product requirements and with the engineering team to implement solutions.
Develop and maintain relationships with key CSP customers, providing them with technical guidance and support on GPU firmware and system software integration.
Participate in the development of NVIDIA's GPU firmware and system software roadmap, ensuring that customer needs are represented and that solutions are developed to meet those needs.
Work with the NVIDIA engineering team to resolve complex technical issues related to GPU firmware and system software, providing expertise and guidance as needed.
Develop and deliver training and documentation to customers on GPU firmware and system software, including best practices for integration and troubleshooting.

What We Are Looking For

15+ years of experience in GPU system software, GPU firmware, or accelerator platform engineering.
BS or MS in Computer Science, Electrical Engineering, or related field (or equivalent experience).
Deep understanding of GPU architecture internals, including streaming multiprocessors, GEMM execution, compute kernels, memory hierarchy, and how firmware/driver decisions impact GPU compute performance.
Understanding of multi-GPU fabric architectures (NVLink, or similar) and how firmware coordinates across multiple GPUs in a rack-scale system.
Experience with firmware update lifecycle management at scale, including multi-device update sequencing, A/B updates, rollback, staged rollout, and emergency recovery.
Understanding of GPU error handling and recovery flows, including how firmware-level errors propagate through the driver stack to application-visible failures.
Experience with GPU health monitoring and telemetry, including Xid errors, thermal events, power events, ECC counters, and their significance for firmware/software teams.
Customer-obsessed mindset, with a genuine passion for simplifying GPU firmware integration for fleet-scale customers.
Proven success influencing engineering teams to improve quality and fleet manageability.

Nice to Have

Direct experience with NVIDIA GPU VBIOS, GPU microcontroller firmware, or GPU driver internals.
Background in GPU fleet management at 10K+ GPU scale, including firmware rollout, health-based remediation, and fleet-wide configuration management.
Experience with GPU error taxonomy (Xid classification, NVLink error counters, ECC events) and building runbooks around GPU firmware behavior.
Understanding of GPU security, including secure boot chain, code signing, attestation, debug authentication, and multi-tenancy isolation at the firmware level.
Familiarity with GPU power management architecture and its impact on workload performance at fleet scale.

Benefits and Perks

Competitive salary and benefits package.
Opportunity to work with a leader in the tech industry, driving innovation in GPU technology.
Collaborative and dynamic work environment, with a team of experienced engineers and industry experts.
Professional development opportunities, including training and education programs.
Flexible work arrangements, including remote work options and flexible hours.
Access to the latest technologies and tools, including NVIDIA GPUs and software development kits.
Recognition and reward programs, including bonuses and stock options.
Comprehensive health and wellness programs, including medical, dental, and vision coverage.
Retirement savings plans, including 401(k) matching and pension plans.

How to Stand Out

To stand out in this role, highlight your experience with GPU firmware and system software, as well as your ability to work with large-scale deployments.
Showcase your understanding of GPU architecture internals, including streaming multiprocessors and memory hierarchy.
Emphasize your customer-obsessed mindset and your ability to simplify complex technologies for fleet-scale customers.
Prepare to discuss your experience with firmware update lifecycle management at scale, including multi-device update sequencing and rollback strategies.
Be ready to provide examples of your success in influencing engineering teams to improve quality and fleet manageability.
Review your knowledge of GPU error handling and recovery flows, as well as GPU health monitoring and telemetry.
Consider developing a portfolio that showcases your experience with GPU firmware and system software development, including any relevant projects or contributions to open-source repositories.

This is a remote position listed on WFA Digital, the platform for professionals who work from anywhere. Browse more remote jobs across all categories.