Training Performance Engineer

About the Role

The team's focus on high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement, along with performant, high-uptime, fault-tolerant training frameworks, positions the Training Performance Engineer at the heart of Openai's efforts to accelerate researcher throughput and model training speeds. The role demands a blend of technical expertise, analytical skills, and collaboration, making it both challenging and rewarding for the right candidate.

Openai's commitment to pushing the boundaries of AI capabilities means that the Training Performance Engineer will be working at the forefront of technological innovation. The company's dedication to safety and human needs at the core of AI development ensures that the work done here contributes to a broader mission of benefiting humanity through general-purpose artificial intelligence.

What You Will Do

• Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
• Optimize GPU utilization and throughput for large-scale distributed model training.
• Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
• Implement model graph transforms to improve end-to-end throughput.
• Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
• Partner with researchers to ensure new model architectures scale efficiently during pre-training.
• Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.
• Analyze and optimize the performance of distributed training frameworks, including training loop, state management, resilient checkpointing, deterministic orchestration, and observability.
• Work on distributed process management for long-lived, job-specific, and user-provided processes.
• Investigate and resolve issues related to I/O bottlenecks, collective communication, and other performance limitations.

What We Are Looking For

• Strong programming skills in Python and C++ (Rust or CUDA is a plus).
• Experience running distributed training jobs on multi-GPU systems or HPC clusters.
• Ability to debug complex distributed systems and measure efficiency rigorously.
• Exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
• Strong analytical and problem-solving skills, with the ability to identify performance bottlenecks and design optimizations.
• Experience with performance optimization techniques, including profiling, benchmarking, and performance modeling.
• Ability to collaborate across teams, including researchers, runtime engineers, and systems engineers.
• Strong understanding of distributed systems, including distributed computing, networking, and storage.
• Experience with containerization (e.g., Docker) and orchestration tools (e.g., Kubernetes).

Nice to Have

• Familiarity with NCCL, MPI, or UCX communication libraries.
• Experience with large-scale data loading and checkpointing systems.
• Prior work on training runtime, distributed scheduling, or ML compiler optimization.
• Knowledge of machine learning principles and deep learning architectures.
• Experience with cloud computing platforms (AWS, GCP, Azure) and their services related to machine learning and distributed computing.

Benefits and Perks

• Competitive salary and equity package.
• Opportunity to work on cutting-edge AI research and deployment projects.
• Collaborative and dynamic work environment with a team of experts in AI and machine learning.
• Flexible work arrangements, including remote work options and a hybrid work model.
• Access to the latest technologies and tools in AI and machine learning.
• Professional development opportunities, including training, mentorship, and conference attendance.
• Comprehensive health, dental, and vision insurance.
• Generous parental leave policy and family support benefits.

• Develop a strong foundation in distributed systems, performance optimization, and machine learning frameworks like PyTorch or TensorFlow.
• Highlight any experience with GPU programming, collective communication, and I/O optimization in your resume and cover letter.
• Prepare to talk about your approach to debugging complex distributed systems and how you measure efficiency.
• Showcase projects or contributions to open-source repositories that demonstrate your skills in distributed training and performance optimization.
• Be ready to discuss how you stay current with the latest developments in AI, machine learning, and distributed computing.
• Consider creating a portfolio that includes examples of your work on optimizing distributed systems or improving the performance of machine learning models.
• When negotiating salary, be prepared to discuss your experience, skills, and the market rate for similar positions in the industry.