Big Data Lead

## About the Role The Big Data Lead position at DemandMatrix is a critical role that will drive the company's ability to leverage big data technology to solve complex data issues. This is a hands-on position that requires a strong technical background and the ability to work closely with cross-functional teams. The successful candidate will be responsible for designing, building, and maintaining large-scale data pipelines and architectures that can handle high volumes of data.

Day-to-day, the Big Data Lead will spend a significant amount of time coding, working with cloud-native services in AWS and GCP, and optimizing data processing systems for performance and efficiency. This role is perfect for someone who is genuinely excited about technology and has a proven track record of delivering complex data projects from scratch.

The Big Data Lead will be part of a dynamic team that is passionate about using data to drive business decisions. The team is committed to innovation and is always looking for ways to improve processes and stay ahead of the curve. This is an excellent opportunity for someone who is looking to join a company that values its employees and offers a collaborative and supportive work environment.

## What You Will Do - Design and build large-scale data pipelines and architectures that can handle high volumes of data - Develop and optimize data processing systems using cloud-native services in AWS and GCP - Work closely with cross-functional teams to identify and prioritize data projects - Develop and maintain data models, sharding, and data location strategies for distributed data stores - Optimize data pipelines for performance and efficiency, considering CPU core to network IO to disk IO - Collaborate with the development team to design and implement data-driven solutions - Troubleshoot and resolve data processing issues, ensuring high availability and reliability - Stay up-to-date with the latest big data technologies and trends, applying this knowledge to improve existing systems - Mentor and guide junior team members, sharing knowledge and expertise to improve overall team performance

## What We Are Looking For - 7+ years of hands-on experience in software development with a focus on big data and large data pipelines - Minimum 3 years of experience building services and pipelines using Python - Expertise with a variety of data processing systems, including streaming, event, and batch (Spark, Hadoop/MapReduce) - Understanding of at least one NoSQL store like MongoDB, Elasticsearch, HBase - Experience with running scalable and high-available systems with AWS or GCP - Solid grounding in computer engineering, Unix, data structures, and algorithms - Proven track record of delivering complex data projects from scratch - Excellent problem-solving skills, with the ability to troubleshoot and resolve complex data processing issues

## Nice to Have - Experience with Docker/Kubernetes - Exposure to CI/CD pipelines - Knowledge of crawling/scraping - Experience with machine learning and AI technologies

## Benefits and Perks - Opportunity to work with a dynamic and innovative company that is disrupting the sales and marketing intelligence industry - Collaborative and supportive work environment - Flexible working hours and remote work arrangements - Professional development opportunities, including training and conferences - Access to the latest big data technologies and tools - Competitive compensation and benefits package - Birthday leave and additional time off for relaxation and rejuvenation

- To stand out as a candidate, make sure to highlight your hands-on experience with big data technologies and cloud-native services in your resume and cover letter.

• Showcase your ability to work with large volumes of data and optimize data processing systems for performance and efficiency.
• Be prepared to answer technical questions during the interview process, such as how you would design a data pipeline or optimize a data processing system.
• Emphasize your problem-solving skills and ability to troubleshoot complex data processing issues.
• Don't be afraid to ask questions during the interview process, such as what the team is like or what opportunities there are for professional development.
• Make sure to research the company and its products before the interview, and be prepared to discuss how your skills and experience align with the company's goals and mission.