Lead Data Engineer(Partner Company Role)
WFA Digital Insight
As demand for skilled data engineers continues to surge, with a 22% increase in remote data engineering roles in the past year, standing out in this market requires a unique blend of technical expertise and leadership acumen. This Lead Data Engineer role at Name presents a compelling opportunity for those with a passion for shaping data architectures and guiding teams towards excellence. With the fintech sector experiencing rapid growth, professionals with experience in payments domain and data quality engineering are particularly in high demand. Before applying, candidates should be aware that this role requires not only technical prowess but also the ability to define and enforce data contracts and drive requirements with backend engineering teams.
Job Description
About the Role
The Lead Data Engineer position at Name is a hands-on leadership role that offers the chance to build the foundation of the company's data engineering function from the ground up. This role entails designing and implementing the target-state data architecture, which involves Postgres CDC, Kafka, Snowflake, and client-facing data products. The successful candidate will have the opportunity to shape how data is collected, structured, and delivered, and will be responsible for growing a team around them as the function matures.The role is critical to the company's success, as it will enable the organization to make data-driven decisions and drive business growth. The Lead Data Engineer will be working closely with cross-functional teams, including backend engineering, to ensure seamless data integration and delivery.
What You Will Do
- Design the target-state data architecture, including Postgres CDC, Kafka, Snowflake, and client-facing data products
- Own tooling decisions across ingestion, orchestration, transformation, and quality layers
- Implement CDC-based ingestion from PostgreSQL services using Debezium or equivalent
- Build streaming and near-real-time pipelines with defined SLAs
- Build a data quality control layer, including checksums, reconciliation, schema validation, anomaly detection, and alerting
- Define quality checkpoints across the full pipeline, from source capture through Snowflake to client delivery
- Define and enforce data contracts with service-owning teams for core entities
- Build the external data delivery layer, including financial settlement, transaction status, processor reconciliation, and client analytics
- Design tenant separation and implement replay/reload mechanisms for failure recovery
- Start hands-on and gradually hire and grow a small data engineering team as the function matures
- Build a pragmatic roadmap with concrete deliverables at 3, 6, and 12 months
What We Are Looking For
- 5+ years of experience in data engineering with end-to-end ownership of production pipelines
- Hands-on experience with Snowflake, PostgreSQL CDC, and Kafka
- Solid AWS experience, including S3, RDS, Aurora, and cloud data infrastructure
- Data quality engineering mindset, with experience in monitoring, reconciliation, and lineage
- Comfortable defining data contracts and driving requirements with backend engineering teams
- Technical leadership experience, including project ownership, cross-team alignment, and delivery under constraints
- Experience with Kubernetes, Airflow/Prefect/Dagster, and dbt is a plus
- Payments domain knowledge, including settlement, transaction lifecycle, and processor integrations, is a strong plus
- Familiarity with gRPC, RabbitMQ, and reading Go/Python service code is expected
Nice to Have
- Experience with cloud-based data warehousing and ETL tools
- Knowledge of data governance and data security best practices
- Certification in data engineering or a related field
Benefits and Perks
- Competitive compensation package, with regular performance reviews
- Opportunity to work with a young and ambitious company, with huge growth prospects
- Collaborative and dynamic work environment, with passionate and motivated colleagues
- Flexible remote work arrangements, with 20 days of vacation time, bank holidays, sick leaves, and an additional birthday day off
- Professional development opportunities, including training and conference sponsorships
- Access to the latest tools and technologies, including cloud-based data platforms and engineering software
How to Stand Out
- Ensure you have a strong understanding of data engineering principles and experience with cloud-based data platforms, such as Snowflake and AWS.
- Develop a portfolio that showcases your expertise in designing and implementing data architectures, including data quality and data delivery.
- Prepare to discuss your experience with data contracts and how you have driven requirements with backend engineering teams in the past.
- Be ready to walk through your experience with Kubernetes, Airflow/Prefect/Dagster, and dbt, and how you have applied these tools in previous roles.
- Research the company's technology stack and be prepared to discuss how you can contribute to the development of their data engineering function.
- Highlight your ability to work independently and as part of a team, and your experience with remote collaboration tools and technologies.
This is a remote position listed on WFA Digital, the platform for professionals who work from anywhere. Browse more remote jobs across all categories.