Lead Data Engineer - Databricks & Cloud Data Platforms

Overview

We are seeking a Lead Data Engineer to serve as the technical lead on complex, enterprise-scale data engineering initiatives. This is a hands-on role for someone who enjoys solving challenging data problems, partnering with client stakeholders, and building scalable cloud-native data platforms from the ground up.

The ideal candidate brings deep expertise in Databricks, modern data architectures, and cloud technologies, along with the ability to lead technical delivery while mentoring engineering teams.

What You'll Bring

• 8+ years of hands-on experience designing, building, and delivering enterprise-scale data platforms.
• Expert-level knowledge of Apache Spark, including performance tuning, optimization, and distributed processing.
• Strong Python development skills with experience writing clean, production-grade code; Scala experience is a plus.
• Extensive experience building and deploying solutions on Databricks, including Delta Lake, Unity Catalog, and Databricks Workflows. Databricks certifications are highly desirable.
• Experience working across at least two major cloud platforms (AWS, Azure, or GCP), with deep expertise in one or more cloud-native data services.
• Proven success leading end-to-end data engineering initiatives within enterprise or consulting environments.
• Strong understanding of DataOps best practices, including CI/CD, infrastructure as code, automated testing, monitoring, and observability.
• Experience with ETL/ELT development, data warehousing, Lakehouse architecture, and modern data modeling techniques.
• Excellent communication skills with the ability to collaborate effectively with both technical and business stakeholders.

Responsibilities

• Lead the technical delivery of enterprise data engineering initiatives from architecture through production deployment.
• Design, build, and optimize scalable data pipelines and Lakehouse architectures using Databricks and modern cloud technologies.
• Translate business requirements into scalable technical solutions, reference architectures, and production-ready data models.
• Drive project execution by managing technical scope, mitigating delivery risks, and ensuring successful implementation.
• Provide technical leadership through code reviews, mentoring, and coaching of engineering team members.
• Establish engineering best practices around DataOps, including automated testing, CI/CD, version control, monitoring, and documentation.
• Develop reusable technical assets, architecture patterns, and internal best practices to improve delivery consistency across projects.

Eames Consulting is acting as an Employment Business in relation to this vacancy.