We're Hiring: Lead Data Engineer – Databricks (Data Ingestion & Lakeflow Connect)

Role: Lead Data Engineer

Location: Aguascalientes, Mexico (Hybrid)

Specialization: Databricks, Lakeflow Connect, Data Engineering & Cloud Data Platforms

Job Description

We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, enterprise data ingestion, and large-scale data engineering to lead the design and implementation of scalable, high-performance data pipelines.

The adecuado candidate should have hands-on experience with Databricks Native Data Loader, Lakeflow Connect, Delta Lake, Apache Spark, and enterprise data integration from relational databases such as Oracle, MySQL, and SQL Server.

The role involves driving data engineering architecture, optimizing ETL/ELT workflows, ensuring data quality and platform reliability, and providing technical leadership to engineering teams.

Key Responsibilities

- Design and implement scalable data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities.
- Develop and optimize batch and streaming data pipelines integrating Oracle, MySQL, SQL Server, APIs, and other enterprise data sources.
- Build robust ETL/ELT pipelines using Databricks, Apache Spark, Delta Lake, and cloud-native technologies.
- Architect end-to-end data ingestion, transformation, storage, validation, and monitoring solutions.
- Design and maintain scalable data pipeline architectures to transform raw and unstructured data into analytics-ready datasets.
- Implement data quality checks, validation frameworks, pipeline monitoring, and error-handling mechanisms.
- Perform performance tuning, root cause analysis, and optimization of data pipelines and distributed processing workloads.
- Drive automation, infrastructure optimization, scalability,



and continuous improvement initiatives.
- Implement unit testing, integration testing, and Test-Driven Development (TDD) practices.
- Ensure data governance, security, reliability, and compliance with enterprise engineering standards.
- Provide technical leadership, mentorship, and guidance to junior data engineers.
- Collaborate with Product, Architecture, Analytics, Engineering, and Business teams to translate requirements into technical solutions.
- Drive engineering best practices across coding standards, CI/CD, DevOps, documentation, and code reviews.
- Participate in architecture discussions, technical design reviews, Agile ceremonies, sprint planning, and estimation.

Required Technical Skills

- Databricks: Databricks Native Data Loader, Lakeflow Connect, Delta Lake.
- Data Engineering: ETL/ELT, Data Ingestion, Data Integration, Data Transformation.
- Big Data Technologies: Apache Spark, PySpark, Distributed Data Processing.
- Databases: Oracle, MySQL, Microsoft SQL Server.
- Data Processing: Batch Processing, Streaming, Data Pipelines.
- Cloud & Architecture: Cloud Data Platforms, Scalable Data Architecture.
- Engineering Practices: CI/CD, DevOps, TDD, Unit Testing, Integration Testing.

Experience & Qualifications

- 8+ years of experience in Data Engineering, Data Integration, or Big Data technologies.
- Strong hands-on experience with Databricks and enterprise-scale data ingestion solutions.
- Proven experience designing and implementing scalable data pipelines and ETL/ELT frameworks.
- Experience leading technical teams, mentoring engineers, and driving architecture decisions.
- Strong communication, problem-solving, and stakeholder collaboration skills.

#Hiring #LeadDataEngineer #Databricks #LakeflowConnect #DataEngineering #DeltaLake #ApacheSpark #PySpark #DataIngestion #ETL #ELT #BigData #CloudDataEngineering #DataArchitecture

📌 Data Engineer (Aguascalientes)
🏢 Pyramid Consulting
📍 Aguascalientes