ob Description

We are looking for an experienced Lead Data Engineer with strong hands-on expertise in Databricks data ingestion, particularly Databricks Native Data Loader and Lakeflow Connect. The adecuado candidate will be responsible for designing, developing, and leading scalable data ingestion and transformation solutions, bringing data from enterprise databases and external sources into the Databricks Lakehouse platform.

Key Responsibilities

- Design and implement scalable data ingestion pipelines using Databricks-native ingestion capabilities, including Lakeflow Connect.
- Build data pipelines to ingest data from enterprise databases such as Oracle, MySQL, and SQL Server, as well as APIs and other data sources.
- Develop and optimize batch and streaming data pipelines using Apache Spark and PySpark.
- Implement ETL/ELT processes, incremental data loading, and Change Data Capture (CDC) mechanisms.
- Work with Delta Lake to manage, transform, and optimize large-scale datasets.
- Implement data quality checks, validation, monitoring, and error-handling mechanisms.
- Ensure pipeline reliability, performance, scalability, and data governance.
- Collaborate with architects, business stakeholders, and engineering teams to define data ingestion strategies and technical solutions.
- Lead and mentor data engineers, review code,



and establish engineering best practices.
- Support CI/CD implementation and automated testing for data engineering workflows.

Required Skills & Experience
- 8 years of experience in Data Engineering, including experience in a technical leadership capacity.
- Strong hands-on experience with Databricks, particularly data ingestion capabilities.
- Practical experience with Databricks Lakeflow Connect and/or Databricks Native Data Loader.
- Strong programming experience in Python, PySpark, and SQL.
- Hands-on experience with Apache Spark and Delta Lake.
- Experience ingesting data from Oracle, MySQL, SQL Server, APIs, and other enterprise data sources.
- Strong understanding of ETL/ELT, CDC, incremental loading, batch processing, and streaming.
- Experience designing scalable, reliable, and high-performance data pipelines.
- Knowledge of data quality, pipeline monitoring, testing, and CI/CD practices.
- Excellent communication, problem-solving, and technical leadership skills.

Preferred Qualifications
- Experience with Databricks Auto Loader and Lakehouse architecture.
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Familiarity with data governance, security, and access management.
- Experience working in Agile development environments.

📌 Senior Data Engineer (México)
🏢 Pyramid Consulting
📍 México