We are seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture . This role is focused on migrating legacy data solutions into scalable, governed, and performance-optimized pipelines using Databricks, PySpark, and Delta Lake.

The engineer will play a key role in defining and building Bronze, Silver, and Gold data layers , enabling both batch and near real-time processing, and improving the reliability and usability of enterprise data platforms. This position will work closely with data engineers, architects, and stakeholders while providing technical leadership and guidance to more junior team members.

Key Responsibilities
Architecture & Engineering

Architect and implement end-to-end data solutions within the Databricks Lakehouse environment

Design and build ingestion, transformation, and serving pipelines using PySpar k, Spark SQ L, and Delta Lake

Establish and enforce medallion architectur e practices across Bronze, Silver, and Gold layer

Migration & Modernizatio

nLead the migration of legacy Azure SQ L, SSI S, and AD F pipelines into Databricks-native workflows

Convert existing T-SQ L and ETL logic into scalable Spark-based processing

Support modernisation of data platforms with a focus on governance, scalability, and maintainability

Streaming, Governance & Optimization

Implement Auto Loade r, Delta Lake Change Data Feed (CDF ), and streaming/batch ingestion patterns

Define and enforce governance using Unity Catalo g, including RBAC and data lineage





Optimise performance through Photo n, partitionin g, Z-Order indexin g, and cluster tuning

Validation & Technical Leadership

Guide validation and reconciliation strategies between legacy SQL outputs and Delta-based pipelines

Perform code reviews and promote engineering best practices

Mentor mid-level and junior engineers across design, development, and troubleshooting activities

Required Experience & Qualifications

Bachelor’s degree in Computer Science, Engineering, Information Systems, or equivalent experience

6+ years of experien ce in data engineering, wit h 3+ years of hands-on Databricks experience

Strong expertise i n PySpa rk , Spark S QL, an d Delta Lake

Proven experience migrating legacy data platforms such a s SQL Serv er , SS IS, an d ADF

Strong understanding o f ETL/ELT desi gn , data modeli ng, and pipeline architecture

Hands-on experience with th e Azure ecosyst em, includin g AD LS , A DF, and related services

Strong knowledge of performance tuning techniques for distributed data processing

Experience leading technical implementation efforts and supporting junior engineers

Preferred Qualifications

Experience wi th Unity Cata lo g, Lakehouse Federat ion, or CDC frameworks

Databricks certifications (Associate or Professional level)

Exposure to large-scale regulated datasets such as healthc ar e, cla ims, or financial data

Experience with bo th ba tch a nd stream ing data pipelines

Familiarity with enterprise-grade data governance and lineage practices

#J-18808-Ljbffr

📌 Databricks Engineer (Ciudad de México)
🏢 Apex Systems
📍 Ciudad de México