16 nov
|
Pyramid Consulting
|
Aguascalientes
16 nov
Pyramid Consulting
Aguascalientes
We're Hiring: Lead Data Engineer – Databricks (Data Ingestion & Lakeflow Connect)
Role: Lead Data Engineer
Location: Aguascalientes, Mexico (Hybrid)
Specialization: Databricks, Lakeflow Connect, Data Engineering & Cloud Data Platforms
We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, enterprise data ingestion, and large-scale data engineering to lead the design and implementation of scalable, high-performance data pipelines.
The adecuado candidate should have hands-on experience with Databricks Native Data Loader, Lakeflow Connect, Delta Lake, Apache Spark , and enterprise data integration from relational databases such as Oracle, MySQL, and SQL Server.
The role involves driving data engineering architecture, optimizing ETL/ELT workflows, ensuring data quality and platform reliability, and providing technical leadership to engineering teams.
Key Responsibilities
- Design and implement scalable data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities.
- Develop and optimize batch and streaming data pipelines integrating Oracle, MySQL, SQL Server, APIs, and other enterprise data sources.
- Build robust ETL/ELT pipelines using Databricks, Apache Spark, Delta Lake, and cloud-native technologies.
- Architect end-to-end data ingestion, transformation, storage, validation, and monitoring solutions.
- Design and maintain scalable data pipeline architectures to transform raw and unstructured data into analytics-ready datasets.
- Implement data quality checks, validation frameworks, pipeline monitoring, and error-handling mechanisms.
- Perform performance tuning, root cause analysis, and optimization of data pipelines and distributed processing workloads.
- Drive automation, infrastructure optimization, scalability,
and continuous improvement initiatives.
- Implement unit testing, integration testing, and Test-Driven Development (TDD) practices.
- Ensure data governance, security, reliability, and compliance with enterprise engineering standards.
- Provide technical leadership, mentorship, and guidance to junior data engineers.
- Collaborate with Product, Architecture, Analytics, Engineering, and Business teams to translate requirements into technical solutions.
- Drive engineering best practices across coding standards, CI/CD, DevOps, documentation, and code reviews.
- Participate in architecture discussions, technical design reviews, Agile ceremonies, sprint planning, and estimation.
Required Technical Skills
- Databricks: Databricks Native Data Loader, Lakeflow Connect, Delta Lake.
- Data Engineering: ETL/ELT, Data Ingestion, Data Integration, Data Transformation.
- Big Data Technologies: Apache Spark, PySpark, Distributed Data Processing.
- Databases: Oracle, MySQL, Microsoft SQL Server.
- Data Processing: Batch Processing, Streaming, Data Pipelines.
- Cloud & Architecture: Cloud Data Platforms, Scalable Data Architecture.
- Engineering Practices: CI/CD, DevOps, TDD, Unit Testing, Integration Testing.
Experience & Qualifications
- 8+ years of experience in Data Engineering, Data Integration, or Big Data technologies.
- Strong hands-on experience with Databricks and enterprise-scale data ingestion solutions.
- Proven experience designing and implementing scalable data pipelines and ETL/ELT frameworks.
- Experience leading technical teams, mentoring engineers, and driving architecture decisions.
- Strong communication, problem-solving, and stakeholder collaboration skills.
#Hiring #LeadDataEngineer #Databricks #LakeflowConnect #DataEngineering #DeltaLake #ApacheSpark #PySpark #DataIngestion #ETL #ELT #BigData #CloudDataEngineering #DataArchitecture
📌 Data Engineer (Aguascalientes)
🏢 Pyramid Consulting
📍 Aguascalientes