15 nov
|
Pyramid Consulting
|
Aguascalientes
15 nov
Pyramid Consulting
Aguascalientes
We're Hiring: Lead Data Engineer – Databricks (Data Ingestion & Lakeflow Connect)
Role: Lead Data Engineer
Specialization: Databricks, Lakeflow Connect, Data Engineering & Cloud Data Platforms
Job Description
We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, enterprise data ingestion, and large-scale data engineering to lead the design and implementation of scalable, high-performance data pipelines.
The adecuado candidate should have hands‑on experience with Databricks Native Data Loader, Lakeflow Connect, Delta Lake, Apache Spark , and enterprise data integration from relational databases such as Oracle, MySQL, and SQL Server.
The role involves driving data engineering architecture, optimizing ETL/ELT workflows, ensuring data quality and platform reliability, and providing technical leadership to engineering teams.
Key Responsibilities
- Design and implement scalable data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks‑native capabilities.
- Develop and optimize batch and streaming data pipelines integrating Oracle, MySQL, SQL Server, APIs, and other enterprise data sources.
- Build robust ETL/ELT pipelines using Databricks, Apache Spark, Delta Lake, and cloud-native technologies.
- Architect end‑to‑end data ingestion, transformation, storage, validation, and monitoring solutions.
- Design and maintain scalable data pipeline architectures to transform raw and unstructured data into analytics‑ready datasets.
- Implement data quality checks, validation frameworks, pipeline monitoring,
and error‑handling mechanisms.
- Perform performance tuning, root cause analysis, and optimization of data pipelines and distributed processing workloads.
- Drive automation, infrastructure optimization, scalability, and continuous improvement initiatives.
- Implement unit testing, integration testing, and Test‑Driven Development (TDD) practices.
- Ensure data governance, security, reliability, and compliance with enterprise engineering standards.
- Provide technical leadership, mentorship, and guidance to junior data engineers.
- Collaborate with Product, Architecture, Analytics, Engineering, and Business teams to translate requirements into technical solutions.
- Drive engineering best practices across coding standards, CI/CD, DevOps, documentation, and code reviews.
- Participate in architecture discussions, technical design reviews, Agile ceremonies, sprint planning, and estimation.
Required Technical Skills
- Databases: Oracle, MySQL, Microsoft SQL Server.
- Data Processing: Batch Processing, Streaming, Data Pipelines.
Experience & Qualifications
- 8+ years of experience in Data Engineering, Data Integration, or Big Data technologies.
- Strong hands‑on experience with Databricks and enterprise‑scale data ingestion solutions.
- Proven experience designing and implementing scalable data pipelines and ETL/ELT frameworks.
- Experience leading technical teams, mentoring engineers, and driving architecture decisions.
- Strong communication, problem‑solving, and stakeholder collaboration skills.
#J-18808-Ljbffr
📌 Data Engineer (Aguascalientes)
🏢 Pyramid Consulting
📍 Aguascalientes