Data Engineer

- We are seeking a skilled Databricks Engineer with strong experience in Azure Databricks and Python (PySpark) to design, build, and optimize scalable data processing pipelines.

- The adecuado candidate will work closely with data engineers, analysts, and stakeholders to deliver high-performance data solutions on the Azure cloud platform.

Key Responsibilities

- Design, build, and maintain data pipelines and ETL processes using Databricks and Apache Spark.

- Optimize data workflows for performance, scalability, and cost efficiency.

- Implement data Lakehouse architecture and manage data ingestion from multiple sources.

- Collaborate with data scientists and analysts to enable advanced analytics and machine learning workloads.

- Ensure data quality, governance, and security across all data assets.

- Monitor and troubleshoot Databricks clusters, jobs, and workflows.

- Integrate Databricks with cloud services (AWS, Azure, or GCP) and other enterprise systems.

- Document processes, standards, and best practices for data engineering.





Required Skills & Qualifications

- Bachelor’s degree in Computer Science, Data Engineering, or related field.

- 3+ years of experience in data engineering or big data technologies.

- Hands-on experience with Databricks, Apache Spark, and PySpark.

- Strong knowledge of SQL, Python, and data modeling principles.

- Experience with cloud platforms (AWS, Azure, or GCP) and their data services.

- Familiarity with Delta Lake, Lakehouse architecture, and data governance.

- Understanding of CI/CD pipelines and DevOps practices for data workflows.

- Excellent problem-solving and communication skills.

Nice to Have

- Experience with Delta Lake and Lakehouse architecture.

- Knowledge of Azure Data Factory (ADF) for orchestration.

- Exposure to CI/CD pipelines for data engineering.

- Experience with SQL and Spark SQL.

- Azure certifications related to data engineering.

📌 Data Engineer (México)
🏢 Pyramid Consulting
📍 México