Big Data Lead

Job ID: AR 678967

Salary: $85K Gross

Work Arrangement: Remote

Job Summary

We are looking for an experienced Big Data Lead to design, develop, and support scalable data engineering solutions in a modern cloud-based environment. The adecuado candidate will have strong experience with data integration, ETL/ELT, distributed data processing, cloud platforms, SQL, and programming, along with a solid understanding of data lakes, data warehouses, and resilient data pipelines.

Key Responsibilities

- Design, develop, and support scalable data integration, ETL/ELT, and data engineering solutions.
- Process and manage large volumes of data across distributed and cloud-based environments.
- Develop efficient data pipelines using SQL and programming languages such as Python, Java, Scala, or Spark SQL.
- Build and manage data workflows using orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Step Functions.
- Design resilient and fault-tolerant pipelines with logging, monitoring, retry mechanisms, idempotency, and failure recovery.
- Work with relational and non-relational databases, data warehouses, and data lake architectures.
- Troubleshoot data pipeline and integration issues and implement effective solutions.
- Collaborate with engineering, application, and business teams to deliver reliable data solutions.
- Contribute to data architecture, modeling, governance, metadata management, and master data management initiatives.
- Support containerized deployments, CI/CD pipelines,



and Infrastructure as Code practices.

Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field; equivalent practical experience may be considered.
- 5+ years of experience developing and supporting data integration, ETL/ELT, or data engineering solutions.
- Proven experience working with large data volumes in distributed or cloud-based environments.
- Strong proficiency in SQL.
- Strong programming experience with at least one of the following:
- Python
- Java
- Scala
- Spark SQL
- Experience with workflow and orchestration technologies such as:
- Apache Airflow
- Azure Data Factory
- AWS Step Functions
- Working knowledge of relational and non-relational databases.
- Experience with data warehouses and data lake architectures.
- Experience designing reliable production data pipelines with monitoring, retry logic, logging, and failure recovery.
- Strong analytical, troubleshooting, communication, and collaboration skills.

Preferred Qualifications
- Hands-on experience with AWS cloud data platforms.
- Experience with AWS Lambda and Apache Kafka.
- Knowledge of data modeling, metadata management, data governance, and Master Data Management (MDM).
- Experience with Docker/containerization and deployment technologies.
- Experience with CI/CD pipelines and Infrastructure as Code (IaC).
- Experience working in financial services, healthcare, regulated industries, or other highly controlled environments.

📌 Big Data Specialist (México)
🏢 Pyramid Consulting
📍 México