Big Data Lead

Job ID:

AR 678967

Salary:

$85K Gross

Work Arrangement:

Remote

Job Summary

We are looking for an experienced

Big Data Lead to design, develop, and support scalable data engineering solutions in a modern cloud-based environment. The idóneo candidate will have strong experience with data integration, ETL/ELT, distributed data processing, cloud platforms, SQL, and programming

, along with a solid understanding of data lakes, data warehouses, and resilient data pipelines.

Key Responsibilities

- Design, develop, and support scalable data integration, ETL/ELT, and data engineering solutions

.

- Process and manage large volumes of data across distributed and cloud-based environments

.

- Develop efficient data pipelines using

SQL and programming languages such as Python, Java, Scala, or Spark SQL

.

- Build and manage data workflows using orchestration tools such as

Apache Airflow, Azure Data Factory, or AWS Step Functions

.

- Design resilient and fault-tolerant pipelines with logging, monitoring, retry mechanisms, idempotency, and failure recovery

.

- Work with relational and non-relational databases, data warehouses, and data lake architectures

.

- Troubleshoot data pipeline and integration issues and implement effective solutions.

- Collaborate with engineering, application, and business teams to deliver reliable data solutions.

- Contribute to data architecture, modeling, governance, metadata management, and master data management initiatives.

- Support containerized deployments,

CI/CD pipelines,



and Infrastructure as Code practices.

Required Qualifications

- Bachelor's degree in

Computer Science, Engineering, Information Systems, or a related field

; equivalent practical experience may be considered.

- 5+ years of experience developing and supporting data integration, ETL/ELT, or data engineering solutions.

- Proven experience working with large data volumes in distributed or cloud-based environments

.

- Strong proficiency in

SQL

.

- Strong programming experience with at least one of the following:

- Python

- Java

- Scala

- Spark SQL

- Experience with workflow and orchestration technologies such as:

- Apache Airflow

- Azure Data Factory

- AWS Step Functions

- Working knowledge of relational and non-relational databases

.

- Experience with data warehouses and data lake architectures

.

- Experience designing reliable production data pipelines with monitoring, retry logic, logging, and failure recovery.

- Strong analytical, troubleshooting, communication, and collaboration skills

.

Preferred Qualifications

- Hands-on experience with

AWS cloud data platforms

.

- Experience with

AWS Lambda and Apache Kafka

.

- Knowledge of data modeling, metadata management, data governance, and Master Data Management (MDM)

.

- Experience with

Docker/containerization and deployment technologies.

- Experience with

CI/CD pipelines and Infrastructure as Code (IaC).

- Experience working in financial services, healthcare, regulated industries, or other highly controlled environments

.

📌 Big Data Specialist (México)
🏢 Pyramid Consulting
📍 México