Data Engineering Pipeline Engineer – Role Description

Data Engineering Pipeline Engineer – Role

ABOUT THE ROLE

We are looking for a skilled Data Engineering Pipeline Engineer to design, build, and maintain scalable data infrastructure that powers our analytics, machine learning, and business intelligence platforms. You will work across the full data lifecycle — from ingestion and transformation to storage and delivery — ensuring reliability, performance, and governance at every stage. This is a high-impact role where you will collaborate closely with data scientists, analysts, and platform engineers to ship robust pipelines that enable data-driven decisions across the organization.

KEY RESPONSIBILITIES

• Design, build, and maintain scalable batch and real-time data pipelines using tools such as Apache Spark, Kafka, Flink, or Airflow

• Develop and optimize ETL/ELT workflows to ingest data from diverse sources including APIs, databases, event streams, and flat files

• Architect and manage cloud-based data infrastructure on AWS, GCP, or Azure (e.g., S3, BigQuery, Redshift, Databricks, Snowflake)

• Implement data quality monitoring, alerting, and observability frameworks to ensure pipeline reliability and SLA compliance

• Collaborate with data scientists and ML engineers to support model training, feature engineering, and inference pipelines

• Partner with analytics engineers to maintain and evolve data warehouse models (dbt, dimensional modeling)

• Define and enforce data governance standards including cataloging, lineage tracking, and access control policies

• Optimize pipeline performance through profiling, query tuning, partitioning strategies, and cost management

• Document pipeline architecture, data contracts, and runbooks for operational clarity

• Participate in on-call rotation and incident response for critical data infrastructure

REQUIRED QUALIFICATIONS

• 4+ years of experience in data engineering, ETL development, or a related software engineering discipline

• Proficiency in Python and/or Scala for pipeline development; strong SQL skills across multiple dialects

• Hands-on experience with distributed processing frameworks such as Apache Spark, Beam, or Flink

• Experience with workflow orchestration tools such as Apache Airflow, Prefect, or Dagster

• Deep familiarity with cloud data platforms (AWS, GCP, or Azure) and managed services such as BigQuery, Redshift, or Synapse

• Experience designing and maintaining data warehouses or lakehouses (Snowflake, Databricks, Delta Lake, Iceberg)

• Strong understanding of data modeling concepts: normalization, star/snowflake schema, slowly changing dimensions

• Experience with streaming and event-driven architectures using Kafka, Kinesis, or Pub/Sub • Familiarity with CI/CD practices and infrastructure-as-code tools (Terraform, Pulumi) for data platform deployments

• Excellent communication skills with the ability to translate business requirements into technical solutions