Job Description – GCP Data Engineer

Role Overview

We are looking for an experienced GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT frameworks on the Google Cloud Platform. The candidate will work extensively with BigQuery, Python/PySpark, Airflow/Cloud Composer, and Google Cloud Storage to process and transform large volumes of campaign, customer, and clickstream data.

The ideal candidate should have strong hands-on experience in data engineering, excellent SQL and Python skills, and a good understanding of cloud-based data pipeline architecture. The role will involve building reliable production pipelines, optimizing BigQuery performance and costs, implementing automation, and supporting critical data workflows.

Key Responsibilities

Data Pipeline Development

• Design, develop, and maintain scalable and reliable data pipelines on GCP.
• Build batch and, where required, near-real-time data ingestion and transformation pipelines.
• Develop robust ETL/ELT frameworks for large volumes of structured and semi-structured data.
• Ingest data from multiple sources into Google Cloud Storage and BigQuery.
• Develop reusable and modular data engineering components.
• Implement appropriate error handling, retry mechanisms, logging, and data validation.

BigQuery & Data Engineering

• Develop complex and optimized SQL queries in BigQuery for data transformation and analysis.
• Optimize BigQuery performance through:
  • Partitioning
  • Clustering
  • Query optimization
  • Efficient table design
  • Appropriate data types and storage strategies
• Monitor and optimize BigQuery processing costs.
• Design scalable data models suitable for large-scale campaign, customer, and clickstream datasets.
• Troubleshoot data quality, performance, and pipeline-related issues.

Airflow / Cloud Composer

• Develop and maintain DAGs using Apache Airflow / Cloud Composer.
• Implement scheduling, dependency management, retries, failure handling, and alerting.
• Monitor production workflows and proactively resolve failed or delayed jobs.
• Build reusable operators and workflow components where required.
• Ensure critical data pipelines meet agreed SLAs.

Python / PySpark

• Develop data processing and transformation logic using Python and/or PySpark.
• Write clean, reusable, scalable, and production-ready code.
• Optimize PySpark jobs for performance and efficient resource utilization.
• Implement appropriate testing and validation for data transformation processes.

GCP Services

Work extensively with GCP services, particularly:

• Google BigQuery
• Google Cloud Storage (GCS)
• Cloud Composer / Airflow

Exposure to additional GCP services such as Cloud Functions, Pub/Sub, Dataflow, Cloud Run, Secret Manager, IAM, or Cloud Monitoring would be an advantage.

Automation, CI/CD & DevOps

• Automate deployment and execution of data engineering workflows.
• Work with Git-based development and version-control workflows.
• Implement and maintain CI/CD pipelines using tools such as Jenkins or equivalent.
• Follow code review, branching, deployment, and release-management processes.
• Implement monitoring, logging, and alerting for production data pipelines.

Production Support & Troubleshooting

• Monitor daily data pipeline execution and resolve production issues within defined SLAs.
• Investigate pipeline failures, data discrepancies, performance issues, and processing delays.
• Perform root-cause analysis and implement permanent fixes.
• Coordinate with application, analytics, infrastructure, and business teams to resolve data-related issues.
• Participate in production deployments and provide post-deployment support.

Stakeholder Collaboration

• Work closely with Data Analysts, Data Scientists, Product Teams, Business Stakeholders, and Technology Teams to understand data requirements.
• Translate business requirements into scalable technical solutions.
• Communicate technical issues, risks, dependencies, and delivery status effectively.
• Participate in technical discussions, design reviews, and solution development.

Required Technical Skills

Must Have

• 4–8 years of experience in Data Engineering.
• Strong hands-on experience with GCP.
• Strong proficiency in SQL, preferably extensive experience with BigQuery SQL.
• Strong hands-on experience in Python and/or PySpark.
• Experience developing ETL/ELT data pipelines.
• Hands-on experience with BigQuery.
• Hands-on experience with Google Cloud Storage (GCS).
• Experience with Apache Airflow / Cloud Composer.
• Good understanding of data pipeline architecture and data engineering best practices.
• Experience with Git and CI/CD practices.
• Exposure to Jenkins or similar CI/CD tools.
• Experience in production support, monitoring, troubleshooting, and performance optimization.

Good to Have

• Experience working with campaign, marketing, customer, or clickstream data.
• Experience with large-scale data processing.
• Knowledge of Dataflow / Apache Beam.
• Knowledge of Pub/Sub and event-driven architectures.
• Experience with real-time or streaming data pipelines.
• Knowledge of data warehousing and dimensional data modeling.
• Experience with data quality frameworks and validation.
• Knowledge of GCP IAM and security concepts.
• Experience with Cloud Monitoring / Logging.
• Experience working in Agile/Scrum environments.

Candidate Profile

The ideal candidate should:

• Have strong hands-on technical expertise rather than only theoretical knowledge.
• Be comfortable writing complex SQL and Python/PySpark code.
• Have experience independently designing and developing data pipelines.
• Understand how to build scalable and cost-efficient solutions on GCP.
• Be capable of troubleshooting production issues and taking ownership until resolution.
• Have good analytical and problem-solving skills.
• Be comfortable working with multiple stakeholders and managing delivery timelines.
• Demonstrate good communication and documentation skills.