Assistant Manager - Data Engineering

Designation : Assistant Manager - Data Engineering

Level : L3

Location : Chennai

Experience : 5 to 8 Years

Job Role :

Build and maintain scalable data pipelines and data warehouse solutions using Python, PySpark, SQL, Databricks, and Airflow.
Drive data quality, workflow orchestration, performance optimization, and Linux-based data engineering, while collaborating with cross-functional teams to deliver reliable analytics solutions.

Responsibilities :

Data Pipeline Development:

·         Design, develop, and maintain scalable data pipelines using Python, PySpark, and SQL to ingest, process, and transform data from various sources.

·         Implement data quality checks and monitoring to ensure the reliability and integrity of data pipelines.

Data Warehouse Management:

·         Build and optimize data warehouse solutions for efficient storage and retrieval of large volumes of structured and unstructured data.

·         Collaborate with cross-functional teams to design and implement data models for analytical purposes.

Databricks Expertise:

·         Utilize Databricks platform for data engineering tasks, including data ingestion, ETL processing, and data analysis.

·         Optimize Databricks jobs and clusters for performance and cost efficiency.

Airflow Workflow Orchestration:

·         Configure and manage Airflow workflows to automate data pipeline orchestration and scheduling.

·         Implement error handling and monitoring within Airflow DAGs to ensure reliable execution of data workflows.

Linux Administration:

·         Manage Linux-based servers and environments for data engineering tasks, ensuring security, reliability, and performance.

·         Troubleshoot and resolve issues related to Linux infrastructure and server configurations.

Collaboration and Documentation:

·         Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

·         Document data engineering processes, workflows, and best practices for knowledge sharing and future reference.

Required skills : 

·         Bachelor's degree in Computer Science, Engineering, or related field.

·         Proven experience as a Data Engineer or similar role, with a minimum of 6 to 8 years of relevant experience.

·         Proficiency in Python, PySpark, SQL, and Databricks.    

·         Good to have familiarity with Claude Code or Github Copilot or Codex ,GEN AI - Databricks Genie, LLM , Prompt engineering

·         Familiarity with Airflow for workflow orchestration.

·         Experience with Linux administration and shell scripting.

·         Strong understanding of data modelling, ETL processes, and data warehousing concepts.

·         Excellent problem-solving skills and attention to detail.

·         Strong communication and collaboration skills, with the ability to work effectively in a team environment.