Sr. Data Engineer/Sr. BI Developer-DA&A

Job Title

Sr. Data Engineer / Sr. BI Developer - DA&A

Expected Work Experience

4 to 7 Years

Job Location

India

Job Summary

We are seeking a Sr. Data Engineer / Sr. BI Developer - DA&A with 4-7 years of experience in designing and building scalable cloud-based data platforms using AWS, Python, AWS Glue, Apache Airflow, Amazon S3, Amazon Redshift, and AWS Step Functions. Candidate must have relevant experience of 4 to 7 Years. The ideal candidate should have hands-on expertise in developing robust ETL/ELT pipelines, orchestrating complex workflows, implementing data ingestion frameworks, and supporting enterprise data warehousing solutions. Experience with Python, PySpark, SQL, and cloud-native data processing is essential. The role involves building and optimizing data pipelines using AWS Glue, managing workflow orchestration through Airflow and Step Functions, designing data lake architectures on Amazon S3, and supporting analytics and reporting solutions on Amazon Redshift. The candidate should be proficient in data modeling, performance tuning, monitoring, troubleshooting, CI/CD practices, and automation. Exposure to modern data engineering tools such as Snowflake, Databricks, dbt, Terraform, or similar cloud technologies will be considered an added advantage.

Key Responsibilities

- Design, develop, and maintain scalable, cloud-based data platforms and solutions on AWS.
- Build, optimize, and manage ETL/ELT data pipelines using AWS Glue, Python, and PySpark.
- Orchestrate complex data workflows and batch processes using Apache Airflow and AWS Step Functions.
- Design and implement data lake architectures on Amazon S3 to support analytics and reporting needs.
- Develop and support enterprise data warehousing solutions on Amazon Redshift, including data loading, transformations, and performance tuning.
- Implement robust data ingestion frameworks to integrate data from multiple structured and unstructured sources.
- Collaborate with data analysts, BI developers, and business stakeholders to understand data requirements and translate them into technical solutions.
- Apply data modeling best practices for analytical and reporting use cases, including dimensional and relational models.
- Monitor data pipelines and workflows, proactively identify issues, and perform root cause analysis and troubleshooting.
- Implement CI/CD practices for data engineering workflows, including version control, automated testing, and deployment automation.
- Ensure data quality, integrity, and consistency through validation, reconciliation, and automated checks.
- Optimize data processing performance and costs by leveraging appropriate AWS services and configurations.
- Create and maintain technical documentation for data pipelines, data models, and workflows.
- Support operationalization and automation of recurring data engineering tasks and processes.
- Explore and evaluate modern data engineering tools such as Snowflake, Databricks, dbt, Terraform, or similar technologies to enhance the data platform capabilities.

Other Capabilities

- Strong problem-solving and analytical skills with attention to detail.
- Ability to work in an agile, collaborative environment with cross-functional teams.
- Effective communication skills to explain technical concepts to non-technical stakeholders.
- Demonstrated ability to prioritize, manage multiple tasks, and deliver high-quality outcomes within timelines.
- Continuous learning mindset with interest in emerging data engineering tools and cloud technologies.
- Experience working in data analytics and data automation environments, including Data Engineering, Data Analytics, and Data Modelling (Technical).

Qualifications and Skills

- Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 4-7 years of hands-on experience as a Data Engineer, Sr. Data Engineer, or Sr. BI Developer in data and analytics environments.
- Strong proficiency in Python and PySpark for data processing and transformation.
- Practical experience with AWS services including AWS Glue, Amazon S3, Amazon Redshift, AWS Step Functions, and related data services.
- Experience with Apache Airflow for workflow orchestration and scheduling of data pipelines.
- Strong SQL skills for querying, data transformation, and performance tuning in analytical databases.
- Solid understanding of data warehousing concepts, data modeling techniques (dimensional and relational), and BI/analytics use cases.
- Experience with monitoring, logging, and troubleshooting data pipelines and workflows in production environments.
- Familiarity with CI/CD practices, version control (e.g., Git), and automated deployment for data engineering projects.
- Knowledge of cloud-native data processing patterns and best practices for scalable, resilient data architectures.
- Exposure to modern data engineering tools such as Snowflake, Databricks, dbt, Terraform, or similar cloud technologies is an added advantage.
- Experience in Data Engineering, Data Analytics, and Data Modelling (Technical) with a focus on Data Automation & AI will be beneficial.