Senior Data Engineer
Pune · Staff/Principal
Applicants who checked fit first are 3.1× more likely to hear back
Your score for this role already exists
ASAI compared this JD against 41 signals - skills, seniority, domain, stack overlap etc. Add a resume and it unlocks in about 30 seconds.
Always free · no credit card · 1 tap with Google
What we know about this role
HIGH
Our tracked job listings for NCSi show strong recent activity.
First 72 hours
Window passed - posted 178d agoEarly applicants get seen before the pile builds.
Not a repost
The first time we've seen this listing - it hasn't been closed and reopened.
Skills required
17 listedYou almost certainly match several of these already. Unlock your skill map to see the matches, the gaps, and what to fix first.
Job description
Data Pipeline Development & Operations
• Design, build, and operate scalable and reliable data pipelines on the Databricks platform
• Develop end-to-end data workflows from ingestion through transformation to consumption
• Implement robust error handling, monitoring, and alerting mechanisms
• Ensure data pipeline reliability, performance, and maintainability
• Optimize pipeline performance through efficient Spark job design and cluster configuration
• Manage and orchestrate complex data workflows using Databricks Jobs and workflows
Legacy Code Modernization
• Refactor legacy code and data pipelines to PySpark for improved performance and scalability
• Migrate traditional ETL processes to modern ELT patterns on Databricks
• Assess existing codebases and identify opportunities for optimization and modernization
• Ensure backward compatibility and data integrity during migration processes
• Document refactoring approaches and create migration playbooks
• Collaborate with stakeholders to minimize disruption during code transitions
Data Engineering Excellence
• Implement data quality checks and validation frameworks
• Design and maintain Delta Lake tables with appropriate optimization strategies
• Develop reusable code libraries and frameworks for common data engineering tasks
• Follow software engineering best practices including version control, testing, and CI/CD
• Participate in code reviews and provide constructive feedback to team members
• Troubleshoot and resolve data pipeline issues in production environments
Collaboration & Knowledge Sharing
• Work closely with data architects, analysts, and business stakeholders
• Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams
• Share knowledge and best practices with Team NCS
• Mentor junior data engineers on PySpark and Databricks technologies
• Document technical solutions and maintain comprehensive documentation
Essential Technical Skills
• Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns
• PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization
• Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration
• Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration
• Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures
• Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques
• Python: Strong Python programming skills for data processing and automation
Additional Technical Skills
• SQL proficiency for data querying and transformation
• Experience with cloud platforms (Azure, AWS, or GCP)
• Understanding of data governance and security best practices
• Knowledge of streaming data processing (Structured Streaming)
• Familiarity with DevOps practices and CI/CD pipelines
• Experience with version control systems (Git)
• Understanding of data quality frameworks and testing methodologies
Professional Experience
• Minimum 8 years in data engineering or related roles
• At least 2-3 years of hands-on experience with Databricks platform
• Proven track record of refactoring legacy code to modern frameworks
• Experience building and maintaining production data pipelines at scale
• Background working across multiple data sources and formats
• Experience in agile development environments
Required Certifications - mandatory to have at least one certification
• Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional
Additional Certifications (Preferred)
• Databricks Certified Associate Developer for Apache Spark
• Cloud platform certifications (Azure Data Engineer Associate, AWS Certified Data Analytics, or Google Cloud Professional Data Engineer)
• Relevant data engineering or big data certifications
Soft Skills
• Strong problem-solving and analytical thinking abilities
• Excellent communication skills to explain technical concepts clearly
• Ability to work collaboratively in cross-functional teams
• Self-motivated with strong attention to detail
• Adaptable to changing priorities and technologies
• Client-focused mindset with commitment to quality delivery
Department
Apps
Open Positions
1
Skills Required
data engineer,Migration,databricks
Role
Data Engineer, Databricks, Migration
Years Of Exp
5 to 8 years
Designation
Senior Data Engineer
Tools to land this role
PremiumYour match score, skill map and tracker stay free.
Ground Reality
PremiumSpot the dealbreakers in this 615-word post before you apply
Skip jobs with hidden dealbreakers. See what they actually need, what's easy to miss, and how they describe the team.
Ground Reality on every job, plus Insider Path, practice interviews and resume tailoring.
Insider Path
PremiumFind someone who can refer you here
We help you find people there who worked or studied where you did, and draft your first message and follow-up.
Free · no signup
Get tomorrow's jobs before you have to search
Daily job drops, skill trends and free resources - posted straight to the group. Leave any time.
No spam. Just jobs and resources.
Why people use ASAI
Someone shared one job with you. ASAI keeps finding the rest.
Scored, not searched. Every role ranked against your actual profile.
Alerts as often as hourly. Reach new roles while the pile is still small.
Skill gaps, spelled out. See exactly which requirements you don't meet yet.
Verified jobs, only. Say no to ghost jobs. Your time deserves respect.
More Data Engineer roles in Pune
See all
Keep browsing