Data Engineering Manager
Bhubaneshwar · Director
Applicants who checked fit first are 3.1× more likely to hear back
Your score for this role already exists
ASAI compared this JD against 41 signals - skills, seniority, domain, stack overlap etc. Add a resume and it unlocks in about 30 seconds.
Always free · no credit card · 1 tap with Google
What we know about this role
HIGH
Our tracked job listings for KFintech show strong recent activity.
First 72 hours
Window passed - posted 4d agoEarly applicants get seen before the pile builds.
Not a repost
The first time we've seen this listing - it hasn't been closed and reopened.
Skills required
25 listedYou almost certainly match several of these already. Unlock your skill map to see the matches, the gaps, and what to fix first.
Job description
About the Role
We are looking for a Data Engineering Manager (or a senior Tech Lead ready to step into management) to lead the teams building and running our data lake platform for enterprise clients. This is a player-coach role: you will manage 10–15 data engineers while staying deep enough in the stack to review pipeline designs, debug the ugly production incident, and set the engineering standards the platform is held to.
The work spans two distinct modes, and you will own both:
- Implementation projects — greenfield and migration builds: standing up data lake environments for new clients, building ingestion and transformation pipelines, and taking them through UAT to production
- Maintenance and support engagements — running live, SLA-bound production environments: pipeline reliability, data quality, batch-window adherence, incident response, and continuous enhancement
Hands-on Scala and production data pipeline experience are mandatory. This is a non-negotiable screening bar — our core processing layer is Scala/Spark, and you cannot lead this team, review its work, or earn its respect from behind a status report.
What You'll Own
1. Team Leadership: Implementation and Maintenance Squads
Manage 10–15 data engineers across build and run workstreams. Own hiring, onboarding, one-on-ones, performance management, and growth paths. Rotate engineers deliberately between implementation and maintenance so the team builds full-lifecycle judgment — engineers who build pipelines they'll never have to operate build worse pipelines. Grow senior engineers into tech leads who can own a client workstream independently.
2. Pipeline Architecture and Engineering Quality
Own the technical quality of the data platform:
- Ingestion pipelines — batch and file-based ingestion from heterogeneous client source systems, handling late arrivals, reprocessing, and upstream format drift
- Transformation layers — Scala/Spark processing jobs: design reviews, performance tuning (partitioning, shuffle behavior, skew handling, memory management), and code quality standards
- Orchestration — dependency-aware scheduling, retry and recovery semantics, idempotent job design, and batch-window management
- Data modeling — lake zoning (raw/curated/consumption), file formats and layout strategy, schema evolution, and downstream consumption contracts
- Data quality engineering — validation frameworks, reconciliation checks, anomaly detection, and quality gates that stop bad data before clients see it
3. Implementation Delivery
Own the engineering side of new client implementations: estimation, technical planning, environment setup, pipeline build, data migration and historical loads, UAT support, and cutover to production. Work with delivery and program management on committed timelines, and flag technical risk early with options rather than at the deadline.
4. Production Ownership and SLA Management
Own live environments for maintenance clients:
- Batch SLA adherence — pipelines complete inside committed windows, and misses are caught and communicated before the client's business day starts
- Incident management — triage, root cause, permanent fixes, and post-incident reviews that reduce recurrence
- Monitoring and observability — pipeline health, data freshness, quality metrics, and alerting that catches issues before downstream consumers do
- Capacity and performance management as client data volumes grow
- Disciplined change management in production environments — releases, hotfixes, and reprocessing runs that don't create new incidents
5. Multi-Client Platform Discipline
Keep the platform coherent across clients: config-over-fork as the default, client-specific customization contained and documented, and a core codebase that improvements flow back into. Resist the entropy of every client engagement drifting into its own snowflake — the economics of the practice depend on it.
6. Standards and Enforcement
Define and enforce the engineering standards for the data practice:
- Scala coding standards, review discipline, and shared libraries
- Pipeline design patterns — templates for ingestion, transformation, and reconciliation that new workstreams start from
- Testing expectations for data pipelines: unit tests on transformations, data validation suites, regression testing on reprocessing
- CI/CD for data workloads — versioned, promotable, rollback-capable deployments
- Documentation standards: pipeline runbooks, data dictionaries, and operational handover packs
Standards are measured in the codebase and the incident trends, not in the wiki.
7. Client and Stakeholder Engagement
Be the senior technical face of the data team to client counterparts: design walkthroughs, technical governance calls, incident explanations, and enhancement scoping. Give delivery leadership honest engineering status. Translate client asks into technical plans, and push back with alternatives when asks threaten platform coherence or committed SLAs.
What You'll Need
Must-Haves
- 8–12+ years in data engineering, with 2–4+ years leading data engineering teams as a Tech Lead or Engineering Manager
- Mandatory: deep, hands-on Scala expertise — production Scala/Spark development you have written, reviewed, and tuned yourself, not overseen from a distance
- Mandatory: production data pipeline experience at scale — you have built, operated, and debugged batch pipelines processing large volumes under committed time windows
- Demonstrated experience leading 8+ data engineers, including performance management and technical mentoring
- Strong command of the Spark internals that matter in production: partitioning strategy, shuffle and skew behavior, memory tuning, join strategies, and job troubleshooting from the Spark UI down
- Data lake / lakehouse implementation experience — lake zoning, file format and layout decisions (Parquet/ORC, Delta/Iceberg/Hudi or equivalent), schema evolution, and metadata management
- Experience with orchestration frameworks (Airflow or equivalent) and designing dependency-aware, restartable, idempotent job flows
- Experience running SLA-bound production data environments — batch windows, incident response, reconciliation, and the operational discipline live clients demand
- Strong SQL and data modeling fundamentals across relational and lake paradigms
- Experience with cloud data platforms (at least one of the major hyperscalers' data stacks; Databricks, Snowflake, or BigQuery exposure valued) and comfort with on-premise or hybrid deployments
- Ability to review a pull request in the morning and walk a client through an incident root cause in the afternoon — credible in both rooms
Strong-to-Have
- Experience in BFSI — ideally Mutual Fund / Asset Management — familiarity with NAV cycles, T+1 processing pressure, regulatory reporting, and reconciliation-heavy data flows
- Experience in multi-tenant or multi-client platform environments where one codebase serves many deployments
- Background spanning both implementation projects and long-running support engagements
- Hands-on cloud experience on AWS and/or GCP, with working knowledge of their core data engineering services — e.g. S3/GCS, Glue/Dataproc, EMR/Dataflow, Redshift/BigQuery, IAM and networking basics for data workloads — and comfort designing pipelines that run natively on cloud-managed infrastructure rather than only on self-managed clusters
- Exposure to modern lakehouse and transformation tooling — Snowflake, dbt, and/or Databricks — including how these fit alongside or replace hand-rolled Scala/Spark transformation layers for parts of the platform
- Experience with real-time streaming architectures — Kafka and/or Flink — for use cases needing continuous ingestion or low-latency processing, beyond the batch-dominant core of this platform
- Experience with data governance tooling — lineage, cataloging, access control — in regulated environments
- Infrastructure-as-code and containerization familiarity (Terraform, Kubernetes) for environment provisioning
How You'll Work
- Player-coach — reviews Scala, tunes the stubborn Spark job, joins the 6 a.m. incident bridge when it matters — but measures success by what the team delivers without you
- Operationally paranoid, in a good way — assumes pipelines fail, designs for restart and reconciliation, and treats monitoring as a first-class deliverable
- Standards with teeth — enforces quality through review gates, templates, and CI, not reminders
- Direct and low-drama — straight feedback to engineers, straight status to leadership and clients
- Pragmatic over purist — knows when the elegant refactor is worth it and when the client's batch window says otherwise
Tools to land this role
PremiumYour match score, skill map and tracker stay free.
Ground Reality
PremiumSpot the dealbreakers in this 1174-word post before you apply
Skip jobs with hidden dealbreakers. See what they actually need, what's easy to miss, and how they describe the team.
Ground Reality on every job, plus Insider Path, practice interviews and resume tailoring.
Insider Path
PremiumFind someone who can refer you here
We help you find people there who worked or studied where you did, and draft your first message and follow-up.
Free · no signup
Get tomorrow's jobs before you have to search
Daily job drops, skill trends and free resources - posted straight to the group. Leave any time.
No spam. Just jobs and resources.
Why people use ASAI
Someone shared one job with you. ASAI keeps finding the rest.
Scored, not searched. Every role ranked against your actual profile.
Alerts as often as hourly. Reach new roles while the pile is still small.
Skill gaps, spelled out. See exactly which requirements you don't meet yet.
Verified jobs, only. Say no to ghost jobs. Your time deserves respect.
More Data Engineer roles in Bhubaneshwar
See all
Keep browsing