Data Architect - Data Engineering 4D

Data Architect

Ready to turn bold ideas into real-world impact?
 At Genpact, we don’t just adapt to change, we lead it. AI and digital innovation are transforming the way businesses work, and we’re at the forefront of it. Genpact’s AI Gigafactory, our industry-first accelerator, exemplifies how we scale advanced technology solutions to help global enterprises work smarter, grow faster, and transform at scale. Whether tackling complex challenges through large-scale models or agentic AI, our breakthrough solutions tackle companies’ most complex challenges.
 
If you thrive in a fast-moving, innovation-driven environment, love building and deploying cutting-edge AI solutions, and want to push the boundaries of what’s possible, this is your moment. 
 
Genpact (NYSE: G) is an agentic and advanced technology solutions company. We leverage process intelligence and artificial intelligence to deliver measurable outcomes. With a strong partner ecosystem and decades of client trust, we provide innovative solutions that transform how businesses run. Powered by a team with an active learning mindset and client centricity at its core, we deliver lasting value for the world’s leading enterprises.
 Get to know us at genpact.com and on LinkedIn, YouTube, X, and Facebook.

Job Description

Senior DataOps Engineer

Team: Global Digital Operations — Data Platform

About the Role

Data Platform Services is building a governed, metadata-driven data platform on Azure Databricks / Unity Catalog — spanning source ingestion, a source-faithful central repository, long-term archival, and controlled distribution to multiple downstream systems. We're looking for a DataOps Engineer who treats pipeline and access-control logic as configuration, not one-off code — someone who has built or maintained systems where a metadata table drives what gets generated (jobs, views, permissions), rather than hand-coding each new dataset or consumer.

You'll work across ingestion (landing → central → archive), retention/partitioning strategy, and the routing-rules engine that determines how data is split, filtered, or masked across four major downstream systems.

What You'll Do

• Build and operate Auto Loader–based ingestion pipelines (batch and incremental) from Unity Catalog volumes into managed Delta tables, using MERGE patterns for upsert/dedup logic

• Design and maintain metadata-driven pipeline generation — where dataset onboarding (new source, new schema, new retention rule) is a config change against a rules table, not new pipeline code per dataset

• Implement and maintain partition-based retention (year partitioning; rolling window of current + N prior years) and the archive extraction process for datasets requiring longer history

• Build and maintain a dataset routing-rules engine: a governed metadata table (splittable, target_systems[], row_filter_expr, column_mask_expr) that programmatically generates consumption-layer views per downstream system

• Own Unity Catalog governance: catalog/schema/storage-location design, access grants, lineage, and audit trail across landing, central, archive, and consumption layers

• Manage Databricks Asset Bundles (DAB) or equivalent IaC for repeatable environment and job deployment across dev/test/prod

• Monitor pipeline health, checkpoint integrity, schema drift/evolution, and job SLAs; build alerting for ingestion failures and data-freshness breaches

• Collaborate with upstream source-system owners to define data contracts (e.g. ensuring _source_changed_at semantics are honored) and with downstream teams to translate their access/filter requirements into routing rules

• Document pipeline architecture, retention policy, and routing logic for audit and onboarding purposes

Required Skills & Experience

Core platform

• Azure Databricks and Unity Catalog (catalogs, schemas, managed/external volumes, managed/external tables, storage-location hierarchy, grants)

• Strong PySpark and SQL, including Structured Streaming / Auto Loader (cloudFiles, foreachBatch, trigger(availableNow=True)), MERGE INTO, and Delta Lake internals (transaction log, OPTIMIZE, VACUUM, ZORDER)

• Practical experience with partitioning strategy and retention management (partition-drop vs. rolling delete tradeoffs) at production data volumes

Metadata-driven engineering (critical)

• Demonstrable background designing or maintaining metadata/config-driven code generation — e.g. a rules or registry table that drives dynamically generated SQL, views, jobs, or DDL, rather than hardcoded per-object pipelines

• Comfort with templated/parameterized pipeline patterns (e.g. DAB with variable substitution, dbt-style codegen, or custom Python/SQL generation from a config table)

• Understanding of how to keep generated artifacts (views, jobs) in sync with a source-of-truth ruleset, including versioning and change auditability

Platform engineering / DataOps

• Azure fundamentals: ADLS Gen2, Azure Data Factory or equivalent orchestration, networking/private endpoints, cost management basics

• CI/CD for data pipelines (Azure DevOps or GitHub Actions), ideally with Databricks Asset Bundles or Terraform for Databricks

• Git-based workflow discipline; comfortable owning pipeline code as a shared, reviewed codebase rather than notebook-only development

• Monitoring/observability for data pipelines (job alerting, data quality checks, schema drift detection)

Governance & collaboration

• Working knowledge of data governance concepts: access control models, data lineage, retention/compliance-driven deletion, PII masking

• Experience working with multiple downstream consumers with differing data needs — building filtered/masked views rather than granting broad table access

• Clear written documentation habits; able to explain pipeline and governance logic to both technical and non-technical stakeholders

Nice to Have

• Experience with Delta Live Tables or Databricks Streaming Tables (declarative SQL ingestion)

• SnowFlake, Iceberg

• Exposure to multi-tenant data platforms serving several internal business-unit consumers with differing retention/access needs

What Success Looks Like in the First 6 Months

• New source onboarding (landing → central) reduced to a config-driven task, not custom pipeline development

• Partition-based retention live and operating automatically across central repository schemas

• Routing-rules table operational, with at least the initial set of downstream consumption views generated from it rather than hand-written

• Clear, current documentation of the ingestion → central → archive → consumption architecture

Qualifications

Bachelors - Business Analytics, Bachelors - Computer Science, Bachelors - Statistics, Masters - Data Science

Certifications

Certified Data Management Professional - RiversandRiversand, Databricks Certified Associate Developer for Apache Spark 3.0 - Databricks AcademyDatabricks Academy, Databricks Certified Associate Developer for Apache Spark - Databricks AcademyDatabricks Academy, Microsoft Certified: Azure Data Engineer Associate - MicrosoftMicrosoft, Oracle Database 12c Certified Implementation Specialist - OracleOracle

Required Skills

Agile Methodology, Change Management, Consumer Packaged Goods (CPG), Databricks Platform, Data Engineering, Snowflake GenAI

Language

English

Language Proficiency -

Proficient - C2

Additional Job Location -

Job Type

Regular

Master Skill List -

Data Engineering

Remote Type -

Office

Work Shift -

Flex Time (India)

Why join Genpact?
•    Lead AI-powered transformation – Drive innovation and solve real-world business challenges that matter
•    Make an impact – Help global enterprises solve business challenges that matter 
•    Accelerate your career – Gain hands-on experience, mentorship, and world-class learning opportunities to stay ahead
•    Work with the best – Join 140,000+ bold thinkers and problem-solvers who push boundaries every day 
•    Thrive in a values-driven culture – Our courage, curiosity, and incisiveness - built on a foundation of integrity and inclusion - allow your ideas to fuel progress 
 
Come join the 140,000+ coders, tech shapers, and growth makers at Genpact and take your career in the only direction that matters: Up.  
Let’s build tomorrow together. 
 
Genpact is an Equal Opportunity Employer and considers applicants for all positions without regard to race, color, religion or belief, sex, age, national origin, citizenship status, marital status, military/veteran status, genetic information, sexual orientation, gender identity, physical or mental disability or any other characteristic protected by applicable laws. Genpact is committed to creating a dynamic work environment that values respect and integrity, customer focus, and innovation.  
Furthermore, please do note that Genpact does not charge fees to process job applications and applicants are not required to pay to participate in our hiring process in any other way. Examples of such scams include purchasing a 'starter kit,' paying to apply, or purchasing equipment or training.