Data Engineer II

Key Expectations

Data Engineering — the foundation

• Ingestion & integration

  • Build and operate ingestion pipelines from enterprise source systems — cloud and on-premise ERP, CRM, ITSM, billing/revenue, planning and quoting platforms, and third-party operational systems — into the enterprise data lakehouse.
  • Work across multiple ingestion patterns — change data capture (CDC), orchestrated batch pipelines, event streaming, packaged analytics/replication tooling, and API-based ingestion — and select the pattern that fits the latency, volume, and cost profile of the use case, including near-real-time ingestion where the business value justifies it.
  • Own new-source onboarding end to end: source system analysis with application and ERP teams, schema and field availability confirmation, ingestion design, historical backfill, incremental load strategy, and exposure through the curated serving layers.

  Modeling & curation

  • Build and extend the medallion architecture — raw landing, cleansed/conformed, and curated business layers — including a conformed star-schema layer of facts and dimensions and a denormalized, query-optimized serving layer consumed by BI tools and downstream applications.
  • Write and optimize SQL/stored-procedure and Spark-based transformations; model dimensional entities and conformed keys (customer, supplier, product, order, contract, site) so data joins cleanly across Finance, Sales, Service and Supply Chain domains.
  • Build the semantic and reporting layer, landing business definitions, hierarchies and metric logic once in the curated layer with documented ownership — so definitions are reusable and AI-consumable rather than rebuilt inside every tool.

  Data quality, reconciliation & observability

  • Build and extend source-to-target reconciliation frameworks — record counts, change-timestamp deltas, and critical-column matching — and push beyond count-based recon toward content-level sync validation across replication, packaged-analytics and API-sourced data.
  • Implement data accuracy, completeness and freshness/recency metrics per data domain and surface them through the platform's observability layer.
  • Investigate and resolve production data disconnects end to end — isolate root cause across source, pipeline and curation; repair or reprocess; and put a control in place so the same failure doesn't recur.

  Reliability, performance & cost

  • Tune long-running transformations and reduce compute consumption; contribute to platform right-sizing, storage cleanup and cost-optimization work.
  • Manage schema drift and upstream release changes — programmatically detect deprecated or altered objects, assess downstream impact, and regression-test pipelines across lower environments before production.
  • Work within CI/CD and Git-based change control, with automated test generation and validation rather than manual regression cycles.
  • Support production: monitoring, alerting, incident triage and RCA for the pipelines you own, including month-end and quarter-close critical windows.

  Automation

  • Automate what the team does repeatedly — period-close processes, report generation, reconciliation, environment and admin workflows, infrastructure-as-code, and test generation — rather than absorbing the manual effort.