Principal Data Lead (Data Engineering with Full-Stack Dev) - Investment Systems

POSITION SUMMARY
We are seeking a Principal Data Lead to set the technical direction of our Reference Data Platform — the system of record that masters, governs, and distributes the firm’s core reference data (parties, instruments, portfolios, and accounts. This is a hands-on technical leadership role: you will own the end-to-end architecture of a layered, event-driven data platform — spanning sourcing, ingestion, entity mastering, a bitemporal golden copy, and an API-first distribution layer — and drive it from vision to production. You will be the senior-most engineering voice on reference data, shaping platform strategy, establishing engineering standards across teams, and partnering with product, data, and infrastructure leaders to deliver a scalable, vendor-agnostic master data capability. The ideal candidate is an expert Python engineer and data architect with deep experience in master data management (MDM) and reference data within the financial industry — including market data and vendor feeds — combined with hands-on experience applying Generative AI to data platforms, distributed data processing, and building reusable, independently scalable platform components.

ROLES & RESPONSIBILITIES
• Own the architecture of a layered reference-data platform, defining the contracts between sourcing, ingestion, mastering, golden-copy, and distribution layers so each is independently buildable, deployable, and scalable.
• Set technical direction for entity mastering — standardization, blocking, matching, survivorship, and crosswalk/lineage — designing rules as configuration (data, not code) that scale from exact-match fast paths to full-population fuzzy resolution.
• Architect the durable golden copy: a bitemporal, point-in-time source of truth with match evidence, per-attribute provenance, and a schema-last serving model that adapts to new sources and domains without migration.
• Define the event-driven backbone and cadence model (start-of-day, intraday, end-of-day, and on-demand) enabling batch today and real-time streaming (CDC / Kafka) without re-architecture.
• Design the API-first, self-documenting distribution surface (REST, curated data products, subscriptions) governed by data contracts and consumable by applications, analytics, and AI/agents.
• Drive the platform’s storage and compute strategy over an open lakehouse (Apache Iceberg, Snowflake) and elastic compute (Spark), ensuring an open, vendor-agnostic, no-lock-in foundation.
• Establish engineering standards, reference patterns, and a reusable component library (a layered monorepo) that make capabilities build-once, reuse-everywhere across domains.
• Lead cross-team technical design reviews, resolve the hardest architectural trade-offs, and align engineering execution with the firm’s data strategy.
• Champion data quality, governance, security, entitlements, audit, and observability as first-class, cross-cutting foundations of the platform.
• Mentor senior and staff engineers, grow technical depth across the team, and set the bar for engineering excellence.
• Partner with product and business stakeholders to translate reference-data needs into a durable, scalable platform roadmap.

QUALIFICATIONS

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 12+ years of professional software engineering experience, including significant time as a technical lead or architect of data-intensive platforms.
• Expert-level proficiency in Python for building production services, data pipelines, and platform frameworks.
• Proven experience architecting master data management (MDM) or reference/entity-data systems — entity resolution, survivorship, golden records, crosswalk, and lineage.
• Deep data engineering expertise: data modeling, ETL/ELT design, distributed processing (Spark), and data distribution patterns at scale.
• Hands-on experience designing API-first services and event-driven / streaming architectures (e.g., Kafka, CDC).
• Strong command of modern data storage — relational, non-relational, and lakehouse formats (e.g., Apache Iceberg) — and SQL.
• Experience with infrastructure-as-code (Terraform) and cloud-native deployment (containers, Kubernetes/EKS, CI/CD).
• Deep domain experience with reference and master data in the financial industry — parties / legal entities, instruments, portfolios, and accounts — and the identifier, hierarchy, and cross-referencing challenges specific to it.
• Hands-on experience integrating and mastering market data and vendor/administrator feeds (e.g., Bloomberg, exchange, and fund-servicing sources), including normalization, cross-referencing, and point-in-time handling.
• Practical experience applying Generative AI / LLMs to data platforms — e.g., grounded retrieval over governed data, entity-resolution and data-quality assistance, or AI/agent-ready data access with provenance guarantees.
• A track record of setting technical direction, driving architecture across teams, and mentoring senior engineers.
• Excellent judgment on architectural trade-offs, and the communication skills to align engineering, product, and business stakeholders.

PREFERRED QUALIFICATIONS
• Familiarity with bitemporal / point-in-time data modeling and governed, auditable data systems.
• Experience with workflow orchestration (e.g., Dagster, Airflow) and lakehouse catalogs (e.g., Polaris, Nessie, Glue).
• Exposure to commercial or vendor MDM/ontology platforms (e.g., Palantir Foundry) and experience integrating them as pluggable engines within an open architecture.
• Experience building agentic workflows or retrieval-augmented systems on top of enterprise data, and familiarity with LLM evaluation, guardrails, and grounding techniques.
• Experience with legal-entity identifiers and standards (e.g., LEI, GLEIF) and regulatory reference-data requirements.