Lead data Engineer

Job Responsibilities • Responsible for leading a team of talented data engineers responsible for designing, building, and maintaining scalable data pipelines and infrastructure • Work closely with cross-functional teams to ensure client data systems meet the highest standards of quality and performance • Lead, mentor, and develop a team of data engineers, fostering a collaborative and inclusive team environment • Conduct regular performance reviews, provide feedback, and set goals for team members • Identify and address skill gaps, and provide opportunities for professional development • Plan, execute, and deliver data engineering projects on time and within scope • Coordinate with stakeholders to gather requirements, set priorities, and define project timelines. • Ensure projects align with overall business objectives and data strategy • Oversee the design, development, and maintenance of data pipelines, ETL processes, and data warehouse • Ensure data quality, integrity, and security across all data engineering projects. • Identify opportunities for process improvements and drive initiatives to enhance the efficiency and effectiveness of data operations. • Has strong conceptual understanding of Data Warehousing and ETL, Data Governance and Security, Cloud Computing, and Batch & Real Time data processing • Ability to build/drive reusable frameworks that can drive efficiency of the overall data system • Has executed and lead multiple projects including on - streaming, batch, large data pipelines, etc. • Manages conversation with the client stakeholders to understand the Internal Use Only requirement and translate it into technical outcomes. Required Tech Stack • Strong experience with Databricks, Spark, and cloud platforms (Azure, AWS, GCP). • Architect and deploy cloud-based data solutions (Azure, AWS, GCP). • Define CI/CD strategies for data pipelines using Terraform, Azure DevOps, or GitHub Actions. • Implement data cataloging, lineage tracking, and access control (Unity Catalog, Collibra, Alation). • Ensure compliance with GDPR, CCPA, and industry-specific data security policies. • Develop strategies for distributed computing, parallel processing, and caching mechanisms. Required Non-Tech Stack • Partner with data architects, product managers, and business leaders to define data requirements and align engineering efforts with business objectives. • Define data engineering standards and playbooks to streamline development. • Oversee end-to-end project execution, from scoping to delivery. • Stay updated with emerging trends in data engineering, AI, and analytics to continuously improve architectures. • Evaluate and recommend new data tools, frameworks, and best practices. • Ability to translate complex technical concepts into business-friendly language. • Excellent communication, leadership, and stakeholder management. Good to Have Tech Stack • Experience with machine learning and advanced analytics technologies • Familiarity with data visualization tools and techniques • Knowledge of data security and privacy practices • Understanding of data governance and compliance frameworks • Experience with containerization and orchestration technologies (e.g., Docker, Kubernetes) • Experience with graph databases and graph processing frameworks • Experience with data virtualization and data federation techniques • Proficiency in data profiling and data quality management