Sr. Specialist, Platform Engineering

Brief introduction - Role Summary/Purpose:

Platform Engineer is focused in maintaining/ supporting the data ingestion related platforms that are used in Colgate for extracting data and taking the complete ownership of these tools. You must be self-directed and comfortable supporting multiple teams and their request for platform support. You will be thrilled by the prospect of working with the best data ingestion tools and do additional automations to fully use the capability of the tools . You will have  to drive standards in the usage of the platforms. This team is a group of innovators and technologists that love to learn and collaborate!

 

Responsibilities :

• Maintain and operate data ingestion platforms, specifically focusing on the Airflow instance hosted on the GCP Kubernetes cluster, ensuring high availability and optimal performance.

• Identify, design, and implement internal process improvements, automating manual processes, and optimizing data delivery and infrastructure for greater scalability.

• Support and monitor all the Airflow instances. Perform root cause analysis on data pipelines and processes to drive continuous improvement.

• Plan and execute timely Airflow namespaces upgradation  to the latest stable version and provide guidance to the data delivery teams on how to adapt to the latest version of the Airflow as part of the standard documentation.

• Create and update technical documentation for architectural governance, support, and educating others on standard data ingestion processes (e.g., ETL design, job scheduling, and monitoring).

• Collaborate with Data Delivery teams to assist with technical issues and support their data infrastructure needs.

• Assist and guide the teams with the Airflow standards that have been set by the platform team.

• Proactively share knowledge on emerging technologies and tools relevant to upcoming projects.

• Act as a technical subject matter expert, guiding the team through Airflow and its standards.

 

Required Qualifications:

• Bachelor's Degree or relevant

• Minimum 3 years of experience supporting data ingestion tools and data pipelines.

• Proven expertise in managing and maintaining Apache Airflow environments, including deployment, monitoring, and troubleshooting.

• Proficiency in object-oriented scripting languages, particularly Python.

• Working knowledge of SQL, relational databases, and experience building processes supporting data transformation, dependency, and workload management.

• Experience supporting and working with multi-functional teams in a dynamic environment.

 

Preferred Qualifications :

• Master's Degree in a quantitative field (Computer Science, Statistics, Informatics, etc.).

• Experience with cloud services, specifically GCP, and DevOps practices using tools like Kubernetes and Terraform.

• Familiarity with big data tools (e.g., Hadoop, Spark, Kafka) and various data ingestion tools (e.g., HVR, Airbyte).

• Strong analytic skills related to working with unstructured datasets

• Build processes supporting data transformation, data structures, metadata, dependency and workload management 

• A successful history of harmonizing, processing and extracting value from large disconnected datasets

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases

 

#LI-HR1

#colgateemployee, #teamcolgate, #wearecolgate, #buildafuture, #CPValues