About the Role

As a Data Engineer in the Edge of Technology Center, you will play a critical role in designing and implementing scalable data infrastructure to power advanced analytics, AI/ML, and business intelligence. This position demands a hands-on technologist who can architect reliable pipelines, manage real-time event streams, and ensure smooth data operations across cloud-native environments. You will work closely with cross-functional teams to enable data-driven decisionmaking and innovation across the organization.

Key Responsibilities:

Design, implement, and maintain robust ETL/ELT pipelines using tools like Argo Workflows or Apache Airflow.

Manage and execute database schema changes with Alembic or Liquibase, ensuring data consistency.

Configure and optimize distributed query engines like Trino and AWS Athena for analytics.

Deploy and manage containerized workloads on AWS EKS or GCP GKE using Docker, Helmfile, and Argo CD.

Build data lakes/warehouses on AWS S3 and implement performant storage using Apache Iceberg.

Use Terraform and other IaC tools to automate cloud infrastructure provisioning securely.

Develop CI/CD pipelines with GitHub Actions to support rapid and reliable deployments.

Architect and maintain Kafka-based real-time event-driven systems using Apicuro and AVRO.

Collaborate with product, analytics, and engineering teams to define and deliver data solutions.

Monitor and troubleshoot data systems for performance and reliability issues using observability tools (e.g., Prometheus, Grafana).

Document data flows and maintain technical documentation to support scalability and knowledge sharing.

Key Deliverables:

Fully operational ETL/ELT pipelines supporting high-volume, low-latency data processing.

Zero-downtime schema migrations with consistent performance across environments.

Distributed query engines tuned for large-scale analytics with minimal response time.

Reliable containerized deployments in Kubernetes using GitOps methodologies.

Kafka-based real-time data ingestion pipelines with consistent schema validation.

Infrastructure deployed and maintained as code using Terraform and version control.

Automated CI/CD processes ensuring fast, high-quality code releases.

Cross-functional project delivery aligned with business requirements.

Well-maintained monitoring dashboards and alerting for proactive issue resolution.

Internal documentation and runbooks for operational continuity and scalability.

Qualifications:

Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field from a recognized institution.

Technical Skills:

Orchestration Tools: Argo Workflows, Apache Airflow
Database Migration: Alembic, Liquibase
SQL Engines: Trino, AWS Athena
Containers & Orchestration: Docker, AWS EKS, GCP GKE
Data Storage: AWS S3, Apache Iceberg
Relational Databases: Postgres, MySQL, Aurora
Infrastructure Automation: Terraform (or equivalent IaC tools)
CI/CD: GitHub Actions or similar
GitOps Tools: Argo CD, Helmfile
Event Streaming: Kafka, Apicurio, AVRO
Languages: Python, Bash
Monitoring: Prometheus, Grafana (preferred)

Soft Skills:

Strong analytical and problem-solving capabilities in complex technical environments.
Excellent written and verbal communication skills to interact with both technical and non-technical stakeholders.
Self-motivated, detail-oriented, and proactive in identifying improvement opportunities.
Team player with a collaborative approach and eagerness to mentor junior team members.
High adaptability to new technologies and dynamic business needs.
Effective project management and time prioritization.
Strong documentation skills for maintaining system clarity.
Ability to translate business problems into data solutions efficiently.