Senior Data Engineer

The Senior Data Engineer is responsible for designing and implementing robust and complex data pipelines, optimizing data storage solutions, and ensuring high availability and scalability of data systems. This role involves working closely with cross-functional teams to deliver efficient and secure data solutions while mentoring junior engineers and contributing to architectural decisions. Key Responsibilities: 1. Data Pipeline Design and Implementation: Design, build, and optimize complex, scalable data pipelines for ingestion, processing, and storage. Implement data transformation and enrichment processes to support analytics and reporting needs. 2. Data Storage and Optimization: Develop and maintain scalable and cost-effective data storage solutions, including data lakes and warehouses. Optimize database performance and storage efficiency for large-scale data operations. 3. System Monitoring and High Availability: Ensure data system reliability and availability through monitoring, fault tolerant design, and proactive issue resolution. Implement disaster recovery and backup strategies for critical data systems. 4. Collaboration and Stakeholder Engagement: Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver tailored solutions. Act as a technical liaison between engineering and analytics teams to ensure seamless integration of data systems. 5. Technical Leadership and Mentorship: Provide technical guidance and mentorship to junior data engineers. Contribute to architectural discussions and decision-making processes. 6. Compliance and Documentation: Ensure compliance with data governance, security, and privacy policies. Maintain clear and up-to-date documentation of systems, processes, and workflows. Requirements Education and Qualifications • Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field. • Master’s degree or industry certifications in data engineering, big data, or cloud computing is a plus. Technical Skills • Advanced knowledge of relational (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., Cassandra, MongoDB). • Hands-on experience with data engineering tools and frameworks (e.g., Apache Spark, Kafka, Airflow). • Proficiency in cloud services for data engineering (AWS Redshift, Azure Data Factory, Google BigQuery). • Familiarity with DevOps practices and tools (e.g., Docker, Kubernetes, Terraform).