Big Data Engineer – Java, Apache Spark, AWS EMR, Lambda, EKS & Airflow

Job Summary

Synechron is seeking a Big Data Engineer with 5+ years of experience in designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Apache Spark. The role will focus on implementing big data processing pipelines using Apache Spark on AWS EMR, developing and deploying serverless applications using AWS Lambda, utilizing Amazon EKS for container orchestration and microservices management, and designing workflow orchestration using Apache Airflow. The position will contribute to business objectives by delivering reliable, scalable and maintainable data-processing solutions that support efficient data operations and application services.

Software Requirements

Required

• Java: Strong hands-on experience in Java development.
• Apache Spark: Proficiency in Apache Spark for distributed data processing.
• AWS EMR: Experience implementing big data processing pipelines using Apache Spark on AWS EMR.
• AWS Lambda: Experience developing and deploying serverless applications using AWS Lambda.
• AWS Service Integration: Experience integrating AWS Lambda with other AWS services.
• Amazon EKS: Experience utilizing Amazon EKS for container orchestration and microservices management.
• Apache Airflow: Experience designing and implementing workflow orchestration for complex data pipelines.
• AWS: Experience designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Spark.
• Project-Supported Versions: Ability to work with the versions of Java, Apache Spark, AWS services and related tools supported by the project.

Preferred

• No additional preferred software skills are specified in the requirements. Additional experience relevant to Java, Apache Spark, AWS, AWS EMR, AWS Lambda, Amazon EKS or Apache Airflow may be considered beneficial.

Overall Responsibilities

• Design, develop and deploy scalable and reliable data-processing solutions on AWS using Java and Apache Spark.
• Implement and maintain big data processing pipelines using Apache Spark on AWS EMR.
• Develop and deploy serverless applications using AWS Lambda and integrate them with other AWS services.
• Utilize Amazon EKS for container orchestration and microservices management.
• Design, implement and maintain Apache Airflow workflows for complex data pipelines.
• Develop maintainable Java applications and data-processing components aligned with technical requirements.
• Contribute to solution design, technical discussions and implementation decisions within agreed project standards.
• Test, troubleshoot and resolve application, pipeline, integration and deployment issues.
• Support reliable operation of data-processing solutions across relevant development, testing and production environments.
• Monitor application and pipeline behavior and contribute to improvements in scalability, reliability and maintainability.
• Collaborate with engineering, architecture, QA and delivery teams to clarify requirements and coordinate implementation activities.
• Contribute to code reviews, technical documentation, release activities and knowledge sharing.
• Apply appropriate data protection and cloud security practices during development and deployment.
• Use AWS resources efficiently and consider sustainable practices that reduce unnecessary compute, storage and network consumption.
• Deliver assigned work in accordance with agreed priorities, timelines and quality expectations.

Technical Skills (By Category)

Programming Languages

Essential

• Strong hands-on experience in Java development.
• Ability to develop clean, modular, reusable and maintainable Java code.
• Experience using Java to develop data-processing solutions and supporting application components.
• Ability to debug Java code and resolve functional, integration and deployment issues.

Preferred

• Additional experience with languages used in data engineering may be considered beneficial.

Databases and Data Management

Essential

• Experience implementing big data processing pipelines.
• Understanding of distributed data processing using Apache Spark.
• Ability to design data-processing workflows that support reliability and scalability.
• Ability to manage pipeline execution, dependencies and processing failures.

Preferred

• Additional experience with data storage, transformation, validation or integration may be considered beneficial.

Cloud Technologies

Essential

• Experience designing, developing and deploying data-processing solutions on AWS.
• Experience with AWS EMR for Apache Spark-based big data processing.
• Experience with AWS Lambda for serverless application development and deployment.
• Experience integrating AWS Lambda with other AWS services.
• Experience with Amazon EKS for container orchestration and microservices management.
• Experience using project-supported AWS service configurations and versions.

Preferred

• Additional experience relevant to AWS-based data-processing and application deployment may be considered beneficial.

Frameworks and Libraries

Essential

• Apache Spark: Proficiency in distributed data processing.
• Apache Airflow: Experience designing and implementing workflow orchestration for complex data pipelines.
• Experience integrating Apache Spark with AWS EMR.
• Experience using frameworks and libraries compatible with project-supported Java, Spark and AWS versions.

Preferred

• Additional experience developing reusable data-processing components may be considered beneficial.

Development Tools and Methodologies

Essential

• Experience with the design, development and deployment lifecycle for data-processing solutions.
• Experience deploying and maintaining applications and pipelines in AWS environments.
• Ability to troubleshoot data pipelines, serverless applications, containerized services and workflow processes.
• Ability to document technical solutions, workflows and deployment-related activities.
• Ability to collaborate with relevant technical and delivery stakeholders.

Preferred

• No additional development tools or methodologies are specified in the requirements.
• Experience with automated testing, deployment or monitoring practices may be considered beneficial.

Security Protocols

Essential

• Understanding of secure development practices for cloud-based applications and data-processing solutions.
• Awareness of appropriate access control for AWS services and deployed applications.
• Ability to protect data during processing, transmission and storage.
• Ability to manage application configuration and credentials securely.
• Ability to apply secure logging and error-handling practices.

Preferred

• No specific additional security protocols are specified in the requirements.
• Experience with cloud security controls for serverless, containerized and distributed data-processing solutions may be considered beneficial.

Experience Requirements

• 5+ years of experience in big data engineering, Java development, cloud engineering or a related technical role.
• Experience designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Apache Spark.
• Experience implementing big data processing pipelines using Apache Spark on AWS EMR.
• Experience developing and deploying serverless applications using AWS Lambda and integrating them with other AWS services.
• Experience utilizing Amazon EKS for container orchestration and microservices management.
• Experience designing and implementing workflow orchestration using Apache Airflow for complex data pipelines.
• Strong hands-on experience in Java development and proficiency in Apache Spark for distributed data processing.
• Experience with AWS services including EMR, Lambda, EKS and Airflow.
• Experience supporting the deployment, troubleshooting and maintenance of AWS-based data-processing solutions is relevant to the role.
• Candidates may qualify through equivalent experience in Java-based data engineering, distributed data processing, AWS engineering or cloud-based application development that demonstrates the required capabilities.

Day-to-Day Activities

• Develop, test and deploy Java and Apache Spark data-processing pipelines on AWS EMR and troubleshoot pipeline issues.
• Build and maintain AWS Lambda applications, Amazon EKS services and Apache Airflow workflows in collaboration with technical teams.
• Participate in technical discussions, requirement reviews, code reviews, planning meetings, defect resolution and release activities.
• Make implementation decisions within agreed technical and security standards, document deliverables and communicate progress, risks and dependencies.

Qualifications

• A bachelor’s degree in Computer Science, Information Technology, Engineering or a related field is preferred; equivalent relevant experience may be considered.
• Certifications in AWS, Java, Apache Spark, data engineering or related technical disciplines are preferred but not mandatory.
• Practical training or demonstrated experience in Java, Apache Spark, AWS EMR, AWS Lambda, Amazon EKS and Apache Airflow is required.
• Training or experience in distributed systems, serverless applications, container orchestration and cloud-based data processing is beneficial.
• Commitment to continuous professional development in Java, Apache Spark, AWS services, Apache Airflow and related data engineering practices is expected.

Professional Competencies

• Applies critical thinking to understand data-processing requirements and design scalable and reliable technical solutions.
• Uses structured problem-solving to troubleshoot pipeline failures, application defects, integration issues and deployment problems.
• Collaborates effectively with engineering, architecture, QA and delivery teams and contributes to shared technical outcomes.
• Communicates implementation decisions, progress, risks, dependencies and technical concerns clearly to relevant stakeholders.
• Adapts to evolving Java, Apache Spark, AWS, serverless, container orchestration and workflow-orchestration technologies.
• Manages priorities, technical dependencies and delivery timelines while maintaining solution quality, reliability and maintainability.

S​YNECHRON’S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice