AWS Data Engineer

Key Responsibilities

• Design, develop, and maintain robust data pipelines utilizing PySpark on AWS.
• Create and manage ETL/ELT workflows leveraging AWS Glue and AWS Step Functions.
• Develop serverless applications and automate processes using AWS Lambda.
• Produce clean, efficient, and maintainable Python/PySpark code in accordance with engineering best practices.
• Provision and oversee cloud infrastructure using Terraform (Infrastructure as Code).
• Establish and maintain CI/CD pipelines to streamline code deployment, testing, and infrastructure modifications.
• Monitor, troubleshoot, and enhance data pipelines for optimal performance, reliability, and cost efficiency.
• Collaborate with business stakeholders to deliver effective data solutions.
• Adhere to DevOps, security, and coding standards throughout the project lifecycle.

Required Skills

• Extensive hands-on experience with PySpark and Python in data engineering.
• Proven experience in developing ETL pipelines using AWS Glue.
• Expertise in utilizing AWS Step Functions for workflow orchestration.
• Experience in building serverless applications with AWS Lambda.
• Practical experience with Terraform for Infrastructure as Code (IaC).
• Familiarity with implementing CI/CD pipelines using tools like GitLab, GitHub Actions, Jenkins, or similar.
• Solid understanding of AWS services, data lakes, IAM, S3, CloudWatch, and monitoring strategies.