Aws Developer

Job Description:

• We are seeking a seasoned AWS Glue Developer Data Engineer to create, develop, and enhance scalable ETL and ELT data pipelines on the AWS platform.
• The ideal candidate must possess robust hands-on expertise in AWS Glue, Python, PySpark, SQL, data lake technologies, and AWS data services.

Key Responsibilities:

• Design, develop, and sustain ETL and ELT pipelines utilizing AWS Glue.
• Create AWS Glue jobs using Python, PySpark, and SQL for large-scale data processing.
• Construct data ingestion and transformation pipelines for both structured and semi-structured data sources.
• Set up and oversee AWS Glue Data Catalog jobs, workflows, and triggers.
• Engage with AWS services such as Amazon S3, Redshift, and IAM.
• Execute data transformations, validations, data quality checks, and reconciliation procedures.
• Enhance Glue Spark jobs for optimal performance, scalability, reliability, and cost-effectiveness.
• Implement incremental loads, partitioning strategies, error handling, restart recovery, and audit frameworks.
• Diagnose ETL pipeline failures and resolve production performance challenges.
• Establish essential security measures, including IAM roles, encryption, and access management.
• Facilitate the migration of legacy mainframe or on-premises ETL workloads to the AWS cloud data platform.
• Create automated deployment processes utilizing CI/CD and Infrastructure as Code practices where applicable.
• Collaborate with data architects, analysts, business units, and cloud platform teams to convert requirements into scalable data solutions.
• Prepare technical design documents, mapping specifications, operational documentation, and deployment runbook materials.

Technical Requirements:

• Primary skills include Technology Cloud Platform AWS Environment Provisioning, Technology Data On Cloud NoSQL Amazon DynamoDB, Technology Data On Cloud Platform AWS.

Additional Responsibilities:

• Technical Skills
• AWS Glue
• Python, PySpark, Apache Spark
• SQL
• Amazon S3
• Data Lake and Cloud Data Warehouse concepts
• ETL and ELT design and development
• Performance optimization and troubleshooting
• Experience
• 5 years of comprehensive Data Engineering and ETL development experience
• Strong practical experience in creating production-grade AWS Glue and PySpark solutions
• Experience in managing high-volume datasets and complex transformations within AWS environments
• Preferred Certification
• AWS Certified Data Engineer Associate or another relevant AWS certification

Preferred Skills:

Technology->Cloud Platform->AWS Core services, Technology->Cloud Platform->AWS Database, Technology->Data On Cloud - Platform->AWS