Aws Developer

Job Overview:

• We are seeking a skilled AWS Glue Developer Data Engineer to architect, build, and enhance scalable ETL and ELT data pipelines within the AWS ecosystem.
• The ideal candidate will possess substantial hands-on expertise with AWS Glue, Python, PySpark, SQL, data lake technologies, and various AWS data services.

Primary Duties:

• Design, develop, and sustain ETL and ELT pipelines utilizing AWS Glue.
• Create AWS Glue jobs employing PySpark and SQL for extensive data processing tasks.
• Construct data ingestion and transformation workflows across both structured and semi-structured data sources.
• Set up and oversee AWS Glue Data Catalog Jobs, Workflows, and Triggers.
• Engage with AWS services including Amazon S3, Redshift, and IAM.
• Conduct data transformations, validations, quality checks, and reconciliation processes.
• Enhance Glue Spark jobs for optimal performance, scalability, reliability, and cost-effectiveness.
• Implement incremental loads, partitioning strategies, error handling, restart recovery, and audit frameworks.
• Diagnose ETL pipeline failures and address production performance challenges.
• Apply suitable security measures, encompassing IAM roles, encryption, and access management.
• Facilitate the migration of legacy mainframe or on-premises ETL workloads to the AWS cloud data platform.
• Establish automated deployment workflows through CI/CD and Infrastructure as Code methodologies where applicable.
• Collaborate with data architects, analysts, business units, and cloud platform teams to convert requirements into scalable data solutions.
• Draft technical design documents, mapping specifications, operational documentation, and deployment runbook materials.

Additional Duties:

• Technical Proficiencies
• AWS Glue
• Python, PySpark, Apache Spark
• SQL
• Amazon S3
• Data Lake and Cloud Data Warehouse concepts
• ETL and ELT design and development
• Performance tuning and troubleshooting
• Experience Requirements
• 7+ years of comprehensive Data Engineering and ETL development experience
• Proven hands-on experience in building production-ready AWS Glue solutions using Python and PySpark
• Expertise in managing high-volume datasets and complex transformations within AWS environments
• Preferred Certification
• AWS Certified Data Engineer Associate or another relevant AWS certification

Desired Skills:

Technology->Cloud Platform->AWS Database->AWS, Technology->Cloud Platform->AWS Core Services, Technology->Data on Cloud - Platform->AWS