Aws Developer

Job Overview:

• We are seeking a skilled AWS Glue Developer Data Engineer who will be responsible for the design, development, and optimization of scalable ETL and ELT data pipelines on the AWS platform.
• The ideal candidate must possess extensive hands-on experience with AWS Glue, Python, PySpark, SQL, data lake technologies, and various AWS data services.

Core Responsibilities:

• Design, develop, and sustain ETL and ELT pipelines utilizing AWS Glue.
• Create AWS Glue jobs leveraging PySpark and SQL for large-scale data processing tasks.
• Construct data ingestion and transformation pipelines that handle both structured and semi-structured data sources.
• Configure and oversee AWS Glue Data Catalog, Jobs, Workflows, and Triggers.
• Engage with AWS services such as Amazon S3, Redshift, and IAM.
• Implement data transformations, validations, data quality checks, and reconciliation processes.
• Enhance Glue Spark jobs for optimal performance, scalability, reliability, and cost-effectiveness.
• Establish incremental loads, partitioning strategies, error handling, restart recovery, and audit frameworks.
• Diagnose and resolve ETL pipeline failures and production performance challenges.
• Implement necessary security measures, including IAM roles, encryption, and access management.
• Facilitate the migration of legacy mainframe or on-premises ETL workloads to the AWS cloud data platform.
• Develop automated deployment processes by employing CI/CD and Infrastructure as Code methodologies where applicable.
• Collaborate with data architects, analysts, business units, and cloud platform teams to transform requirements into scalable data solutions.
• Create technical design documentation, mapping specifications, operational documentation, and deployment runbook documentation.

Essential Qualifications:

• Technical Skills Required
• AWS Glue
• PySpark and Apache Spark
• SQL
• Amazon S3
• Data Lake and Cloud Data Warehouse concepts
• ETL and ELT design and development
• Performance optimization and troubleshooting expertise
• Experience
• A minimum of 3 years in Data Engineering and ETL development.
• Robust hands-on experience in developing production-grade AWS Glue solutions using Python and PySpark.
• Experience in managing high-volume datasets and complex transformations in AWS environments.
• Preferred Certification
• AWS Certified Data Engineer Associate or an equivalent AWS certification.

Preferred Skills:

Technology->Cloud Platform->AWS Database, Technology->Data On Cloud - Platform->AWS