Aws Developer

Job Description:

• We are seeking a skilled AWS Glue Developer Data Engineer to create, enhance, and optimize robust ETL and ELT data pipelines on the AWS platform.
• The ideal candidate should possess extensive hands-on experience with AWS Glue, Python, PySpark, SQL, data lake technologies, and various AWS data services.

Key Responsibilities:

• Key duties include:
• Designing, developing, and maintaining ETL and ELT pipelines leveraging AWS Glue.
• Creating AWS Glue jobs using PySpark and SQL to facilitate large-scale data processing.
• Building data ingestion and transformation pipelines for both structured and semi-structured data sources.
• Configuring and managing AWS Glue Data Catalog, Jobs, Workflows, and Triggers.
• Utilizing AWS services, including Amazon S3, Redshift, and IAM.
• Implementing data transformations, validations, quality checks, and reconciliation processes.
• Optimizing Glue Spark jobs to ensure performance, scalability, reliability, and cost-effectiveness.
• Applying incremental load strategies, partitioning, error handling, restart recovery, and audit frameworks.
• Troubleshooting failures in ETL pipelines and addressing production performance issues.
• Establishing appropriate security measures, including IAM roles, encryption, and access management.
• Assisting in the migration of legacy mainframe or on-premises ETL workloads to the AWS cloud data platform.
• Developing automated deployment processes using CI/CD and Infrastructure as Code methodologies when applicable.
• Collaborating with data architects, analysts, business teams, and cloud platform teams to convert requirements into scalable data solutions.
• Preparing technical design documents, mapping specifications, operational documentation, and deployment runbooks.

Technical Requirements:

• Core skills include: Technology Data On Cloud NoSQL Amazon DynamoDB, Technology Data On Cloud Platform AWS.

Additional Responsibilities:

• Minimum of 2 years of comprehensive experience in Data Engineering and ETL development.
• Proven expertise in developing production-grade AWS Glue solutions using Python and PySpark.
• Experience in managing high-volume datasets and executing complex transformations within AWS environments.
• Preferred Certification:
• AWS Certified Data Engineer Associate or another relevant AWS certification.

Preferred Skills:

• Technology -> Cloud Platform -> AWS Database, Technology -> Data On Cloud - Platform -> AWS.