AWS Data Engineer

Job Overview:

• We are seeking an experienced AWS Data Engineer who has practical knowledge in constructing and managing scalable data pipelines utilizing AWS services, PySpark, SQL, and ETL frameworks.
• The ideal candidate will possess a solid foundation in data engineering principles and have experience working with extensive structured and semi-structured datasets.

Primary Responsibilities:

• Design, develop, and sustain scalable data pipelines on the AWS platform.
• Create ETL and ELT workflows for data ingestion, transformation, and processing.
• Develop data processing solutions using PySpark and SQL.
• Conduct data transformations, validations, and ensure data quality checks.
• Enhance the performance and scalability of data pipelines and SQL queries.
• Integrate various data sources into cloud-based data platforms.
• Diagnose and resolve data pipeline failures and related issues.
• Collaborate with business analytics and engineering teams to comprehend data needs.

Technical Qualifications:

• Extensive hands-on experience with AWS Data Engineering services such as S3, Glue, Lambda, EMR, Redshift, Athena, and IAM.
• Proficient programming skills in PySpark.
• Strong command of SQL, including the ability to write complex queries and perform data transformations.
• Practical experience in developing ETL and ELT pipelines.
• Solid understanding of data warehousing, data lakes, and data modeling concepts.
• Experience in managing large-scale datasets and optimizing distributed data processing tasks.
• Familiarity with Python is a plus.
• Experience with Git, CI/CD orchestration tools, and Agile methodologies will be advantageous.

Additional Requirements:

• Strong analytical and problem-solving abilities.
• Excellent communication and teamwork skills.
• Capability to independently understand requirements and provide data engineering solutions.
• Experience with cloud migration or modernization initiatives will be considered an asset.

Preferred Competencies:

Technology->Cloud->AWS