AWS Data Engineer

Key Responsibilities:

• Extensive hands-on experience in AWS Data Engineering within large-scale cloud data environments
• Proficient in AWS services including S3, AWS Glue, AWS Lambda, AWS Redshift, AWS Lake Formation, AWS CloudWatch, and AWS IAM
• Demonstrated expertise in real-time and streaming data processing utilizing Amazon Kinesis Data Streams, Kinesis Firehose, Kinesis Analytics, Amazon Kafka, or similar streaming technologies
• Proficient in coding with Python, PySpark, and SQL, with a strong focus on optimization techniques
• Experience in constructing high-volume, low-latency data ingestion and transformation pipelines
• Extensive knowledge of Databricks Delta Lake, Structured Streaming, and Spark Performance Tuning
• Ability to manage TB to PB scale datasets, including partitioning strategies, file optimization, schema evolution, and data lifecycle management
• Solid understanding of Data Lake, Lakehouse, and Modern Data Platform architectures
• Practical experience in data modeling for analytical, operational, and reporting workloads
• Experience in implementing CI/CD pipelines using AWS CodePipeline, CodeBuild, CodeDeploy, GitHub Actions, or Jenkins
• Strong troubleshooting capabilities for production issues, performance bottlenecks, and cost optimization
• Exceptional client communication, stakeholder management, and problem-solving abilities

Technical Requirements:

• Overview of responsibilities and job description along with additional criteria for consideration
• Design and create scalable AWS Data Engineering solutions and cloud-native data platforms
• Develop both batch and real-time data pipelines using AWS Glue, PySpark, Databricks, S3, Kinesis, Redshift, and Kafka MSK
• Enhance and optimize Databricks Delta Lake and Spark workloads for large-scale data processing using PySpark and Databricks transformations
• Implement streaming solutions with Amazon Kinesis Data Streams, Firehose, and Analytics for near-real-time data ingestion and processing
• Manage high-volume TB to PB scale data pipelines with emphasis on performance, scalability, and cost efficiency
• Create complex SQL transformations, data models, and analytics-ready datasets
• Implement CI/CD and Infrastructure as Code practices using Terraform, CloudFormation, and DevOps methodologies
• Ensure data quality, governance, security, monitoring, and production support
• Collaborate with architects, stakeholders, and cross-functional teams in Agile delivery settings
• Troubleshoot and optimize comprehensive AWS data platforms, streaming pipelines, and cloud infrastructure

Preferred Skills:

Technology->Cloud->AWS Core Services, Technology->Cloud->AWS - Database Technologies, Technology->Cloud Platform->AWS Data Analytics