Pyspark Expert

Roles & Responsibilities:

• Design, develop, and optimize data processing pipelines using PySpark
• Collaborate with multiple teams to contribute to key technical and business decisions
• Provide effective solutions to complex problems across teams
• Develop and manage Airflow workflows (DAGs) for data orchestration
• Perform data analysis and validation using SQL
• Monitor, debug, and optimize performance of distributed data processing systems
• Work with AWS services such as EMR, S3, and IAM for data pipeline deployment
• Ensure efficient deployment and performance of applications on cloud platforms
• Mentor junior team members and support their technical growth
• Continuously improve software development processes and ensure adherence to Agile methodologies