DataBricks - Performance Tuning

Key Responsibilities

• Develop and Optimize: Design, implement, and optimize robust data processing pipelines using PySpark for large-scale data processing tasks.
• Data Transformation: Collaborate with data teams to transform and aggregate data from various sources, ensuring data quality and integrity throughout the process.
• Performance Tuning: Analyze and tune Spark applications to improve performance and efficiency, employing best practices for Spark configuration and resource management.
• Testing and Validation: Conduct comprehensive testing and validation of data workflows, ensuring the accuracy and reliability of processed data.
• Technical Collaboration: Work closely with data engineers, data scientists, and business analysts to gather requirements and provide technical solutions tailored to data analytics needs.
• Documentation: Maintain clear documentation of data processing workflows, Spark applications, and best practices to support knowledge sharing within the team.