IN
Job Description
- Develop and optimize scalable data pipelines using Databricks and Apache Spark.
- Design and implement solutions for data integration, data processing, and data storage on Databricks.
- Collaborate with data scientists and analysts to build end-to-end analytics solutions.
- Build and maintain automated workflows, ensuring reliability and efficiency.
- Design and manage ETL processes in Databricks.
- Write optimized Spark SQL and PySpark code for data transformations.
- Work with large datasets and ensure performance, scalability, and reliability of data pipelines.
- Monitor and troubleshoot existing data pipelines and workflows.
- Integrate Databricks with other tools and systems like AWS, Azure, or Google Cloud.