PE
Job Description
What You'll Do:
- Install, configure, and maintain Databricks clusters and workspaces.
- Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
- Implement and manage access controls and security policies to protect sensitive data.
- Perform regular software updates and patches.
- Provide troubleshooting and issue resolution.
- Optimize cluster configurations for performance and cost-effectiveness.
- Monitor and manage auto-scaling to ensure resources are allocated efficiently.
- Implement and enforce security policies, access controls, and encryption mechanisms.
- Stay up-to-date with security best practices and compliance requirements.
- Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
- Collaborate with data engineers to integrate Databricks with other data sources, data warehouses, and data lakes.
- Monitor and manage platform costs, including optimizing resource utilization to control expenses.
- Maintain detailed documentation of configurations, procedures, and best practices.
- Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts,
- To understand their requirements and provide technical solutions
Expertise You'll Bring:
- Managing Cluster at Scale, troubleshooting within Databricks and Working with Databricks System tables
- Strong knowledge of Databricks infrastructure and administration to support the Analyst community in addressing Databricks-related queries and troubleshooting issues in notebooks,
- Good knowledge of AWS, Databricks
- Unity catalog, managing cluster, deployment, strong fundamental of DW/DH
- Strong in troubleshooting's
- Spark/Python either coding skills
- Docker, Kubernetes
- AWS(S3, Lambda, EMR)
- Certified Databricks Engineer
