Databricks Administration

Job Description:

• We are looking for a proactive Databricks Administrator to oversee and maintain the Databricks platform, facilitating data and analytics for the enterprise.
• This position is responsible for leading platform governance, enhancing security posture, implementing automation strategies, ensuring compliance, and managing reliability and cost, while establishing standards for extensive analytics and AI/ML workloads.

Key Responsibilities:

• Possess 7 years of experience in data platform engineering, with a specific focus on Databricks Spark environments.
• 7 years of expertise in administering AI LLM Agentic platforms at an enterprise level.
• Demonstrate strong hands-on knowledge of Databricks Lakehouse architecture, including Unity Catalog, Delta Live Tables (DLT), and Serverless SQL.
• Exhibit proficiency in Python, PySpark, and Spark SQL.
• Have a deep understanding of one or more leading cloud providers such as AWS, Azure, or GCP, along with their respective services.
• Take ownership of the administration of Databricks accounts and multi-workspace environments throughout the software development lifecycle (SDLC).
• Define and uphold platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations.
• Lead the governance strategy for Unity Catalog, encompassing catalogs, schemas, permissions, and external locations.
• Design and manage enterprise monitoring, alerting, compliance dashboards, and operational service level objectives (SLOs).
• Oversee management of high-severity incidents, conduct root cause analyses, and implement preventive measures.
• Establish and sustain security audit and regulatory compliance controls.
• Drive cost governance, capacity forecasting, and performance optimization across various workloads.
• Define and execute Infrastructure as Code (IaC) and automation strategies utilizing Terraform, APIs, and CI/CD pipelines.
• Standardize CI/CD processes for Databricks asset bundles and DevOps operating models.
• Collaborate with cloud platform security and data engineering teams on architecture and governance decisions.
• Manage platform documentation, onboarding standards, and the roadmap for operational maturity.
• Mentor and guide junior and mid-level Databricks administrators, serving as a senior technical escalation point.
• Administer Databricks accounts and workspaces across development, testing, and production environments.
• Configure and manage clusters, jobs, SQL warehouses, runtimes, repositories, and workspace settings.
• Implement monitoring and alerting systems, dashboards, and operational runbooks.
• Provide Tier 2 and Tier 3 support for platform incidents, ensuring effective resolutions and root cause analyses.
• Enforce security and governance protocols utilizing Unity Catalog, RBAC, service principals, and least privilege access principles.
• Manage secrets, audit logging, and compliance reporting.
• Administer Unity Catalog objects, external locations, and cloud storage integrations.

Technical Requirements:

• 7 to 9 years of experience in Databricks Administration and Big Data Platforms.
• Strong hands-on capabilities with Databricks Workspace, Clusters, Jobs, and Notebooks.
• Experience with Apache Spark, PySpark, Scala, and distributed data processing techniques.
• Familiarity with cloud platforms, with a preference for Azure; AWS and GCP are also acceptable.
• Experience with data storage systems such as ADLS, S3, and Blob Storage.
• Solid understanding of data engineering concepts, including ETL and ELT pipelines.
• Knowledge of security best practices, including IAM roles, data encryption, and private endpoints.
• Proficiency in SQL and scripting languages such as Python and Shell.
• Experience in performance tuning and cost optimization for clusters and workloads.

Additional Responsibilities:

• Infosys is a global leader in next-generation digital services and consulting.
• We assist clients in over 50 countries with navigating their digital transformation journeys.
• With more than four decades of experience managing the systems and operations of global enterprises, we expertly guide clients through their digital transitions.
• We achieve this by empowering the enterprise with an AI-driven core that prioritizes the execution of change.
• Additionally, we enable businesses with agile digital solutions at scale to achieve unparalleled levels of performance and customer satisfaction.
• Our continuous learning agenda fosters ongoing improvement through the development and transfer of digital skills, expertise, and insights from our innovation ecosystem.
• Infosys is committed to providing equal employment opportunities to all candidates and employees, regardless of race, color, sex, gender identity, sexual orientation, religious beliefs, national origin, pregnancy, childbirth or related medical conditions, status as a protected veteran or as a family member of a protected veteran, or disability.

Preferred Skills:

Technology->Data Engineering->Databricks