Databricks Administration

Job Description:

• We are looking for an experienced Databricks Administrator to oversee and support the Databricks platform for enterprise-level data and analytics initiatives.
• This position will be responsible for leading platform governance, enhancing security measures, automating strategies for compliance and reliability, and managing costs, all while establishing standards to facilitate large-scale analytics and AI/ML workloads.

Key Responsibilities:

• Minimum of 5 years of experience in data platform engineering with a specific focus on Databricks Spark environments.
• 5 years of experience in administering AI LLM Agentic platforms at an enterprise level.
• Databricks Expertise: Extensive hands-on experience with Databricks Lakehouse architecture, Unity Catalog, Delta Live Tables (DLT), and Serverless SQL.
• Technical Proficiency: Strong skills in Python, PySpark, and Spark SQL.
• Cloud Platforms: In-depth knowledge of one or more leading cloud providers such as AWS, Azure, or GCP, along with their associated services.
• Manage Databricks accounts and multiple workspace environments throughout the software development lifecycle (SDLC).
• Establish and enforce platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations.
• Lead the governance strategy for Unity Catalog, including management of catalogs, schemas, permissions, and external locations.
• Design and implement enterprise-level monitoring, alerting, compliance dashboards, and operational Service Level Objectives (SLOs).
• Oversee high-severity incident management, conducting root cause analysis and preventive measures.
• Implement and maintain security audits and regulatory compliance controls.
• Drive cost governance, capacity forecasting, and performance optimization for various workloads.
• Define and execute an Infrastructure as Code and automation strategy utilizing Terraform, APIs, and CI/CD pipelines.
• Standardize CI/CD processes, Databricks Asset Bundles, and DevOps operating models.
• Collaborate with cloud security and data engineering teams on architectural and governance decisions.
• Take ownership of platform documentation, onboarding standards, and the roadmap for operational maturity.
• Guide and mentor junior and mid-level Databricks administrators, serving as a senior technical escalation point.
• Administer Databricks accounts and workspaces across development, testing, and production environments.
• Configure and manage clusters, jobs, SQL warehouses, runtimes, repositories, and workspace settings.
• Implement monitoring, alerting, dashboards, and operational runbooks.
• Provide Tier 2 and Tier 3 platform support, incident resolution, and root cause analysis.
• Enforce security and governance protocols using Unity Catalog, RBAC, service principals, and least privilege access principles.
• Manage secrets, audit logging, and compliance reporting.
• Administer Unity Catalog objects, external locations, and cloud storage integrations.

Technical Requirements:

• 5 to 7 years of experience in Databricks Administration and Big Data Platforms.
• Strong hands-on experience with Databricks Workspace, Clusters, Jobs, and Notebooks.
• Experience with Apache Spark, PySpark, Scala, and distributed data processing techniques.
• Familiarity with cloud platforms, with a preference for Azure; AWS and GCP are also acceptable.
• Experience with data storage systems such as ADLS, S3, and Blob Storage.
• Robust understanding of data engineering concepts, including ETL and ELT pipelines.
• Knowledge of security best practices encompassing IAM roles, data encryption, and private endpoints.
• Proficiency in SQL and scripting languages such as Python and Shell.
• Experience in performance tuning and cost optimization for clusters and workloads.

Additional Responsibilities:

• Infosys is a global leader in next-generation digital services and consulting.
• We assist clients in over 50 countries in navigating their digital transformation journeys.
• With more than four decades of experience managing global enterprises, we expertly guide our clients through their digital evolution.
• We achieve this by empowering enterprises with an AI-driven core that prioritizes change execution.
• Additionally, we equip businesses with agile digital solutions at scale, delivering unmatched performance and customer satisfaction.
• Our continuous learning agenda fosters ongoing improvement by building and transferring digital skills, expertise, and insights from our innovation ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, regardless of race, color, sex, gender identity, sexual orientation, religious practices, national origin, pregnancy, childbirth or related medical conditions, protected veteran status, or disability.

Preferred Skills:

Technology->Data Engineering->Databricks