Databricks Administration

Job Overview:

• We are in search of a proactive Databricks Administrator to oversee and enhance the Databricks platform dedicated to enterprise data and analytics.
• This position is responsible for leading platform governance, security posture, automation strategy, compliance, reliability, and cost management, while establishing standards to facilitate large-scale analytics and AI/ML workloads.

Primary Responsibilities:

• Minimum of 2 years of experience in data platform engineering, with a strong focus on Databricks Spark environments.
• 2 years of experience in administering enterprise-grade AI LLM Agentic platforms.
• Databricks Expertise: Extensive hands-on experience with Databricks Lakehouse architecture, Unity Catalog, Delta Live Tables (DLT), and Serverless SQL.
• Technical Proficiency: Skilled in Python, PySpark, and Spark SQL.
• Cloud Platforms: In-depth knowledge of one or more major cloud providers (AWS, Azure, GCP) and their associated services.
• Manage the administration of Databricks accounts and multi-workspace environments throughout the software development lifecycle (SDLC).
• Establish and enforce platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations.
• Lead the governance strategy for Unity Catalog, including catalogs, schemas, permissions, and external locations.
• Design and oversee enterprise-level monitoring, alerting, compliance dashboards, and operational service level objectives (SLOs).
• Manage high-severity incident response, root cause analysis, and preventive measures.
• Establish and maintain security audits and regulatory compliance controls.
• Drive cost governance, capacity forecasting, and performance optimization across various workloads.
• Define and lead Infrastructure as Code strategies and automation using Terraform, APIs, and CI/CD pipelines.
• Standardize CI/CD practices for Databricks Asset Bundles and DevOps operating models.
• Collaborate with cloud platform security and data engineering teams on architecture and governance decisions.
• Maintain platform documentation, onboarding standards, and the operational maturity roadmap.
• Mentor and coach junior and mid-level Databricks administrators, serving as a senior technical escalation point.
• Administer Databricks accounts and workspaces across development, testing, and production environments.
• Configure and manage clusters, jobs, SQL warehouses, runtimes, repositories, and workspace settings.
• Implement monitoring, alerting, dashboards, and operational runbooks.
• Provide Tier 2 and 3 platform support, incident resolution, and root cause analysis.
• Enforce security and governance measures utilizing Unity Catalog, RBAC, service principals, and least privilege access principles.
• Manage secrets, audit logging, and compliance reporting.
• Administer Unity Catalog objects, external locations, and cloud storage integrations.

Technical Qualifications:

• 2 to 5 years of experience in Databricks Administration and Big Data Platforms.
• Extensive hands-on experience with Databricks Workspace, Clusters, Jobs, and Notebooks.
• Experience with Apache Spark, PySpark, Scala, and distributed data processing.
• Familiarity with cloud platforms, with a preference for Azure; AWS and GCP are also acceptable.
• Experience with data storage systems such as ADLS, S3, and Blob Storage.
• Solid understanding of data engineering concepts, including ETL and ELT pipelines.
• Knowledge of security best practices, including IAM roles, data encryption, and private endpoints.
• Proficiency in SQL and scripting languages (Python, Shell, etc.).
• Experience in performance tuning and cost optimization for clusters and workloads.

Additional Information:

• Infosys stands as a global leader in cutting-edge digital services and consulting.
• We assist clients in over 50 countries in navigating their digital transformation journeys.
• With more than four decades of experience managing the operations of global enterprises, we expertly guide our clients through their digital transitions.
• We achieve this by empowering enterprises with an AI-driven core that prioritizes change execution.
• Additionally, we enable businesses with agile digital solutions at scale to deliver exceptional performance and customer satisfaction.
• Our commitment to continuous improvement is fueled by an always-on learning agenda, fostering the development and transfer of digital skills, expertise, and innovative ideas.
• Infosys is an equal opportunity employer, providing employment opportunities to applicants and employees without discrimination based on race, color, sex, gender identity, sexual orientation, religious practices, national origin, pregnancy, childbirth, or related medical conditions, status as a protected veteran or the family member of a protected veteran, or disability.

Preferred Expertise:

Technology->Data Engineering->Databricks