Databricks Administration

Job Overview:

• We are looking for a proactive Databricks Administrator to oversee and enhance the Databricks platform for enterprise-level data and analytics.
• This position is responsible for managing platform governance, security protocols, automation strategies, compliance, reliability, and cost efficiency, while establishing standards to facilitate large-scale analytics and AI/ML workloads.

Primary Responsibilities:

• Possess 9 years of experience in data platform engineering with a specific focus on Databricks Spark environments.
• 9 years of experience in the administration of enterprise-grade AI LLM Agentic platforms.
• Databricks Proficiency: Extensive hands-on experience with Databricks Lakehouse architecture, Unity Catalog, Delta Live Tables (DLT), and Serverless SQL.
• Technical Skills: Strong proficiency in Python, PySpark, and Spark SQL.
• Cloud Computing: In-depth knowledge of one or more leading cloud service providers, including AWS, Azure, or GCP, along with their associated services.
• Manage the administration of Databricks accounts and multi-workspace environments throughout the software development lifecycle (SDLC).
• Establish and uphold platform standards for clusters, jobs, SQL warehouses, runtimes, and workspace configurations.
• Develop and implement a governance strategy for Unity Catalog, encompassing catalogs, schemas, permissions, and external locations.
• Design and manage enterprise monitoring, alerting systems, compliance dashboards, and operational service-level objectives (SLOs).
• Lead management of high-severity incidents, conducting root cause analysis and preventive measures.
• Implement and maintain security audits and regulatory compliance controls.
• Drive cost governance, capacity forecasting, and performance optimization across various workloads.
• Define and steer Infrastructure as Code and automation strategies using Terraform, APIs, and CI/CD pipelines.
• Standardize CI/CD processes for Databricks Asset Bundles and DevOps operational models.
• Collaborate with cloud platform security and data engineering teams on architectural and governance decisions.
• Oversee platform documentation, onboarding standards, and the roadmap for operational maturity.
• Guide and mentor junior and mid-level Databricks administrators, serving as the senior technical escalation point.
• Administer Databricks accounts and workspaces across development, testing, and production environments.
• Configure and manage clusters, jobs, SQL warehouses, runtimes, repositories, and workspace settings.
• Implement monitoring, alerting systems, dashboards, and operational runbooks.
• Provide Tier 2 and Tier 3 platform support, including incident resolution and root cause analysis.
• Enforce security and governance protocols using Unity Catalog, RBAC, service principals, and least privilege access principles.
• Manage secrets, audit logging, and compliance reporting.
• Administer Unity Catalog objects, external locations, and cloud storage integrations.

Technical Qualifications:

• 9 to 11 years of experience in Databricks Administration and Big Data Platforms.
• Strong hands-on experience with Databricks Workspace, including clusters, jobs, and notebooks.
• Familiarity with Apache Spark, PySpark, Scala, and distributed data processing.
• Knowledge of cloud platforms, with a preference for Azure; AWS and GCP are also acceptable.
• Experience with data storage solutions such as ADLS, S3, and Blob Storage.
• Solid understanding of data engineering concepts, including ETL and ELT pipelines.
• Awareness of security best practices, including IAM roles, data encryption, and private endpoints.
• Proficiency in SQL and scripting languages such as Python and Shell.
• Experience in performance tuning and cost optimization for clusters and workloads.

Additional Information:

• Infosys stands as a global leader in next-generation digital services and consulting.
• We assist clients across more than 50 countries in navigating their digital transformation journeys.
• With over four decades of experience managing global enterprise systems, we expertly guide our clients through their digital evolution.
• Our approach includes empowering enterprises with an AI-driven core that prioritizes change execution.
• We also enable businesses to adopt agile digital practices at scale, resulting in exceptional performance and customer satisfaction.
• Our continuous learning agenda drives ongoing improvement through the development and transfer of digital skills, expertise, and innovations from our ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, without discrimination based on race, color, sex, gender identity, sexual orientation, religious practices and observances, national origin, pregnancy-related conditions, status as a protected veteran, or disability.

Desired Skills:

Technology -> Data Engineering -> Databricks