Azure Administration

Job Description:

• We are seeking a skilled Azure Databricks Administrator to oversee the management, operation, and optimization of Azure Databricks platforms within an enterprise Azure environment.
• This position is tasked with ensuring the high availability, security, performance, and cost-effectiveness of Azure Databricks environments while supporting data engineering, analytics, and AI/ML workloads.
• The successful candidate will possess extensive hands-on experience with Azure Databricks, Azure infrastructure, security, and governance, collaborating closely with cloud data and business teams to provide a stable and scalable data platform.

Key Responsibilities:

• Administer and oversee Azure Databricks workspaces across Development, QA, and Production environments.
• Configure and manage interactive and job clusters, including autoscaling, auto termination, cluster pools, and policies.
• Conduct environment setup, workspace configuration, and platform upgrades.
• Ensure high availability and operational stability of Azure Databricks services.
• Integrate Azure Databricks with Azure Data Lake Storage Gen2 (ADLS), Azure SQL Synapse, Event Hub, and other Azure services.
• Manage VNet-injected Databricks workspaces, private endpoints, and network security configurations.
• Collaborate with Azure networking teams to maintain secure and optimized connectivity.
• Administer users, groups, and access controls using Azure Active Directory (Entra ID).
• Implement data governance utilizing Unity Catalog, including catalog schema, table, and column-level security.
• Configure secure authentication mechanisms using AAD SSO, Managed Identities, and Service Principals.
• Manage secrets and credentials through Azure Key Vault integration.
• Ensure compliance with enterprise security audits and regulatory standards.
• Monitor Azure Databricks usage and execute cost optimization strategies, including:
• o Cluster right-sizing
• o Autoscaling and auto termination
• o Instance pool configuration
• Analyze workload patterns and advise on performance tuning best practices.
• Provide usage and cost reports to stakeholders and FinOps teams.
• Deploy and manage Azure Databricks infrastructure utilizing Infrastructure as Code tools such as Terraform or ARM templates.
• Support CI/CD pipelines for Databricks notebooks, jobs, libraries, and workspace artifacts using Azure DevOps or GitHub.
• Automate operational tasks to enhance reliability and minimize manual efforts.
• Establish monitoring and alerting protocols using Azure Monitor, Log Analytics, and Databricks metrics.
• Troubleshoot cluster failures, job issues, workspace access problems, and performance bottlenecks.
• Conduct root cause analysis and implement preventive measures.
• Provide Level 2 and Level 3 support within a 24/7 operational framework.
• Create and maintain platform documentation, runbooks, SOPs, and architecture diagrams.
• Collaborate with data engineers, data scientists, cloud engineers, and security teams.
• Engage in platform reviews and continuous improvement initiatives.

Technical Requirements:

• Mandatory Technical Skills
• 5 years of experience in data platform or cloud administration with a strong emphasis on Azure Databricks.
• Hands-on experience with Azure services including ADLS Gen2, Azure Networking, Azure Monitor, and Azure AD.
• Strong knowledge of Apache Spark architecture and performance optimization.
• Experience managing secure, enterprise-scale Azure Databricks environments.
• Proficiency in Python, PowerShell, or Bash scripting.
• Hands-on experience with Terraform or ARM templates.
• Preferred Good to Have
• Experience with Unity Catalog, data lineage, and governance frameworks.
• Familiarity with Delta Lake, job orchestration, and workload isolation.
• Experience with CI/CD pipelines using Azure DevOps or GitHub Actions.
• Understanding of FinOps practices within Azure.
• Certifications
• o Azure Databricks Certified Associate Professional
• o Microsoft Azure Administrator Solutions Architect
• Soft Skills
• Strong analytical and troubleshooting abilities.
• Excellent communication skills with both technical and business stakeholders.
• Capability to work autonomously in a fast-paced 24/7 support environment.
• Proactive, detail-oriented, and focused on operational excellence.
• Nice to Have
• Experience supporting large enterprise or multi-workspace Azure Databricks platforms.
• Exposure to onsite/offshore delivery models and global teams.
• Experience in regulated environments, such as Healthcare.

Additional Responsibilities:

• Infosys is a leading global player in next-generation digital services and consulting.
• We assist clients across more than 50 countries in navigating their digital transformation journeys.
• With over 40 years of experience managing the systems and processes of global enterprises, we expertly guide our clients through their digital evolution.
• We achieve this by equipping the enterprise with an AI-powered core that prioritizes change execution.
• We empower businesses with agile digital solutions at scale to achieve unprecedented levels of performance and customer satisfaction.
• Our commitment to continuous learning drives ongoing improvement through the development and sharing of digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is an equal opportunity employer, providing fair employment opportunities to applicants and employees without regard to race, color, sex, gender identity, sexual orientation, religious beliefs, national origin, pregnancy, childbirth or related medical conditions, status as a protected veteran or a family member of a protected veteran, or disability.

Preferred Skills:

Technology->Infrastructure-Server Administration->Cloud Infrastructure->Azure Administration