Azure Administration

Job Overview:

• We are seeking a skilled Azure Databricks Administrator to oversee, operate, and enhance Azure Databricks platforms within a comprehensive Azure ecosystem.
• This position is crucial for maintaining high availability, security, performance, and cost efficiency of Azure Databricks environments while facilitating data engineering, analytics, and AI/ML workloads.
• The preferred candidate will possess robust hands-on experience with Azure Databricks, Azure infrastructure, security, and governance, collaborating closely with cloud data and business teams to provide a reliable and scalable data platform.

Primary Responsibilities:

• Administer and oversee Azure Databricks workspaces across Development, QA, and Production environments.
• Configure and manage interactive and job clusters, including autoscaling, auto termination, cluster pools, and policies.
• Conduct environment setups, workspace configurations, and platform upgrades.
• Ensure operational stability and high availability of Azure Databricks services.
• Integrate Azure Databricks with Azure Data Lake Storage Gen2 (ADLS), Azure SQL, Synapse, Event Hub, and other Azure services.
• Manage VNet injected Databricks workspaces, private endpoints, and network security settings.
• Collaborate with Azure networking teams to guarantee secure and optimized connectivity.
• Manage users, groups, and access controls utilizing Azure Active Directory (Entra ID).
• Implement data governance strategies using Unity Catalog, which includes catalog schema, table, and column-level security.
• Configure secure authentication methods using AAD SSO, Managed Identities, and Service Principals.
• Manage secrets and credentials through Azure Key Vault integration.
• Ensure compliance with enterprise security audits and regulatory standards.
• Monitor Azure Databricks usage and execute cost optimization strategies, including:
• o Cluster right sizing
• o Autoscaling and auto termination
• o Instance pool configuration
• Analyze workload patterns and suggest performance tuning best practices.
• Provide usage and cost reports to stakeholders and FinOps teams.
• Deploy and manage Azure Databricks infrastructure utilizing Infrastructure as Code tools such as Terraform or ARM templates.
• Support CI/CD pipelines for Databricks notebooks, jobs, libraries, and workspace artifacts using Azure DevOps or GitHub.
• Automate operational tasks to decrease manual workload and enhance reliability.
• Establish monitoring and alerting systems using Azure Monitor, Log Analytics, and Databricks metrics.
• Troubleshoot cluster failures, job issues, workspace access problems, and performance bottlenecks.
• Conduct root cause analysis and implement preventive strategies.
• Provide Level 2 and Level 3 support within a 24/7 operational framework.
• Create and maintain platform documentation, runbooks, SOPs, and architecture diagrams.
• Collaborate with data engineers, data scientists, cloud engineers, and security teams.
• Engage in platform reviews and continuous improvement initiatives.

Technical Qualifications:

• Essential Technical Skills
• 7 years of experience in cloud administration or data platform management with a strong emphasis on Azure Databricks.
• Hands-on experience with Azure services, including ADLS Gen2, Azure Networking, Azure Monitor, and Azure AD.
• Deep understanding of Apache Spark architecture and performance optimization techniques.
• Experience managing secure, enterprise-scale Azure Databricks environments.
• Proficient in Python, PowerShell, or Bash scripting.
• Hands-on experience with Terraform or ARM templates.
• Desirable Skills
• Experience with Unity Catalog, data lineage, and governance frameworks.
• Familiarity with Delta Lake, job orchestration, and workload isolation.
• Experience with CI/CD pipelines utilizing Azure DevOps or GitHub Actions.
• Understanding of FinOps practices within Azure.
• Certifications
• o Azure Databricks Certified Associate Professional
• o Microsoft Azure Administrator Solutions Architect
• Interpersonal Skills
• Strong analytical and troubleshooting skills.
• Exceptional communication abilities with both technical and business stakeholders.
• Capacity to work autonomously in a dynamic 24/7 support environment.
• Proactive, detail-oriented, and committed to operational excellence.
• Additional Assets
• Experience supporting large enterprise or multi-workspace Azure Databricks platforms.
• Exposure to onsite/offshore delivery models and global teams.
• Experience in regulated environments, such as Healthcare.

Additional Duties:

• Infosys is a leading global provider of next-generation digital services and consulting.
• We assist clients in over 50 countries in navigating their digital transformation journeys.
• With more than four decades of experience managing global enterprises' systems and operations, we expertly guide clients through their digital evolution.
• We achieve this by empowering enterprises with an AI-driven core that helps prioritize change execution.
• Additionally, we enable businesses with agile digital solutions at scale to achieve unprecedented levels of performance and customer satisfaction.
• Our continuous learning agenda promotes ongoing improvement by building and transferring digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, regardless of race, color, gender identity, sexual orientation, religious beliefs, national origin, pregnancy, childbirth, or related medical conditions, status as a protected veteran or spouse, family member of a protected veteran, or disability.

Preferred Expertise:

Technology->Infrastructure-Server Administration->Cloud Infrastructure->Azure Administration