Azure Administration

Job Overview:

• We seek a skilled Azure Databricks Administrator to oversee, manage, and enhance Azure Databricks platforms within a comprehensive Azure ecosystem.
• This position is essential for maintaining high availability, security, performance, and cost efficiency of Azure Databricks environments while facilitating data engineering, analytics, and AI/ML workloads.
• The perfect candidate will possess extensive hands-on expertise with Azure Databricks, Azure infrastructure, security, and governance, collaborating closely with cloud data and business teams to provide a reliable and scalable data platform.

Core Responsibilities:

• Administer and oversee Azure Databricks workspaces across Development, QA, and Production settings.
• Configure and manage interactive job clusters, including autoscaling, auto-termination, cluster pools, and associated policies.
• Conduct environment setup, workspace configuration, and platform upgrades.
• Ensure operational stability and high availability of Azure Databricks services.
• Integrate Azure Databricks with Azure Data Lake Storage Gen2, Azure SQL Synapse, Event Hub, and other Azure services.
• Manage VNet-injected Databricks workspaces, private endpoints, and network security settings.
• Collaborate with Azure networking teams to guarantee secure and optimized connectivity.
• Administer users, groups, and access controls utilizing Azure Active Directory (Entra ID).
• Implement data governance measures through Unity Catalog, including catalog schema, table, and column-level security.
• Configure secure authentication methods using AAD SSO, Managed Identities, and Service Principals.
• Manage secrets and credentials through Azure Key Vault integration.
• Ensure conformity with enterprise security audits and regulatory standards.
• Monitor Azure Databricks usage and implement strategies for cost optimization, including:
• o Cluster right-sizing
• o Autoscaling and auto-termination
• o Instance pool configuration
• Analyze workload patterns and suggest performance tuning best practices.
• Provide usage and cost analytics to stakeholders and FinOps teams.
• Deploy and manage Azure Databricks infrastructure employing Infrastructure as Code tools such as Terraform or ARM templates.
• Support CI/CD pipelines for Databricks notebooks, jobs, libraries, and workspace artifacts using Azure DevOps or GitHub.
• Automate operational processes to minimize manual efforts and enhance reliability.
• Establish monitoring and alerting mechanisms using Azure Monitor, Log Analytics, and Databricks metrics.
• Diagnose cluster failures, job issues, workspace access challenges, and performance bottlenecks.
• Conduct root cause analyses and deploy preventive solutions.
• Provide Level 2 and Level 3 support within a 24/7 operational framework.
• Create and maintain documentation, runbooks, SOPs, and architecture diagrams for the platform.
• Collaborate with data engineers, data scientists, cloud engineers, and security teams.
• Engage in platform reviews and continuous improvement initiatives.

Technical Qualifications:

• Essential Technical Skills
• 2 years of experience in data platform or cloud administration, emphasizing Azure Databricks.
• Proficient in Azure services, including ADLS Gen2, Azure Networking, Azure Monitor, and Azure AD.
• Robust understanding of Apache Spark architecture and performance optimization.
• Experience in managing secure, enterprise-scale Azure Databricks environments.
• Proficient in scripting languages such as Python, PowerShell, or Bash.
• Hands-on experience with Terraform or ARM templates.
• Desirable Additional Skills
• Familiarity with Unity Catalog, data lineage, and governance frameworks.
• Knowledge of Delta Lake, job orchestration, and workload isolation.
• Experience with CI/CD pipelines utilizing Azure DevOps or GitHub Actions.
• Comprehension of FinOps practices within Azure.
• Certifications
• o Azure Databricks Certified Associate Professional
• o Microsoft Azure Administrator Solutions Architect
• Interpersonal Skills
• Strong analytical and problem-solving abilities.
• Excellent communication skills for interacting with both technical and business stakeholders.
• Capability to work autonomously in a dynamic, 24/7 support environment.
• Detail-oriented, proactive, and focused on operational excellence.
• Additional Preferences
• Experience in supporting large enterprise or multi-workspace Azure Databricks platforms.
• Exposure to onsite-offshore delivery models and global teams.
• Background in regulated environments such as Healthcare.

Company Overview:

• Infosys is a global frontrunner in next-generation digital services and consulting.
• We assist clients in over 50 countries in navigating their digital transformation journeys.
• With more than four decades of experience in managing global enterprise systems, we expertly guide our clients through their digital evolution.
• We achieve this by equipping enterprises with an AI-powered core that prioritizes change execution.
• We also empower businesses with agile digital solutions at scale, delivering unmatched performance and customer satisfaction.
• Our continuous learning agenda promotes improvement by building and sharing digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, irrespective of race, color, sex, gender identity, sexual orientation, religious practices, national origin, pregnancy, childbirth, related medical conditions, protected veteran status, disability, or family relationships with a protected veteran.

Desired Skills:

Technology->Infrastructure-Server Administration->Cloud Infrastructure->Azure Administration