Azure Databricks

Responsibilities

• Design and improve scalable data engineering architectures on Azure Databricks to support advanced analytics and AI workloads for enterprise stakeholders and downstream applications
• Configure and administer Databricks Unity Catalog to enable centralized governance, robust access controls, and consistent data security policies across all workspaces and environments
• Create, automate, and monitor Databricks Workflows to orchestrate complex data pipelines, ensuring reliable scheduling, error handling, and recovery for mission-critical business processes
• Implement and maintain Databricks Delta Lake capabilities, including incremental ingestion, data versioning, and time travel to enhance data quality and analytical flexibility
• Develop optimized Databricks SQL queries, dashboards, and data models that produce actionable insights for business teams while sustaining strong performance on large datasets
• Partner with data scientists, analysts, and application teams to translate analytical needs into well structured data solutions that speed up experimentation and deployment
• Document data models, pipeline logic, governance rules, and operational procedures clearly and concisely to support transparency, maintainability, and knowledge sharing across teams
• Apply rigorous data quality validation, monitoring, and framework practices to confirm that curated datasets remain accurate, consistent, and suitable for AI and reporting use cases
• Collaborate with cloud infrastructure and security teams to ensure Databricks configurations, networking, and identity controls align with enterprise standards and regulatory requirements
• Optimize Azure Databricks cluster configuration, storage layout, and resource utilization to balance performance, reliability, and cost efficiency across production and development workloads
• Drive continuous improvement by assessing new Databricks capabilities and industry best practices, then leading adoption to strengthen the data platform’s capability and resilience
• Deliver domain focused data solutions that incorporate life and annuities insurance concepts when applicable, helping business teams better understand risk, policy behavior, and customer outcomes
• Promote responsible and ethical use of data by advancing governance practices that protect customer information, support regulatory compliance, and strengthen trust in data driven decision making

Qualifications

• Demonstrate strong hands on expertise in Azure Databricks, including consistent practical experience with Databricks Unity Catalog, Workflows, SQL, and Delta Lake in complex environments
• Provide proven experience designing and operating large scale data pipelines, with a minimum of nine years in data engineering or AI data specialist roles, plus exposure to modern cloud data stacks
• Show familiarity with life and annuities insurance data structures, processes, and metrics, noted as a beneficial domain skill that can enhance the value of analytics solutions for insurance stakeholders
• Exhibit advanced proficiency in structured query development, performance tuning, and data modeling practices that enable reporting, machine learning, and self service analytics
• Apply strong knowledge of data governance, security, and compliance concepts to configure Unity Catalog permissions, data classification, and audit readiness for enterprise use
• Demonstrate strong problem solving and debugging skills to investigate pipeline failures, data anomalies, and performance bottlenecks while implementing sustainable corrective actions
• Communicate effectively with technical and nontechnical partners, clearly explaining data solutions, trade offs, and risks to support informed decision making and collaboration
• Manage work effectively in an office based day shift model by coordinating with cross functional teams, aligning priorities, and ensuring consistent availability for stakeholder interactions