Databricks Data Engineer

Essential Duties

• Develop, construct, and maintain comprehensive data pipelines (ingestion → transformation → serving) utilizing Databricks and/or Snowflake, adhering to medallion/layered architectural frameworks
• Manage and configure workspaces/accounts, including compute policies, resource allocation, environment configuration, and access hierarchy, in alignment with the SDP reference architecture
• Establish data governance protocols: catalogue and schema design, role-based access control (RBAC), row/column-level security, data masking, and lineage tracking
• Implement CI/CD processes and infrastructure-as-code for the deployment of pipelines and transition of environments (development → testing → production)
• Set up monitoring, telemetry, and audit logging to fulfill SDP's central observability and security requirements
• Assist in user acceptance testing (UAT), integration testing, and parallel-run validation during migration and implementation phases
• Generate handover documentation (including runbooks, access lists, and escalation procedures) for operational teams
• Collaborate directly with clients, agency stakeholders, and Principal (Databricks/Snowflake) solution architects throughout the delivery process

Required Technical Expertise - Databricks

• Unity Catalog - design of catalog/schema, access control, and data lineage management
• Lakeflow / Delta Live Tables for orchestration of pipelines utilizing Delta Lake table format
• Administration of Databricks SQL and cluster/workspace (compute policies, pools, cost management)
• Utilization of Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD processes
• Proficiency in PySpark / Spark SQL for extensive data transformation tasks
• Familiarity with Databricks system tables (audit logs, billing/usage, query history) for enhanced observability
• A minimum of 5 years of practical experience in Data Engineering, Data Platform Engineering, or related fields
• At least 3 years of practical experience with Databricks focused on data pipeline development, platform administration, governance, and performance optimization

Required Technical Skills

• Proficient in SQL and Python (PySpark or general-purpose) for data engineering tasks
• Expertise in data modeling - dimensional design, star/snowflake schemas, and semantic layers
• Experience with CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) tailored for data engineering workflows
• Hands-on experience with infrastructure-as-code (preferably Terraform) for provisioning cloud data platform resources
• Practical experience with at least one hyperscaler - AWS, Azure, or Google Cloud
• Knowledge of data security and compliance standards relevant to government/public sector environments

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional
• SnowPro Core or SnowPro Advanced: Data Engineer certifications
• Previous experience in delivering solutions for government or regulated-sector data platforms, or familiarity with compliance frameworks such as IM8 is advantageous
• Experience as part of a System Integrator (SI) delivery team alongside a platform Principal is a plus