Data Engineer (AWS + Databricks)

Essential Requirement - Data Engineer with expertise in AWS and Databricks.

Primary Responsibilities

• Develop, construct, and manage comprehensive data pipelines (ingestion → transformation → serving) utilizing Databricks and/or Snowflake, adhering to medallion/layered architecture models.
• Set up and manage workspaces/accounts, including compute policies, resource allocation, environment configuration, and access hierarchy in accordance with the SDP reference architecture.
• Establish data governance measures: catalogue and schema development, RBAC, row/column-level security, data masking, and lineage tracking.
• Implement CI/CD and infrastructure-as-code for deploying pipelines and promoting environments (dev → test → prod).
• Configure monitoring, telemetry, and audit logging to align with SDP's centralized observability and security requirements.
• Assist with UAT, integration testing, and parallel-run validation during migration and go-live phases.
• Create handover documentation (runbooks, access lists, escalation procedures) for operations teams within the agency.
• Collaborate closely with agency stakeholders and Principal (Databricks/Snowflake) solution architects throughout the project delivery process.

Required Technical Skills - Databricks

• Unity Catalog - design of catalogue/schema, access control, and data lineage management.
• Lakeflow / Delta Live Tables for orchestrating pipelines using Delta Lake table format.
• Administration of Databricks SQL and cluster/workspace including compute policies, pools, and cost management.
• Utilization of Databricks Asset Bundles (DABs) and Databricks Repos for CI/CD processes.
• Proficiency in PySpark / Spark SQL for extensive data transformation tasks.
• Familiarity with Databricks system tables (audit logs, billing/usage, query history) for enhanced observability.
• A minimum of 5 years of practical experience in Data Engineering, Data Platform Engineering, or a related field.
• A minimum of 3 years of practical experience with Databricks, focusing on data pipeline creation, platform management, governance, and optimization.

Required Technical Skills - General

• Proficient in SQL and Python (PySpark or general-purpose) for data engineering applications.
• Data modeling expertise - dimensional design, star/snowflake schemas, and semantic layers.
• Experience with CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, GitLab CI) for data engineering tasks.
• Infrastructure-as-code experience (preferably Terraform) for deploying cloud data platform resources.
• Hands-on experience with at least one major cloud provider - AWS, Azure, or Google Cloud.
• Knowledge of data security and compliance standards relevant to government/public-sector environments.

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional.
• SnowPro Core or SnowPro Advanced: Data Engineer certification.
• Previous experience in delivering projects within a government or regulated sector data platform, or familiarity with compliance frameworks like IM8 is a plus.
• Experience working as part of a System Integrator (SI) delivery team alongside a platform Principal is an additional advantage.