Data Engineer (Databricks & DevOps)

We are hiring a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. In this role, you will apply data engineering expertise alongside DevOps practices—such as CI/CD, infrastructure as code, and monitoring—to deliver dependable, automated, and secure data platforms for analytics, reporting, and machine learning.

Key Responsibilities

• Design and develop batch and streaming data pipelines using Databricks (PySpark, Spark SQL, Delta Lake) on AWS
• Create and improve ETL/ELT workflows to ingest data from databases, APIs, files, and event streams
• Develop lakehouse architecture with Delta Lake, Unity Catalog, and the medallion model (bronze/silver/gold)
• Coordinate and orchestrate workflows with Databricks Workflows, Airflow, or AWS Step Functions
• Set up and manage cloud infrastructure using Terraform or CloudFormation
• Develop and maintain CI/CD pipelines for data code and infrastructure (GitHub Actions, GitLab CI, Jenkins, or Azure DevOps)
• Establish monitoring, logging, and alerting for pipelines and platform health (CloudWatch, Datadog, etc.)
• Improve Spark job performance and tune cluster configurations for efficiency and cost
• Maintain data quality, governance, security, and access control standards
• Work with data scientists, analysts, and business stakeholders to convert requirements into technical implementations
• Produce documentation for pipelines, architecture, and operational runbooks

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field
• Proven practical experience with Databricks (PySpark, Delta Lake, Jobs/Workflows, cluster management)
• Strong experience with AWS services: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch
• Proficiency in Python and SQL
• Experience with DevOps tools and practices: Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred)
• Knowledge of data modeling, data warehousing, and lakehouse concepts
• Experience with workflow orchestration tools (Airflow, Databricks Workflows)
• Familiarity with Agile/Scrum delivery

Preferred Qualifications

• Databricks Certified Data Engineer (Associate/Professional)
• AWS certifications (Solutions Architect, Data Engineer, or DevOps Engineer)
• Experience with Unity Catalog, DLT (Delta Live Tables), and Auto Loader
• Knowledge of Kubernetes, Kafka, or dbt
• Experience with data governance, lineage, and compliance (GDPR, PDPA, etc.)
• Experience supporting MLOps or ML workloads
• Scripting skills in Bash or PowerShell