Lead Data Engineer (Teradata/Hadoop)

Position Overview

We are in search of a Senior Data Engineer with extensive knowledge in enterprise data platforms, particularly within the banking sector and large-scale data management. This position involves spearheading the design and development of efficient data pipelines across Teradata, Hadoop/Cloudera, and various cloud environments, while also mentoring engineering professionals and fostering a culture of technical excellence.

Your Responsibilities

• Architecture & Development: Take charge of designing and deploying scalable batch, streaming, and API-driven data pipelines, along with frameworks for historical data processing, SCD/CDC mechanisms, and semantic-layer data products.
• Performance & Optimisation: Enhance complex SQL queries, Spark workloads, partition strategies, and storage frameworks to efficiently handle large volumes of data in regulated banking environments.
• Infrastructure & Automation: Develop infrastructure-as-code using Terraform and create automated CI/CD pipelines, containerized deployments, and stringent testing protocols.
• Leadership & Collaboration: Conduct code reviews, uphold engineering standards, mentor junior engineers, and collaborate closely with architects, product teams, DevOps, and business stakeholders.

Qualifications for Success

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field
• A minimum of 10 years of data engineering experience, demonstrating technical leadership in Teradata and Hadoop/big-data solutions.
• Preferred Certifications: Teradata, Cloudera, Databricks, Cloud Data Engineering, or Terraform
• Core Technologies: Proficiency in Teradata, Hadoop/Cloudera, Spark, Hive, Impala, HBase, SQL, Python, Scala/Java, Kafka, Airflow, and Denodo.
• Infrastructure & DevOps: Experience with Terraform, Kubernetes/OpenShift, Git, and CI/CD automation tools.
• Domain Knowledge: In-depth understanding of enterprise banking data, including Customer, Payments, Risk, Finance, AML, and Regulatory Reporting, with a strong emphasis on data quality, reconciliation, lineage, and historical reconstruction.
• Core Competencies: Proven technical leadership, disciplined delivery in regulated environments, outstanding communication skills, and a strong sense of accountability for production-ready results.