Backend Data Platform Engineer (SQL / Python /PySpark)

Role Overview

We are seeking an experienced Data Engineer to enhance the foundational elements of our enterprise data infrastructure. In this position, you will take charge of designing, optimizing, and securing scalable ETL pipelines, historical data frameworks, and analytical datasets, all while maintaining high standards of data quality and reconciliation throughout our ecosystem.

Key Responsibilities

• Pipeline Development: Create and sustain effective batch, streaming, CDC, and API-driven data pipelines, as well as historical snapshots and Slowly Changing Dimensions (SCD) frameworks.
• Data Transformation: Construct transformations for Enterprise Data Warehouses (EDW), data lakes, and semantic layers, including specialized datasets for payment harmonization.
• Performance Optimization: Enhance SQL queries and Spark jobs, implementing strict reconciliation controls, effective error handling, and operational oversight.
• Software Development Life Cycle (SDLC) Support: Assist with System Integration Testing (SIT), User Acceptance Testing (UAT), defect resolution, release management, and production validation in close partnership with architecture and BI teams.

Qualifications:

• A minimum of 6 years of relevant data engineering experience, with a proven track record of independently managing development and testing initiatives.
• Expertise in SQL, Python, Spark/PySpark, Teradata, and the Hadoop/Cloudera ecosystem.
• Familiarity with tools such as Informatica/Talend, Hive, Impala, Kafka, Airflow, APIs, and version control using Git.
• A solid foundation in enterprise data warehousing, data lakes, and banking data operations (preferably in Payments, Risk, Finance, or AML).
• A strong sense of ownership over deliverables, an analytical mindset, and a dedication to maintaining secure and high-quality engineering practices.