Senior Data Engineer (Banking, 1-year renewable contract)

Dear Candidate,

If you or someone you know is interested in this opportunity, please forward the CV directly to [HIDDEN TEXT] (this method is preferred, as I may miss some applications due to the high volume).

Please be aware that visa sponsorship is currently not an option.

Primary Responsibilities

• Deploy and operationalize large-scale Lakehouse platforms, data products, and data marketplace functionalities.
• Design, implement, test, and maintain scalable data ingestion pipelines for batch, streaming, CDC, and APIs.
• Create multimodal data ingestion pipelines, which include:

o Extracting content from diverse file formats.

o Utilizing regex for field-specific extraction.

o Extracting content from embedded images.

o Capturing frames from video files.

o Extracting transcripts from audio files.

• Develop and sustain foundational and business data products, adhering to established data contracts, SLAs, data quality controls, and governance standards.
• Work with contemporary open table formats like Iceberg, Hudi, and Delta Lake.
• Create and support data pipelines for RAG, vector search, Generative AI (GenAI), and agentic AI initiatives.
• Conduct performance tuning, optimization, production support, troubleshooting, and root cause analysis across data platforms and pipelines.
• Establish frameworks for data ingestion, transformation, reconciliation, and data quality.
• Design data architectures that facilitate NLP and AI-driven analytics, ensuring proper ingestion, curation, governance, and management of both structured and unstructured data.
• Support machine learning platforms and workflows, including model development, deployment, and operationalization.
• Create internal engineering tools and full-stack applications using Python, shell scripting, and contemporary web frameworks.
• Expose and integrate data through APIs, event streams, dashboards, and business intelligence platforms.
• Generate and maintain technical documentation, deployment guides, and operational runbooks.
• Ensure adherence to engineering standards, DevSecOps controls, CI/CD practices, security requirements, and software delivery protocols.
• Collaborate proficiently with distributed engineering, data, technology, and business teams across various projects.
• Promote continuous enhancement in data platform reliability, scalability, automation, and operational excellence.

Essential Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 8-12 years of experience in Data Engineering, Big Data, Data Lake, Lakehouse, or extensive data platform implementations.
• Strong hands-on experience with enterprise data platforms such as Databricks, Snowflake, Cloudera, Azure, AWS, GCP, Huawei, or Alibaba Cloud.
• Practical experience in designing and developing data products and data marketplace functionalities.
• Expertise in Spark, PySpark, SQL, Python, and/or Scala, complemented by strong overall programming skills.
• Documented experience in building scalable data ingestion, transformation, streaming, CDC, reconciliation, and data quality pipelines.
• Hands-on familiarity with Iceberg, Hudi, Delta Lake, and object storage platforms.
• Experience with modern data technologies such as Kafka, Flink, Spark Streaming, Airflow, Trino, Dremio, Hive, and/or Impala.
• Strong practical experience with Kubernetes, OpenShift, Docker, Terraform, Jenkins, Git, CI/CD, MLflow, and observability tools.
• Experience designing data architectures and pipelines that support NLP, AI-driven analytics, RAG, vector search, GenAI, and agentic AI applications.
• Experience managing and governing unstructured and multimodal data, including documents, images, audio, and video.
• Familiarity with machine learning platforms and frameworks such as CML, Spark MLlib, scikit-learn, and XGBoost, including model deployment.
• Solid understanding of data modeling, metadata management, data lineage, data governance, and data quality.
• Experience in exposing data through APIs, event streams, dashboards, and business intelligence platforms.
• Experience with full-stack/internal engineering tool development using Python, shell scripting, Flask, React, or similar technologies is a plus.
• Knowledge or experience with Teradata, Netezza, Greenplum, or MPP migration projects is a plus.
• A strong engineering, automation, troubleshooting, and performance optimization mindset.
• Excellent communication and stakeholder management skills, with the ability to work effectively across distributed teams and multiple projects.
• Experience in Agile delivery environments and enterprise-scale technology platforms.
• A strong commitment to quality, operational excellence, automation, and continuous improvement.
• Relevant certifications such as Databricks Certified Data Engineer, Azure Data Engineer Associate, AWS Data Analytics Specialty, Google Professional Data Engineer, SnowPro, or DAMA CDMP would be advantageous.