Data Lakehouse Architecture (Banking, 1-year renewable contract)

Dear Candidate,

If you or someone you know may be interested in this opportunity, please submit your CV directly to [HIDDEN TEXT] (this is preferred, as I may miss some applications due to the high volume).

Please be aware that visa sponsorship is currently unavailable.

Primary Responsibilities

• Lead the comprehensive architecture and technical vision of an enterprise Lakehouse platform.
• Create and implement scalable data products, a data marketplace, knowledge layers, and platforms that support agentic workloads.
• Establish target architectures for applications and platforms, prioritizing reusability, scalability, performance, security, and operational efficiency.
• Develop and maintain technical roadmaps and architectural strategies for the Lakehouse platform.
• Set up technical frameworks and reusable patterns to enhance the operationalization of:

- Unstructured and multimodal content extraction.

- Lambda architecture and deployment methodologies.

- Retrieval-Augmented Generation (RAG) and retrieval-augmented data patterns.

- Vector and graph-based data capabilities.

- Agentic workloads and AI-driven data solutions.

• Design and implement large-scale distributed and MPP compute workloads across on-premise, hybrid, and cloud environments.
• Architect and optimize Lakehouse platforms using open table formats, object storage, data federation, and multimodal query engines.
• Design hybrid and cloud architectures utilizing private connectivity, workload placement strategies, Infrastructure-as-Code, and cloud cost optimization.
• Create data contracts, SLAs, data quality standards, and governance protocols for foundational and business data products in collaboration with business stakeholders.
• Facilitate the consumption of data products by downstream applications through APIs, publish-subscribe mechanisms, generative BI, real-time dashboards, and data marketplaces.
• Assist in the architecture and implementation of RAG, embedding strategies, vector databases, graph databases, prompt engineering, and context management for agentic workloads.
• Ensure technical quality assurance and confirm that delivery adheres to established software development methodologies, engineering standards, and technology practices.
• Review design specifications and technical deliverables produced by development teams.
• Develop and maintain functional and non-functional specifications, architecture/design documents, deployment guides, and training resources.
• Independently install, customize, configure, and integrate software packages and technology solutions.
• Engage in RFPs, proof-of-concepts (POCs), and technology/product selection processes.
• Lead performance engineering, capacity planning, tuning, and optimization of data platforms and workloads.
• Collaborate with business stakeholders, technology teams, vendors, and other technology functions to design and deliver enterprise solutions.
• Support continuous service improvement, process enhancement, and initiatives focused on operational excellence.
• Ensure smooth integration with DevOps, CI/CD, monitoring, testing, and engineering toolchains.
• Offer technical guidance and mentorship while upholding high standards of quality across architecture and engineering deliveries.

Essential Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or equivalent experience.
• 10-15 years of experience in Data Architecture, Big Data, Data Engineering, Data Lake, or Lakehouse implementations.
• Extensive experience in designing and implementing enterprise-scale Data Lakehouse platforms, preferably in the financial services sector.
• Hands-on experience with one or more major data/cloud platforms, such as Databricks, Snowflake, Cloudera, AWS, Azure, GCP, Huawei Cloud, or Alibaba Cloud.
• Strong experience with large-scale Lakehouse architecture, implementation, performance optimization, and distributed computing.
• In-depth knowledge of open table formats like Apache Iceberg, Apache Hudi, and Delta Lake.
• Significant experience with object storage architecture, including strategies for hot, warm, and cold data tiering.
• Experience with data federation technologies such as Trino, Denodo, and Dremio.
• Experience with multimodal/distributed query engines like Hive, Impala, Apache Kudu, or equivalent technologies.
• Proven track record in designing MPP and distributed compute workloads across on-premise, hybrid, and cloud environments.
• Strong understanding of hybrid and cloud architecture, including private connectivity technologies like Direct Connect and ExpressRoute.
• Experience with workload placement, cloud architecture, egress cost optimization, and Infrastructure-as-Code.
• Robust experience in building and serving foundational and business data products through APIs, publish-subscribe/event-driven architectures, real-time dashboards, BI platforms, and data marketplaces.
• Experience supporting AI and agentic workloads, including:

- Retrieval-Augmented Generation (RAG).

- Embedding strategies.

- Vector databases.

- Graph databases.

- Prompt engineering.

- Context management.

- Agentic orchestration and workflows.

• Strong knowledge of contemporary vector database technologies, such as Databricks Vector Search, Azure AI Search, Pinecone, ChromaDB, Weaviate, or Snowflake Cortex.
• Familiarity with graph databases such as Neo4j, JanusGraph, TigerGraph, Cosmos DB, Amazon Neptune, or equivalent.