Data Platform Engineering Lead - AITDS

Key Responsibilities

1. Build & Deploy Data Platform Solutions

· Create and deliver data pipelines for both batch and streaming workloads using Python and Spark/PySpark

· Design, build, and maintain AI-ready datasets, feature pipelines, and the underlying training data foundations

· Develop backend services and APIs (Python – FastAPI or equivalent) to make data products accessible

· Implement microservices and event-driven architectures to support data ingestion, processing, and data serving

· Maintain platform scalability, performance, and long-term maintainability

· Guide and mentor engineers on data engineering patterns and best practices.

2. Enable AI/ML & LLM Integration

· Build AI-ready pipelines that support model training, evaluation, and inference

· Provide feature store functionality and ensure reproducible datasets

· Support integration with LLM/AI services (RAG, embeddings, inference APIs)

· Enable data-to-AI workflows, including vectorization and retrieval processes.

3. Oversee Implementation on Cloud Infrastructure

· Partner with infrastructure teams across Azure, AWS, or GCP

· Build and integrate data lakes, lake-houses, and SQL/NoSQL systems

· Facilitate integration between data platforms and AI/ML systems

· Implement containerized and serverless architectures.

4. Implement Modern Software Engineering Practices

· Establish CI/CD pipelines and observability practices

· Define and enforce data quality frameworks

· Support metadata management, data lineage, and governance

· Improve platform performance, reliability, and scalability.

Required Capabilities / Skills / Experience

· 8+ years in data engineering and backend development

· Strong Python, APIs, and microservices experience

· Deep experience in Spark/PySpark or equivalent

· Expertise in ETL/ELT pipelines and lakehouse architectures

· Experience with feature stores and AI-ready datasets

· Experience with Azure/AWS/GCP

· Familiarity with Docker, Kubernetes, CI/CD

· Familiarity with LLM/GenAI integration patterns

· Familiarity with metadata and governance tooling

· Strong system design and problem-solving skills.