AI Engineering

• AI/ML Development: Create, implement, and roll out machine learning models, LLMs, and end-to-end AI workflows within the Databricks environment.
• Data Pipeline Engineering: Develop scalable, high-performance Delta Lake pipelines with PySpark and Python to ingest, clean, and preprocess large-scale datasets supporting AI initiatives.
• MLOps Implementation: Establish and oversee the full machine learning lifecycle using MLflow for model tracking, versioning, deployment, and monitoring.
• Optimization & Performance: Improve the efficiency of PySpark jobs, Databricks clusters, and Python code to achieve maximum performance while maintaining cost effectiveness.
• Collaboration & Integration: Work closely with Data Scientists, Data Engineers, and DevOps teams to embed AI models into core business applications and APIs.
• Governance & Security: Deploy Unity Catalog to enable data governance, access control, and lineage tracking across AI assets.

Responsibilities

• AI/ML Development: Create, implement, and roll out machine learning models, LLMs, and end-to-end AI workflows within the Databricks environment.
• Data Pipeline Engineering: Develop scalable, high-performance Delta Lake pipelines with PySpark and Python to ingest, clean, and preprocess large-scale datasets supporting AI initiatives.
• MLOps Implementation: Establish and oversee the full machine learning lifecycle using MLflow for model tracking, versioning, deployment, and monitoring.
• Optimization & Performance: Improve the efficiency of PySpark jobs, Databricks clusters, and Python code to achieve maximum performance while maintaining cost effectiveness.
• Collaboration & Integration: Work closely with Data Scientists, Data Engineers, and DevOps teams to embed AI models into core business applications and APIs.
• Governance & Security: Deploy Unity Catalog to enable data governance, access control, and lineage tracking across AI assets.