AI Engineer (Gen AI, Databricks)

Job Overview:

• We are seeking a GenAI Databricks Engineer who possesses practical experience in Data Engineering and Generative AI solutions.
• The ideal applicant should demonstrate proficiency in constructing scalable data pipelines utilizing Databricks Python and cloud computing platforms, while facilitating AI LLM-based applications for business purposes.

Core Responsibilities:

• Core Responsibilities
• Design, develop, and sustain scalable data pipelines utilizing Databricks.
• Create and enhance ETL/ELT workflows for both structured and unstructured data.
• Develop and deploy Generative AI solutions leveraging LLMs.
• Implement RAG (Retrieval-Augmented Generation) architectures and work with vector databases.
• Handle large-scale datasets using Spark and PySpark.
• Integrate AI solutions with cloud services and enterprise applications.
• Collaborate with Data Scientists, Data Engineers, and business stakeholders.
• Monitor, troubleshoot, and enhance data and AI workloads.
• Ensure adherence to data quality, governance, and security protocols.
• Essential Skills
• 2 to 5 years of experience in Data Engineering and/or Generative AI.
• Proficient in Databricks.
• Experienced with PySpark and Apache Spark.
• Strong command of Python.
• Proficient in SQL.
• Data Pipeline Development expertise.
• ETL/ELT Development skills.
• Familiarity with Delta Lake.
• Understanding of Data Warehousing concepts.
• Knowledge of Generative AI.
• Experience with LLMs like OpenAI, Azure OpenAI, Llama, Claude, etc.
• Familiarity with RAG Frameworks.
• Experience with Vector Databases.
• Proficient with Git.

Technical Qualifications:

• Preference for candidates with GenAI Databricks experience in conjunction with Data Engineering.

Additional Responsibilities:

• Preferred Skills
• Experience with Azure Databricks or AWS Databricks.
• Familiarity with LangChain and LangGraph.
• Experience with Azure OpenAI.
• Knowledge of MLflow.
• Experience with Kafka.
• Proficiency in Airflow.
• Familiarity with Snowflake.
• Experience with DBT.
• Understanding of MLOps concepts.
• Desirable Skills
• Knowledge of Prompt Engineering.
• Familiarity with Agentic AI.
• Understanding of Semantic Search.
• Experience with REST APIs.
• Familiarity with Docker.
• Exposure to CI/CD practices.

Desired Skills:

Technology->AI-Generative AI->Generative AI for Data Analytics, Technology->Data Engineering->Databricks