Senior Data Engineer

Role:- Senior Data Engineer

Location:- Pune/Bangalore|

Responsibilities:

• Design and implement data pipelines to cleanse and transform data in preparation for feeding to AI/Forecasting, Price Optimization, and related Supply Chain SaaS softwareproducts.

• Collaborate closely with Data Scientists to support feature engineering and operationalize AI/ML and mathematical optimization models.

• Develop efficient and robust data transformation code in PySpark.

• Build and maintain workflows to automate data pipelines using tools such as Databricks and Airflow.

• Optimize performance of data pipelines to ensure minimal cost.

• Develop data validation tests to assess the quality of the input data and recommend remediation measures.

• Conduct performance testing and profiling of the code using a variety of tools and techniques.

• Create and maintain comprehensive technical documentation for data pipelines and architecture.

Qualifications:

Job Requirements:

• Minimum Education: Bachelor's degree in computer science, Engineering, or a related field (or equivalent work experience).

• Minimum Work Experience (years):

o 3 - 6 years of general Data Engineering experience.

o Within that timeframe, experience is expected in: Python/PySpark, SQL, and relational or NoSQL databases.

• Key Skills and Competencies:

o Experience working with AWS, Azure, or GCP cloud environments.

o Experience building end-to-end data pipelines using PySpark or SQL. Strong understanding of parallel processing, clusters and compute management in the cloud.

o Experience with data orchestration tools like Databricks, Airflow (or similar tools like Snowflake, Dagster, etc.).

o Excellent verbal and written communication skills, especially in technical communications.

o Demonstrated ability to learn new technologies quickly and independently.

o Preferred (but not required): Familiarity with Data Science concepts (e.g., Machine Learning) and a desire to grow into hybrid engineering/data science tasks.

o Ability to work and achieve stretch goals in a very innovative and fast-paced environment.

• Licenses/Certifications, special qualifications: N/A