Job description

Partner closely with leads (development/product owner/stakeholders/ etc.) to ensure a clear understanding of business and technical needs; assisting in selecting the best strategy and translate them into effective data solutions daily .
Design and implement solutions in Spark using industry-leading best practices for life sciences data while ensuring optimal performance, efficient operations, security, and compliance with regulatory standards.
Work with in house product teams to optimize the existing data pipelines from effort and cost perspective
Identify, test prototype solution and proof of concepts. Supporting testing team with identifying the optimal test strategy.
Get reviews on designs to ensure consistency with architectural best practices; participate in regular implementation reviews to ensure consistent quality and adherence with standards.
Work in an agile team to implement data quality checks, setup process orchestrations, mitigate vulnerabilities, devise testing strategies, and automate repeatable tasks along with participating and assisting in root cause analysis activities.
Ensure technical standards and architectural best practices are consistently followed. Govern complete architecture and implementation of the project module.

What You ll Bring

0-3 years of data engineering experience with track record in ETL, data modeling, data warehouse design, development, ingesting data from diverse data sources (databases or APIs) and performing complex transformations using Spark and related technologies.
Strong expertise in Python and SQL
Hands-on experience developing and optimizing jobs with pyspark apis along with sparksql
Hands-on experience on working with Linux operating system
Data-driven mindset with high analytical capability with history of leveraging data and analytical techniques to optimize business decisions.
Understanding of AWS services like EMR, S3, Glue and Redshift.
Good to have exposure with Databricks
Good to have exposure to visualization tools like Power BI or Tableu

Role: Data Engineer

Industry Type: Management Consulting

Department: Data Science & Analytics

Employment Type: Full Time, Permanent

Role Category: Data Science & Machine Learning

Education

UG: Any Graduate

PG: Any Postgraduate

Cloud Administrator (Analytics)