Data Engineer (Iceberg, Snowflake and Python)

Essential Qualifications: Data Engineer with expertise in Iceberg, Snowflake, and Python

Key Responsibilities

• Design, develop, and implement robust data engineering solutions tailored to meet the requirements of the PIPL program.
• Construct and manage end-to-end ETL/ELT pipelines utilizing Python, SQL, and contemporary data engineering frameworks.
• Develop and oversee data lakehouse architectures leveraging Apache Iceberg and Snowflake.
• Establish and maintain a dedicated data environment in Alibaba Cloud (AliCloud) for processing, analytics, machine learning, and reporting tasks.
• Automate processes for data ingestion, transformation, validation, and reporting to enhance operational efficiency.
• Optimize data pipelines, storage solutions, and query performance to ensure both scalability and reliability.
• Work collaboratively with architects, program leaders, and business stakeholders to gather requirements and deliver effective data solutions.
• Ensure compliance with PIPL and enterprise standards regarding data quality, governance, security, and compliance.
• Monitor, troubleshoot, and resolve issues related to the data platform and pipelines.
• Contribute to the development of best practices in DataOps, CI/CD, automation, and cloud-native data engineering.

Job Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a closely related field.
• Over 5 years of experience in Data Engineering, Data Warehousing, or working within Big Data environments.
• Extensive hands-on experience with Apache Iceberg.
• Demonstrated experience utilizing Snowflake for data warehousing and analytics purposes.
• Advanced skills in Python and SQL development.
• Experience in designing and constructing ETL/ELT pipelines and data integration solutions.
• Strong understanding of data lake, lakehouse, and data warehouse architectures.
• Familiarity with cloud platforms, with a preference for Alibaba Cloud (AliCloud).
• Knowledge of distributed data processing technologies such as Apache Spark, Flink, or similar frameworks.
• Experience in data modeling, performance tuning, and managing data quality.
• Proficiency with CI/CD pipelines, Git, and various automation tools.
• Excellent analytical, problem-solving, and stakeholder management abilities.
• Experience in environments focused on regulatory compliance or data privacy (PIPL/GDPR) is a plus.

Core Competencies: Apache Iceberg, Snowflake, Python, SQL, ETL/ELT Development, Data Engineering, Alibaba Cloud (preferable), Data Lakehouse Architecture.