Data Engineer-Data Platforms-Google

Introduction

Build a long-term career with IBM Consulting by partnering closely with clients worldwide. You’ll collaborate with leading companies across industries to help shape their hybrid cloud and AI strategies. Backed by strategic partners, IBM technology, and Red Hat, you’ll gain the capabilities to drive meaningful change and help accelerate measurable client outcomes. At IBM Consulting, curiosity is the foundation for success—encouraging you to challenge conventional thinking, explore new concepts, and develop innovative solutions that deliver tangible results. With a culture centered on growth and empathy, you’ll receive support for your long-term career development while bringing your unique skills and experiences to the team.

Your Role And Responsibilities

As a Data Engineer focused on Google’s data platforms, you will design, build, and maintain data engineering solutions within the Google Cloud ecosystem. This position emphasizes expertise in using multiple Google services to support batch and real-time data pipelines, data migration, and data layer design.

Your Primary Responsibilities Will Include

• Design Data Pipelines: Design and develop batch and real-time data pipelines for Data Warehouse and Datalake using Google services such as DataProc, DataFlow, PubSub, BigQuery, and Big Table.
• Develop Data Engineering Solutions: Use Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python to create and maintain data engineering solutions.
• Manage Data Platforms: Plan, schedule, and manage the data platform using Google Cloud Scheduler and Cloud Composer (Airflow), ensuring smooth and efficient data pipeline execution.
• Implement Data Migration: Create and implement data migration solutions with Google services to ensure seamless data transfer between systems.
• Optimize Data Layer: Design and optimize the data layer using Google services such as BigQuery, Big Table, and Cloud Spanner to support efficient data storage and retrieval.
  
  

Preferred Education

Master's Degree

Required Technical And Professional Expertise

• Google Cloud Ecosystem Expertise: Exposure to designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem, including services such as Google DataProc, DataFlow, PubSub, BigQuery, Big Table, Cloud Spanner, CloudSQL, and AlloyDB.
• Data Pipeline Development Experience: Exposure to developing and managing batch and real-time data pipelines for Data Warehouse and Datalake using Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala.
• Google Cloud Services Proficiency: Experience working with Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python to build and maintain data engineering solutions.
• Data Platform Management Knowledge: Exposure to scheduling and managing the data platform using Google Cloud Scheduler and Cloud Composer (Airflow) for efficient data pipeline operations.
• Data Layer Design Understanding: Experience working with data layer design using Google services such as BigQuery, Big Table, and Cloud Spanner for efficient data storage and retrieval.
  
  

Preferred Technical And Professional Experience

• Open Source Technologies: Exposure to utilizing open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala for developing and managing batch and real-time data pipelines.
• Data Migration Solutions: Experience working with Google services to develop and implement data migration solutions, ensuring seamless data transfer between systems.
• Cloud Composer Expertise: Exposure to using Cloud Composer (Airflow) for scheduling and managing the data platform, ensuring efficient data pipeline operations.