Data Engineer-Data Platforms-Google

Introduction

A career with IBM Consulting revolves around fostering enduring client partnerships and engaging in collaborative efforts globally. You will partner with top-tier companies from various sectors, guiding them in their hybrid cloud and AI transformations. With the backing of our strategic partners, advanced IBM technology, and Red Hat, you will have the resources necessary to instigate significant change and enhance client outcomes. At IBM Consulting, a spirit of inquiry drives achievement. You will be inspired to question the status quo, investigate fresh concepts, and develop inventive solutions that yield tangible results. Our culture emphasizes growth and empathy, prioritizing your long-term career advancement while appreciating your distinct skills and experiences.

Your Role And Responsibilities

As a Data Engineer with a focus on Google's data platforms, you will be responsible for designing, constructing, and maintaining data engineering solutions within the Google Cloud framework. This position calls for proficiency in leveraging a variety of Google services for batch and real-time data pipelines, data migration, and data layer architecture.

Your Primary Responsibilities Will Include

• Design Data Pipelines: Create and implement batch and real-time data pipelines for Data Warehouse and Datalake using Google services such as DataProc, DataFlow, PubSub, BigQuery, and Big Table.
• Develop Data Engineering Solutions: Employ Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python to construct and sustain data engineering solutions.
• Manage Data Platforms: Schedule and oversee the data platform utilizing Google Cloud Scheduler and Cloud Composer (Airflow), ensuring optimal operations of data pipelines.
• Implement Data Migration: Formulate and execute data migration strategies using Google services, guaranteeing smooth data transitions between systems.
• Optimize Data Layer: Design and refine the data layer employing Google services such as BigQuery, Big Table, and Cloud Spanner to ensure effective data storage and access.
  
  

Preferred Education

Master's Degree

Required Technical And Professional Expertise

• Google Cloud Ecosystem Expertise: Familiarity with designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem, encompassing services like Google DataProc, DataFlow, PubSub, BigQuery, Big Table, Cloud Spanner, CloudSQL, and AlloyDB.
• Data Pipeline Development Experience: Knowledge in developing and managing batch and real-time data pipelines for Data Warehouse and Datalake utilizing Google services and open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala.
• Google Cloud Services Proficiency: Experience with Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python for building and maintaining data engineering solutions.
• Data Platform Management Knowledge: Understanding of scheduling and managing the data platform through Google Cloud Scheduler and Cloud Composer (Airflow) to ensure effective data pipeline operations.
• Data Layer Design Understanding: Experience with data layer design using Google services such as BigQuery, Big Table, and Cloud Spanner for efficient data storage and retrieval.
  
  

Preferred Technical And Professional Experience

• Open Source Technologies: Familiarity with using open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala for the development and management of batch and real-time data pipelines.
• Data Migration Solutions: Experience in developing and implementing data migration solutions with Google services to ensure smooth data transfers between systems.
• Cloud Composer Expertise: Proficiency in utilizing Cloud Composer (Airflow) for the scheduling and management of the data platform, ensuring efficient operation of data pipelines.