Data Engineer-Data Platforms-AWS

Introduction

A career at IBM Consulting is grounded in enduring client partnerships and hands-on collaboration across the globe. You’ll work with leading organizations across industries, helping them plan and execute their hybrid cloud and AI strategies. Backed by strategic partners, IBM technology, and Red Hat, you’ll gain the capability to drive meaningful transformation and enhance client outcomes. At IBM Consulting, curiosity powers performance—you’ll be encouraged to challenge conventions, test new ideas, and build inventive solutions that produce measurable impact. Our growth and empathy-centered culture supports your long-term career development and values the distinct skills and perspectives you bring.

Your Role And Responsibilities

As a Data Engineer specializing in Data Platforms on AWS, you will advise on, design, and maintain data engineering solutions within the AWS Cloud environment. You will design, build, and run batch and real-time data pipelines using a range of AWS services.

Your Primary Responsibilities Will Include

• Design and Develop Data Pipelines: Design, build, and operate batch and real-time data pipelines using AWS services including AWS EMR, AWS Glue, Glue Catalog, and Kinesis, ensuring smooth integration and reliable operation of data engineering solutions.
• Create Data Layers: Build data layers on AWS RedShift, Aurora, and DynamoDB, and migrate data using AWS DMS.
• Manage Data Services: Plan, schedule, and oversee data services on the AWS Platform to support efficient execution of data engineering solutions.
• Develop Batch and Real-time Pipelines: Develop batch and real-time data pipelines for Data Warehouse and Datalake, using AWS Kinesis and Managed Streaming for Apache Kafka.
• Utilize Open Source Technologies: Apply open source technologies such as Apache Airflow and dbt, along with Spark / Python or Spark / Scala on the AWS Platform, to enable data engineering solutions.
  
  

Preferred Education

Master's Degree

Required Technical And Professional Expertise

• Exposure to AWS Toolset: Experience with AWS services including AWS EMR, AWS Glue, Glue Catalog, and Kinesis to design, build, and operate batch and real-time data pipelines.
• Data Pipeline Development: Exposure to developing batch and real-time data pipelines for Data Warehouse and Datalake, using AWS Kinesis and Managed Streaming for Apache Kafka.
• Data Layer Creation: Experience working with AWS RedShift, Aurora, and DynamoDB to create data layers and migrate data using AWS DMS.
• Open Source Technologies: Exposure to using open source technologies such as Apache Airflow and dbt, Spark / Python or Spark / Scala on the AWS Platform to support data engineering solutions.
• Data Service Management: Experience scheduling and managing data services on the AWS Platform, ensuring efficient operation of data engineering solutions.
  
  

Preferred Technical And Professional Experience

• Proficiency with AWS Databrew: Experience using AWS Glue Databrew to support data engineering solutions, including data preparation and data quality work.
• Knowledge of Lambda Functions: Exposure to using Lambda functions with Python to support data engineering solutions, including data processing and data transformation tasks.
• Familiarity with RedShift Spectrum: Experience working with RedShift Spectrum to support data engineering solutions, including data warehousing and data analytics activities.