GCP Data Engineer

Job Overview:

• Demonstrated hands-on experience in GCP Data Engineering across scalable cloud data infrastructures
• Proficient in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM, and Secret Manager
• Extensive experience in both batch processing and real-time streaming with tools such as Pub/Sub, Dataflow, Apache Beam, Spark Structured Streaming, or Kafka
• Solid coding skills in Python, PySpark, and SQL, including debugging and performance enhancements
• Experience in constructing high-volume, low-latency ingestion and transformation pipelines with features like retry, idempotency, deduplication, and dead letter queue management
• Strong knowledge of Databricks, Delta Lake, Structured Streaming, Auto Loader, and Spark performance tuning
• Proficient in managing TB to PB scale datasets, including partitioning, clustering, file compaction, schema evolution, and data lifecycle management
• Deep understanding of Data Lake, Lakehouse, and contemporary cloud data platform architectures
• Practical experience in data modeling for analytics, reporting, and downstream usage
• Experience with CI/CD implementations using Cloud Build, GitHub Actions, Jenkins, or similar, along with Terraform for Infrastructure as Code
• Strong skills in troubleshooting, observability, production support, and cloud cost management
• Exceptional communication, stakeholder management, and problem-solving abilities

Primary Duties:

• Overview of responsibilities and job description with additional criteria for consideration
• Design and implement scalable GCP Data Engineering solutions and cloud-native data platforms
• Develop batch and real-time data pipelines utilizing BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Enhance and optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for large-scale processing
• Establish high-throughput streaming solutions via Pub/Sub and Dataflow, incorporating windowing, watermarking, late data handling, duplicate management, and dead letter queues
• Manage large-scale TB to PB data with an emphasis on performance, scalability, reliability, and cost efficiency
• Create complex SQL transformations, data models, and analytics-ready BigQuery datasets using partitioning and clustering techniques
• Orchestrate data pipelines utilizing Cloud Composer and Airflow while implementing CI/CD and Terraform-based Infrastructure as Code
• Ensure data governance, quality, security, monitoring, alerting, and production support
• Collaborate with architects, stakeholders, and cross-functional teams in an Agile environment
• Diagnose and enhance end-to-end GCP Databricks and streaming data pipelines

Technical Qualifications:

• Experience with GCP, including BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, batch and real-time streaming, Databricks, and PySpark
• Proficient in Python and advanced SQL
• Familiarity with Databricks, Delta Lake, and Structured Streaming
• Overview of responsibilities and job description with additional criteria for consideration
• Design and implement scalable GCP Data Engineering solutions and cloud-native data platforms
• Develop batch and real-time data pipelines utilizing BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Enhance and optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for large-scale processing
• Establish high-throughput streaming solutions via Pub/Sub and Dataflow, incorporating windowing, watermarking, late data handling, duplicate management, and dead letter queues
• Manage large-scale TB to PB data with an emphasis on performance, scalability, reliability, and cost efficiency
• Create complex SQL transformations, data models, and analytics-ready BigQuery datasets using partitioning and clustering techniques
• Orchestrate data pipelines utilizing Cloud Composer and Airflow while implementing CI/CD and Terraform-based Infrastructure as Code
• Ensure data governance, quality, security, monitoring, alerting, and production support
• Collaborate with architects, stakeholders, and cross-functional teams in an Agile environment
• Diagnose and enhance end-to-end GCP Databricks and streaming data pipelines

Additional Duties:

• Infosys stands as a global frontrunner in next-generation digital services and consulting
• Within the Data Analytics sector, you will engage in innovative data engineering and analytics projects for international clients across various industries
• This position provides chances to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and extensive enterprise data modernization initiatives
• Employees have access to ongoing learning programs, certifications, and career advancement opportunities within Infosys' data and analytics practice

Preferred Competencies:

Technology->Cloud Platform->GCP Database