GCP Data Engineer

Job Overview:

• Extensive hands-on experience in GCP Data Engineering within scalable cloud data infrastructures
• Proficient in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM, and Secret Manager
• Significant experience in batch and real-time streaming utilizing Pub/Sub, Dataflow, Apache Beam, Spark, Structured Streaming, or Kafka
• Proficient coding skills in Python, PySpark, and SQL, including debugging and performance enhancements
• Experience in developing high-throughput, low-latency ingestion and transformation pipelines featuring retry mechanisms, idempotency, deduplication, and dead-letter handling
• Strong background in Databricks, Delta Lake, Structured Streaming, Auto Loader, and Spark performance optimization
• Experience managing TB to PB scale datasets, focusing on partitioning, clustering, file compaction, schema evolution, and data lifecycle management
• Deep understanding of Data Lake, Lakehouse, and contemporary cloud data platform architectures
• Practical experience in data modeling for analytics, reporting, and downstream applications
• Familiarity with CI/CD implementations using Cloud Build, GitHub Actions, Jenkins, or similar tools, alongside Terraform for Infrastructure as Code
• Strong capabilities in troubleshooting, observability, production support, and optimizing cloud costs
• Excellent communication, stakeholder management, and problem-solving abilities

Primary Duties:

• Summarize responsibilities and JD briefly, along with additional criteria for evaluation
• Design and construct scalable GCP Data Engineering solutions and cloud-native data platforms
• Develop both batch and real-time pipelines utilizing BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Enhance and optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for extensive processing requirements
• Implement high-throughput streaming architectures using Pub/Sub and Dataflow, incorporating windowing, watermarking, late-arriving data management, duplicate handling, and dead-letter queues
• Manage large-scale TB to PB data with an emphasis on performance, scalability, reliability, and cost-effectiveness
• Develop intricate SQL transformations, data models, and analytics-ready BigQuery datasets utilizing partitioning and clustering techniques
• Orchestrate pipelines through Cloud Composer/Airflow and execute CI/CD with Terraform-based Infrastructure as Code
• Ensure data quality, governance, security, monitoring, alerting, and production support
• Engage with architects, stakeholders, and cross-functional teams within an Agile delivery framework
• Troubleshoot and optimize GCP Databricks and streaming data pipelines from end to end

Technical Qualifications:

• GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, batch and real-time streaming, Databricks, PySpark
• Proficiency in Python and strong SQL skills
• Experience with Databricks including Delta Lake and Structured Streaming
• Summarize responsibilities and JD briefly, along with additional criteria for evaluation
• Design and construct scalable GCP Data Engineering solutions and cloud-native data platforms
• Develop both batch and real-time pipelines utilizing BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Enhance and optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for extensive processing requirements
• Implement high-throughput streaming architectures using Pub/Sub and Dataflow, incorporating windowing, watermarking, late-arriving data management, duplicate handling, and dead-letter queues
• Manage large-scale TB to PB data with an emphasis on performance, scalability, reliability, and cost-effectiveness
• Develop intricate SQL transformations, data models, and analytics-ready BigQuery datasets utilizing partitioning and clustering techniques
• Orchestrate pipelines through Cloud Composer/Airflow and execute CI/CD with Terraform-based Infrastructure as Code
• Ensure data quality, governance, security, monitoring, alerting, and production support
• Engage with architects, stakeholders, and cross-functional teams within an Agile delivery framework
• Troubleshoot and optimize GCP Databricks and streaming data pipelines from end to end

Additional Responsibilities:

• Infosys stands as a global leader in next-generation digital services and consulting
• Within the Data Analytics division, you will engage in advanced data engineering and analytics projects for international clients spanning various industries
• This position provides opportunities to work with cloud-native data infrastructures, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization initiatives
• Employees are offered continuous learning programs, certifications, and career advancement opportunities within Infosys' data and analytics sector

Desirable Skills:

Technology->Cloud Platform->GCP Database