GCP Data Engineer

Job Overview:

• Extensive hands-on experience in GCP Data Engineering across expansive cloud data solutions
• Proficient in Google Cloud services including Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM, and Secret Manager
• Robust experience in both batch processing and real-time streaming utilizing Pub/Sub, Dataflow, Apache Beam, Spark Structured Streaming, or Kafka
• Proficient in coding with Python, PySpark, and SQL, with skills in debugging and enhancing performance
• Experience in developing high-volume, low-latency ingestion and transformation pipelines with features like retry, idempotency, deduplication, and dead letter handling
• Strong background in Databricks, Delta Lake, Structured Streaming, Auto Loader, and Spark performance tuning
• Experience managing datasets at TB and PB scales, including partitioning, clustering, file compaction, schema evolution, and data lifecycle management
• Comprehensive understanding of Data Lake, Lakehouse, and modern cloud data platform architectures
• Practical experience in data modeling for analytics, reporting, and downstream use
• Experience in implementing CI/CD pipelines using Cloud Build, GitHub Actions, Jenkins, or similar tools, along with Terraform for Infrastructure as Code
• Strong troubleshooting, observability, production support, and cloud cost optimization capabilities
• Exceptional client communication, stakeholder management, and problem-solving abilities

Core Responsibilities:

• Develop and design scalable GCP Data Engineering solutions and cloud-native data platforms
• Create batch and real-time data pipelines utilizing BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Enhance and optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for large-scale processing
• Implement high-throughput streaming solutions utilizing Pub/Sub and Dataflow, with features such as windowing, watermarking, late data handling, duplicate management, and dead letter queues
• Manage large-scale data at TB and PB levels, focusing on performance, scalability, reliability, and cost efficiency
• Develop intricate SQL transformations, data models, and analytics-ready BigQuery datasets using partitioning and clustering techniques
• Orchestrate data pipelines via Cloud Composer and Airflow, implementing CI/CD and Terraform for Infrastructure as Code
• Ensure data quality, governance, security, monitoring, alerting, and production support
• Collaborate with architects, stakeholders, and cross-functional teams within an Agile framework
• Troubleshoot and enhance end-to-end GCP, Databricks, and streaming data pipelines

Technical Qualifications:

• GCP expertise in BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, along with batch and real-time streaming proficiency in Databricks and PySpark
• Strong command of Python and SQL
• Experience with Databricks, specifically Delta Lake and Structured Streaming
• Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms
• Create batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks
• Optimize Databricks Delta Lake, Spark, and Structured Streaming workloads for extensive data processing
• Implement high-throughput streaming solutions using Pub/Sub and Dataflow, focusing on windowing, watermarking, late data management, duplicate handling, and dead letter queues
• Manage substantial TB and PB scale data with an emphasis on performance, scalability, reliability, and cost management
• Develop advanced SQL transformations, data models, and analytics-ready datasets in BigQuery utilizing partitioning and clustering
• Orchestrate data pipelines with Cloud Composer and Airflow, while implementing CI/CD and Terraform for Infrastructure as Code
• Maintain data quality, governance, security, monitoring, alerting, and production support
• Engage with architects, stakeholders, and cross-functional teams within an Agile delivery model
• Troubleshoot and optimize comprehensive GCP, Databricks, and streaming data pipelines

Additional Responsibilities:

• Infosys stands as a pioneer in next-generation digital services and consulting
• In the Data Analytics division, you will engage in innovative data engineering and analytics projects for global clients across various industries
• This role provides opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and extensive enterprise data modernization initiatives
• Employees have access to continuous learning programs, certifications, and career advancement opportunities within Infosys's data and analytics practice

Desirable Skills:

Technology->Cloud Platform->GCP Database