Data Engineer 2

Key Responsibilities

• Design, develop, and maintain scalable and efficient cloud-based data infrastructure using SQL and PySpark.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver appropriate data solutions.
• Identify potential data sources, define data ingestion architecture, and implement efficient data pipelines.
• Ensure smooth flow of data from sources to data lakes, warehouses, and analytical platforms.
• Troubleshoot and resolve issues related to data processing, pipeline performance, and data quality.
• Implement automated testing frameworks for data validation, integrity, and pipeline performance.
• Integrate automated tests into CI/CD pipelines for real-time testing and deployment.
• Work with streaming data and real-time APIs/services, performing automated validation using custom scripts and assertions.
• Document data infrastructure, ETL processes, and pipelines to ensure knowledge transfer and maintainability.
• Stay updated with emerging cloud technologies, data engineering tools, and best practices.
• Ensure compliance with data governance, security, and quality standards.
• Support DevOps activities including installation, configuration, and integration of automation scripts on continuous integration tools like GitHub and CI/CD pipelines.