Data Engineering Lead

Key Responsibilities:

Technical Leadership

• Lead and mentor a team of data engineers, fostering best practices in coding, design, and delivery.
• Drive adoption of modern data engineering frameworks, tools, and methodologies.
• Translate complex business requirements into data pipelines, architectures, and workflows.

Data Pipeline Development

• Architect, develop, and optimize scalable ETL/ELT pipelines using Apache Spark, Hive, AWS Glue, and Trino.
• Handle complex data workflows involving structured and unstructured data.
• Develop real-time and batch processing systems supporting BI, analytics, and ML applications.

Cloud & Infrastructure Management

• Build and maintain cloud-based data solutions with AWS services like S3, Athena, Redshift, EMR, DynamoDB, and Lambda.
• Design federated query capabilities using Trino.
• Manage Hive Metastore for schema and metadata in data lakes.

Performance Optimization

• Optimize Apache Spark jobs and Hive queries for performance and resource efficiency.
• Implement caching and indexing strategies in Trino.
• Monitor and tune system performance continuously.

Collaboration & Stakeholder Engagement

• Work closely with data scientists, analysts, and business teams to deliver actionable insights.
• Ensure data infrastructure aligns with organizational goals and compliance standards.

Data Governance & Quality

• Establish and enforce data quality standards, governance practices, and monitoring.
• Ensure data security, privacy, and regulatory compliance.

Innovation & Continuous Learning

• Stay updated on industry trends and emerging data engineering technologies.
• Identify and implement improvements in data architecture and processes.