WI

Lead Data Engineer

Wissen Infotech
Bangalore5-8 LPA Posted 30 Jul 2025
FULL TIME
Data Engineer

Job Description

Key Responsibilities:

  • Design & Development:
  • Lead the architecture and implementation of an enterprise-grade data platform using Microsoft Fabric and related Azure services.
  • Design and implement Medallion architecture (Bronze/Silver/Gold layers) to manage raw, validated, and curated data.
  • Develop real-time data ingestion, validation, and transformation pipelines supporting heterogeneous file formats.
  • Build and orchestrate event-driven workflows using serverless Azure Functions, Event Hub, and Durable Functions.
  • Integrate Fabric Lakehouses and Synapse Data Warehousing with transactional systems and UI components.
  • Implement schema evolution, metadata management, and lineage tracking across the platform.
  • Enable business self-service through configurable rules engines and validation UIs.
  • Ensure role-based access control (RBAC), end-to-end encryption, and audit trails across all layers.
  • Build CI/CD pipelines and monitoring for automated deployments and observability.
  • Collaboration & Leadership:
  • Collaborate with UI engineers, solution architects, product owners, and compliance stakeholders.
  • Lead and mentor a team of data engineers; establish coding and documentation standards.

Required Skills and Qualifications:

  • Experience:
  • 8-12 years of hands-on experience in data engineering, including 3+ years in a leadership or architecture role.
  • Core Skills:
  • Strong expertise in Microsoft Azure, especially:
  • Microsoft Fabric (Data Lakehouses, Pipelines, Real-Time Analytics, Synapse).
  • Azure Data Factory, Azure Functions, Event Hub / Service Bus.
  • Delta Lake on ADLS Gen2.
  • Azure API Management and App Services.
  • Advanced knowledge of Python, PySpark, SQL, and workflow orchestration.
  • Experience building event-driven, serverless architectures in a production environment.
  • Deep understanding of schema-agnostic ingestion, validation logic, and dynamic rule engines.
  • Knowledge of data privacy, regulatory compliance, lineage, and auditability.
  • Experience integrating with transactional UIs and REST APIs for CRUD operations.
  • Proven track record delivering large-scale, cloud-native data platforms.

Good to Have:

  • Domain knowledge in insurance, financial services, or regulatory systems.
  • Familiarity with RE7/FSRI, master data integration, or third-party systems.
  • Experience with retention, premium, or commission calculations.
  • Prior exposure to Microsoft Purview, Azure Monitor, and Azure Key Vault.
  • Understanding of multilingual support, global data residency compliance, and deployment automation.

Required Skills

Join WhatsApp Channel