WI
Job Description
Key Responsibilities:
- Design & Development:
- Lead the architecture and implementation of an enterprise-grade data platform using Microsoft Fabric and related Azure services.
- Design and implement Medallion architecture (Bronze/Silver/Gold layers) to manage raw, validated, and curated data.
- Develop real-time data ingestion, validation, and transformation pipelines supporting heterogeneous file formats.
- Build and orchestrate event-driven workflows using serverless Azure Functions, Event Hub, and Durable Functions.
- Integrate Fabric Lakehouses and Synapse Data Warehousing with transactional systems and UI components.
- Implement schema evolution, metadata management, and lineage tracking across the platform.
- Enable business self-service through configurable rules engines and validation UIs.
- Ensure role-based access control (RBAC), end-to-end encryption, and audit trails across all layers.
- Build CI/CD pipelines and monitoring for automated deployments and observability.
- Collaboration & Leadership:
- Collaborate with UI engineers, solution architects, product owners, and compliance stakeholders.
- Lead and mentor a team of data engineers; establish coding and documentation standards.
Required Skills and Qualifications:
- Experience:
- 8-12 years of hands-on experience in data engineering, including 3+ years in a leadership or architecture role.
- Core Skills:
- Strong expertise in Microsoft Azure, especially:
- Microsoft Fabric (Data Lakehouses, Pipelines, Real-Time Analytics, Synapse).
- Azure Data Factory, Azure Functions, Event Hub / Service Bus.
- Delta Lake on ADLS Gen2.
- Azure API Management and App Services.
- Advanced knowledge of Python, PySpark, SQL, and workflow orchestration.
- Experience building event-driven, serverless architectures in a production environment.
- Deep understanding of schema-agnostic ingestion, validation logic, and dynamic rule engines.
- Knowledge of data privacy, regulatory compliance, lineage, and auditability.
- Experience integrating with transactional UIs and REST APIs for CRUD operations.
- Proven track record delivering large-scale, cloud-native data platforms.
Good to Have:
- Domain knowledge in insurance, financial services, or regulatory systems.
- Familiarity with RE7/FSRI, master data integration, or third-party systems.
- Experience with retention, premium, or commission calculations.
- Prior exposure to Microsoft Purview, Azure Monitor, and Azure Key Vault.
- Understanding of multilingual support, global data residency compliance, and deployment automation.
