INInfosys Limited
Azure Databricks & PySpark Principal Consultant
Bangalore Posted 13 Oct 2026
FULL TIME
Azure Databricks
Pyspark
Azure Data Factory
Data Warehousing
Sql
Job Description
Job Overview:
- We are seeking a seasoned Data Engineer with 12 to 16 years of comprehensive IT experience, along with robust proficiency in Azure Databricks and PySpark.
- The perfect candidate will have significant experience in developing scalable data pipelines that handle large datasets, delivering high-quality enterprise-level data solutions on the Azure platform.
Primary Responsibilities:
- Design, develop, and enhance large-scale data pipelines utilizing Azure Databricks and PySpark.
- Create and manage ETL/ELT workflows through Azure Data Factory.
- Develop frameworks for scalable data processing for both batch and near real-time data ingestion.
- Execute data transformation, cleansing, and enrichment processes.
- Work with extensive structured and unstructured datasets within Azure Data Lake.
- Optimize Spark jobs to ensure performance, scalability, and cost-effectiveness.
- Collaborate with business stakeholders, architects, and analytics teams to provide effective data solutions.
- Guarantee adherence to data quality, security, governance, and compliance standards.
- Deliver production support, troubleshoot issues, and promote continuous improvements.
- Guide junior team members and contribute to the design of technical solutions.
Technical Qualifications:
- 12 to 16 years of experience in IT with a strong focus on Data Engineering.
- Proficient hands-on experience in Azure Databricks and PySpark development.
- Comprehensive knowledge of Azure Data Factory (ADF).
- Expertise in SQL and Data Warehousing principles.
- Experience with Azure Data Lake Storage (ADLS Gen2).
- Strong understanding of Spark architecture and performance optimization.
- Proven experience in designing and implementing enterprise-scale data pipelines.
- Familiarity with CI/CD processes and Agile methodologies.