Azure Databricks Data Engineer

Job Description:

• We are looking for a driven Databricks Data Engineer who will be responsible for designing, developing, and optimizing contemporary data platforms utilizing Databricks and cloud technologies.
• The perfect candidate will possess hands-on experience in constructing large-scale data pipelines, implementing Lakehouse architectures, and facilitating advanced analytics solutions through Databricks Spark and cloud ecosystems.
• This position involves close collaboration with data architects, analysts, business stakeholders, and engineering teams to provide scalable and high-performance data solutions that drive enterprise-wide digital transformation efforts.

Key Responsibilities:

• Design, develop, and sustain scalable data pipelines with Databricks and Apache Spark.
• Create and enhance ETL and ELT workflows for processing extensive structured and unstructured datasets.
• Develop frameworks for data ingestion to consolidate data from various sources, including databases, APIs, applications, and cloud storage.
• Establish and manage Lakehouse architecture utilizing Databricks and Delta Lake.
• Optimize Spark jobs and Databricks workloads for improved performance, scalability, and cost-effectiveness.
• Work in partnership with analytics, BI, and business teams to comprehend data requirements and deliver reusable data assets.
• Implement data quality frameworks, monitoring solutions, and governance controls across data pipelines.
• Diagnose production issues and ensure prompt resolution to maintain business continuity.
• Engage in architecture reviews, code reviews, and technical discussions to advocate for best practices.
• Generate and maintain technical documentation, reusable components, and deployment standards.

Technical Requirements:

• 5 to 7 years of substantial hands-on experience with Databricks.
• Expertise in PySpark or Spark SQL for distributed data processing.
• Strong SQL proficiency with a background in query optimization and performance tuning.
• Experience in the design and development of ETL and ELT pipelines.
• Hands-on experience with Delta Lake, DataFrames, and Spark architecture.
• Understanding of data warehousing concepts, dimensional modeling, and data lake architectures.
• Familiarity with cloud platforms such as Azure, AWS, or GCP.
• Experience with Git-based source code management and DevOps methodologies.
• Excellent analytical, debugging, and problem-solving capabilities.

Additional Responsibilities:

• The Infosys Data Analytics practice assists organizations in modernizing their data landscape through cloud-native architectures, advanced analytics, AI, and intelligent data platforms.
• As a Databricks Data Engineer, you will engage in enterprise-scale data modernization initiatives, including Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Preferred Skills:

Technology->Data Engineering->Databricks