Azure Databricks Data Engineer

Job Overview:

• We are looking for a driven Databricks Data Engineer to architect, develop, and enhance state-of-the-art data platforms utilizing Databricks and cloud technologies.
• The successful candidate will possess practical experience in constructing extensive data pipelines, implementing Lakehouse frameworks, and facilitating advanced analytics solutions utilizing Databricks Spark and various cloud environments.
• In this position, you will collaborate closely with data architects, analysts, business stakeholders, and engineering teams to provide scalable, high-performance data solutions that support enterprise-wide digital transformation objectives.

Primary Responsibilities:

• Design, develop, and sustain scalable data pipelines utilizing Databricks and Apache Spark.
• Create and enhance ETL and ELT workflows for processing large-scale structured and unstructured data sets.
• Develop frameworks for data ingestion to merge data from diverse sources, including databases, APIs, applications, and cloud storage.
• Establish and oversee Lakehouse architecture using Databricks and Delta Lake.
• Refine Spark jobs and Databricks workloads to ensure optimal performance, scalability, and cost-effectiveness.
• Work in partnership with analytics, business intelligence, and business teams to comprehend data needs and deliver reusable data resources.
• Implement data quality frameworks, monitoring solutions, and governance protocols across data pipelines.
• Diagnose production issues and ensure prompt resolution to maintain business continuity.
• Engage in architecture evaluations, code assessments, and technical discussions to advocate for best practices.
• Produce and manage technical documentation, reusable components, and deployment standards.

Technical Qualifications:

• 7 to 9 years of robust hands-on experience with Databricks.
• Expertise in PySpark or Spark SQL for distributed data processing.
• Strong SQL capabilities with a background in query optimization and performance enhancement.
• Experience in creating and developing ETL and ELT pipelines.
• Practical knowledge of Delta Lake, DataFrames, and Spark architecture.
• Understanding of data warehousing principles, dimensional modeling, and data lake architectures.
• Experience with cloud platforms, such as Azure, AWS, or GCP.
• Familiarity with Git-based source code management and DevOps methodologies.
• Excellent analytical, debugging, and problem-solving skills.

Additional Duties:

• The Infosys Data Analytics practice aids organizations in modernizing their data landscapes through cloud-native architectures, advanced analytics, AI, and intelligent data platforms.
• As a Databricks Data Engineer, you will contribute to enterprise-scale data modernization initiatives that involve Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Preferred Expertise:

Technology->Data Engineering->Databricks