Databricks Data Engineer

Job Overview:

• We are in search of a driven Databricks Data Engineer to architect, develop, and enhance state-of-the-art data platforms utilizing Databricks and cloud technologies.
• The successful candidate will possess practical experience in constructing large-scale data pipelines, implementing Lakehouse architectures, and facilitating advanced analytics solutions through Databricks, Spark, and various cloud environments.
• This position involves close collaboration with data architects, analysts, business stakeholders, and engineering teams to provide scalable, high-performance data solutions that support comprehensive digital transformation efforts across the enterprise.

Main Responsibilities:

• Design, build, and manage scalable data pipelines utilizing Databricks and Apache Spark.
• Develop and refine ETL and ELT workflows for the processing of extensive structured and unstructured datasets.
• Create data ingestion frameworks to assimilate data from diverse sources, including databases, APIs, applications, and cloud storage.
• Implement and oversee Lakehouse architecture utilizing Databricks and Delta Lake.
• Enhance Spark jobs and Databricks workloads to ensure optimal performance, scalability, and cost-effectiveness.
• Engage with analytics, BI, and business teams to comprehend data needs and deliver reusable data assets.
• Establish data quality frameworks, monitoring solutions, and governance controls throughout data pipelines.
• Diagnose production issues and provide swift resolutions to maintain business continuity.
• Take part in architecture reviews, code assessments, and technical discussions to advocate for best practices.
• Generate and sustain technical documentation, reusable components, and deployment standards.

Technical Qualifications:

• Proven hands-on experience with Databricks.
• Expertise in PySpark or Spark SQL for distributed data processing.
• Strong SQL abilities with a background in query optimization and performance enhancement.
• Experience in designing and implementing ETL and ELT pipelines.
• Hands-on experience with Delta Lake, DataFrames, and Spark architecture.
• Familiarity with data warehousing principles, dimensional modeling, and data lake architectures.
• Experience using cloud platforms such as Azure, AWS, or GCP.
• Knowledge of Git-based source code management and DevOps methodologies.
• Strong analytical, debugging, and problem-solving capabilities.

Additional Duties:

• Infosys Data Analytics practice aids organizations in modernizing their data landscape through cloud-native architectures, advanced analytics, AI, and intelligent data platforms.
• As a Databricks Data Engineer, you will engage in enterprise-level data modernization initiatives, which include Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Preferred Skills:

Technology->Data Engineering->Databricks