Databricks Data Engineer

Job Overview:

• We are in search of a dynamic Databricks Data Engineer to create, develop, and enhance contemporary data platforms utilizing Databricks and cloud technologies.
• The successful candidate will possess practical experience in constructing large-scale data pipelines, implementing Lakehouse architectures, and facilitating advanced analytics solutions through Databricks Spark and cloud environments.
• In this position, you will collaborate closely with data architects, analysts, business stakeholders, and engineering teams to deliver scalable, high-performance data solutions that underpin enterprise-wide digital transformation efforts.

Core Responsibilities:

• Design, develop, and sustain scalable data pipelines with Databricks and Apache Spark.
• Create and enhance ETL/ELT workflows to process extensive structured and unstructured datasets.
• Develop data ingestion frameworks to assimilate data from diverse sources including databases, APIs, applications, and cloud storage solutions.
• Implement and oversee Lakehouse architecture leveraging Databricks and Delta Lake.
• Refine Spark jobs and Databricks workloads for optimal performance, scalability, and cost-effectiveness.
• Collaborate with analytics, BI, and business teams to ascertain data requirements and deliver reusable data assets.
• Establish data quality frameworks, monitoring solutions, and governance controls across data pipelines.
• Diagnose production issues and provide swift resolutions to ensure business continuity.
• Engage in architecture reviews, code evaluations, and technical discussions to advocate for best practices.
• Create and maintain technical documentation, reusable components, and deployment standards.

Technical Qualifications:

• Extensive hands-on experience with Databricks.
• Proficient in PySpark or Spark SQL for distributed data processing.
• Strong SQL expertise with a background in query optimization and performance enhancement.
• Experience in designing and developing ETL/ELT pipelines.
• Hands-on experience with Delta Lake, DataFrames, and Spark architecture.
• Understanding of data warehousing concepts, dimensional modeling, and data lake frameworks.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Familiarity with Git-based version control and DevOps methodologies.
• Strong analytical, debugging, and problem-solving capabilities.

Additional Duties:

• Infosys Data Analytics practice assists organizations in modernizing their data frameworks through cloud-native architectures, advanced analytics, AI, and intelligent data platforms.
• As a Databricks Data Engineer, you will engage in enterprise-scale data modernization projects involving Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Preferred Expertise:

Technology->Data Engineering->Databricks