Databricks Data Engineer

Job Overview:

• We are in search of an enthusiastic Databricks Data Engineer who will be responsible for designing, developing, and enhancing modern data platforms using Databricks and cloud-based technologies.
• The successful applicant will possess hands-on experience in constructing large-scale data pipelines, implementing Lakehouse architectures, and facilitating advanced analytics solutions utilizing Databricks, Spark, and various cloud ecosystems.
• In this position, you will collaborate closely with data architects, analysts, business stakeholders, and engineering teams to deliver scalable, high-performance data solutions that underpin enterprise-wide digital transformation efforts.

Primary Responsibilities:

• Design, develop, and sustain scalable data pipelines utilizing Databricks and Apache Spark.
• Construct and refine ETL/ELT workflows for processing extensive structured and unstructured datasets.
• Develop frameworks for data ingestion to assimilate data from diverse sources, including databases, APIs, applications, and cloud storage.
• Implement and oversee Lakehouse architecture using Databricks and Delta Lake.
• Enhance Spark jobs and Databricks workloads for optimal performance, scalability, and cost-effectiveness.
• Work in conjunction with analytics, BI, and business teams to comprehend data requirements and provide reusable data assets.
• Establish data quality frameworks, monitoring solutions, and governance controls throughout data pipelines.
• Diagnose production issues and deliver prompt resolutions to guarantee business continuity.
• Engage in architecture reviews, code reviews, and technical discussions to advocate for best practices.
• Generate and maintain technical documentation, reusable components, and deployment standards.

Technical Qualifications:

• 2-3 years of experience in Data Engineering and Big Data technologies.
• Substantial hands-on expertise with Databricks.
• Proficient in PySpark or Spark SQL for distributed data processing.
• Strong SQL capabilities with a focus on query optimization and performance tuning.
• Experience in designing and building ETL/ELT pipelines.
• Hands-on experience with Delta Lake, DataFrames, and Spark architecture.
• Familiarity with data warehousing concepts, dimensional modeling, and data lake architectures.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of Git-based source code management and DevOps methodologies.
• Strong analytical, debugging, and problem-solving skills.

Additional Duties:

• Infosys Data Analytics practice aids organizations in modernizing their data landscape through cloud-native architectures, advanced analytics, AI, and intelligent data platforms.
• As a Databricks Data Engineer, you will participate in enterprise-scale data modernization initiatives, encompassing Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Preferred Expertise:

Technology->Data Engineering->Databricks