Databricks Engineer

Job Description:

• We are seeking a seasoned Databricks Engineer with 5 to 8 years of expertise in the design, development, and optimization of large-scale data solutions.
• The ideal candidate will possess robust proficiency in Databricks, PySpark, Spark SQL, and cloud platforms such as AWS, Azure, and GCP, alongside modern data engineering practices aimed at constructing scalable and high-performance data pipelines.

Key Responsibilities:

• Roles and Responsibilities
• Create, develop, and uphold data pipelines utilizing Databricks.
• Design ETL and ELT workflows for extensive data processing.
• Develop and enhance data transformation logic using PySpark and Spark SQL.
• Implement and oversee Delta Lake for dependable and efficient data storage solutions.
• Engage with both batch and real-time data ingestion frameworks.
• Monitor and refine data pipelines to ensure optimal performance and scalability.
• Collaborate with data analysts, data scientists, and business stakeholders.
• Adhere to data quality, security, and governance standards.
• Diagnose production issues and deliver ongoing support.

Technical Requirements:

• 5 to 8 years of experience in Data Engineering and Big Data technologies.
• Extensive hands-on experience with Databricks, PySpark, and Spark SQL.
• Familiarity with Delta Lake, Databricks Workflows, and Unity Catalog.
• Proficient in Python and SQL.
• Experience with cloud platforms including Azure, AWS, and GCP.
• Understanding of Azure Data Factory (ADF), Apache Airflow, or equivalent orchestration tools.
• Comprehensive knowledge of data warehousing concepts and ETL/ELT methodologies.
• Preferred experience with Kafka or Event Hub for streaming data processing.
• Acquainted with Git, CI/CD pipelines, and DevOps practices.
• Demonstrated strong problem-solving, analytical, and communication abilities.

Preferred Skills:

Technology->Data Engineering->Databricks