Databricks, PySpark, Python - Data Engineer

Job Overview:

• We are looking for an experienced Team Lead in Data Engineering, with 7 to 9 years of expertise in creating and implementing scalable data solutions utilizing Databricks, PySpark, and Python.
• This role entails leading a small team focused on technical delivery and constructing high-performance data pipelines for enterprise analytics platforms.

Primary Responsibilities:

• Oversee the design, development, and implementation of data engineering solutions leveraging Databricks and PySpark.
• Create scalable ETL and ELT pipelines, along with data transformation frameworks.
• Write optimized Python and SQL code for data processing and analytics purposes.
• Collaborate with stakeholders to grasp business needs and convert them into technical solutions.
• Enhance Spark jobs for improved performance, scalability, and cost-effectiveness.
• Ensure adherence to data quality governance and security protocols.
• Review code, mentor team members, and provide technical leadership.
• Assist with production deployments, incident resolution, and root cause analysis.
• Engage in Agile practices and manage sprint deliverables.

Technical Qualifications:

• 7 to 9 years of comprehensive IT experience.
• Extensive hands-on knowledge of Databricks and PySpark.
• Proficient in Python programming and SQL.
• Experience in constructing enterprise-scale data pipelines and data lake architectures.
• Solid grasp of distributed computing principles and Spark architecture.
• Background in performance tuning and optimization techniques.
• Exceptional skills in stakeholder management and team collaboration.

Preferred Expertise:

Technology->Data Engineering->Databricks,Technology->AI-Data science->PYTHON,Technology->Big Data - Data Processing->PySpark