Senior PySpark Developer

Job Overview:

• We are in search of a seasoned Senior PySpark Developer, bringing 8 years of expertise in Data Engineering and Big Data technologies.
• The successful candidate will have extensive knowledge in PySpark, Python, Spark SQL, ETL development, Data Warehousing, and Cloud platforms such as AWS, Azure, and GCP.
• This role involves designing, developing, and optimizing complex data processing solutions at scale.

Primary Responsibilities:

• Core Responsibilities
• Create and develop scalable data pipelines utilizing PySpark and Apache Spark.
• Construct and enhance ETL and ELT frameworks to handle significant volumes of both structured and unstructured data.
• Design high-performance Spark applications using Spark SQL, DataFrames, and RDDs.
• Collaborate effectively with data architects and business teams to specify data requirements.
• Establish data quality validation and governance protocols.
• Refine Spark jobs for optimal performance, memory utilization, and scalability.
• Incorporate data from a variety of sources including databases, APIs, cloud storage, and streaming platforms.
• Assist in production deployments and troubleshoot performance issues.
• Guide junior developers and engage in code review processes.

Technical Qualifications:

• Essential Skills
• 8 years of experience in IT.
• Proficient hands-on experience in PySpark and Python.
• Expertise in Apache Spark, Spark SQL, DataFrames, and ETL development.
• Strong understanding of SQL and database fundamentals.
• Familiarity with the Hadoop Ecosystem, including Hive, HDFS, and YARN.
• Experience in Data Warehousing and Data Modeling.
• Practical experience with Unix/Linux environments.

Desirable Skills:

Technology->Big Data - Data Processing->PySpark