PySpark Developer

Job Overview:

• We are seeking a talented PySpark Developer with a maximum of 3 years of experience in the development and maintenance of large-scale data processing frameworks.
• The perfect candidate will possess practical experience in PySpark, Python, SQL, and Big Data technologies, and will be adept at creating efficient ETL workflows for enterprise-level data platforms.

Primary Responsibilities:

• Design, develop, and sustain data pipelines utilizing PySpark.
• Create ETL and ELT processes for the ingestion, transformation, and loading of substantial data volumes.
• Craft optimized PySpark code for effective data processing and transformation tasks.
• Handle structured and semi-structured data from various sources.
• Develop and enhance SQL queries for data extraction and validation purposes.
• Identify and resolve data quality and performance challenges.
• Collaborate with Data Engineers, Analysts, and Business teams to ascertain project requirements.
• Engage in code reviews and adhere to data engineering best practices.
• Oversee and support production-level data pipelines.

Technical Qualifications:

• Proven experience with PySpark.
• Solid programming expertise in Python.
• Practical knowledge of SQL.
• Familiarity with ETL and ELT methodologies and data warehousing principles.
• Experience in managing large datasets and distributed processing environments.
• Understanding of Spark SQL, DataFrames, and Spark transformations.
• Proficiency in a Linux/Unix operating environment.

Additional Responsibilities:

• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of Databricks.
• Experience with workflow orchestration tools like Airflow.
• Understanding of CI/CD principles.

Preferred Skills:

Technology->Big Data - Data Processing->PySpark