Pyspark Developer

Job Overview:

• We are in search of an experienced Pyspark Developer with 9 to 11 years of expertise in creating scalable solutions utilizing Pyspark.
• The successful candidate will possess in-depth knowledge of Pyspark, along with experience in Spark, Hadoop, SQL, Hive, Sqoop, and UNIX shell scripting.

Core Responsibilities:

• A minimum of 9 years of hands-on experience in the design and development of large-scale distributed data processing pipelines leveraging PySpark, Hadoop, and associated technologies.
• Proficiency in Pyspark, Hadoop, Spark Core, Spark SQL, batch processing, and Spark Streaming is essential.
• Experience with Hadoop HDFS, Hive, and other Big Data technologies is required.
• Understanding of data warehousing and ETL methodologies and practices.
• Familiarity with UNIX shell scripting is advantageous for scheduling and executing application jobs.
• At least 5 years of experience in various project development life cycle activities, including development and maintenance projects.
• Collaborate with business stakeholders and subject matter experts to gather high-level business requirements.
• Engage with Solution Designers to aid in the development of project plans, contributing to the scoping and estimation of prospective projects.
• Leverage technical expertise and business acumen to elicit systems requirements for projects effectively.
• Strong knowledge of Spark architecture and transformation processes using Spark and PySpark.
• Participate in an Agile setting, attending scrum daily stand-ups, sprint planning sessions, reviews, and retrospectives.
• Comprehend project requirements and convert them into technical solutions that align with project quality standards.
• Demonstrate the ability to collaborate within diverse teams and with multiple stakeholders while troubleshooting and resolving data-related issues.

Technical Qualifications:

• Core competencies include:
• Pyspark, Hadoop, Spark Core, Spark SQL, batch processing, and Spark Streaming.
• Familiarity with Hadoop HDFS, Hive, and various Big Data technologies.
• Strong analytical and problem-solving abilities.
• Excellent verbal and written communication skills.
• Interest and experience in working within a global delivery environment.
• Commitment to staying current with emerging technologies and industry trends in development.

Additional Duties:

• Infosys stands as a global frontrunner in next-generation digital services and consulting.
• Within the Data Analytics division, you will engage in pioneering data engineering and analytics projects for global clients spanning various industries.
• This role provides the opportunity to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and extensive enterprise data modernization efforts.
• Employees benefit from ongoing learning programs, certifications, and career advancement opportunities within the Infosys data and analytics practice.

Preferred Skills:

Technology->Big Data - Data Processing->PySpark