Big Data Developer

Experience: 3 – 10 Years

Job Requirements

Core Skills & Expertise

• Comprehensive knowledge of concurrent software systems to ensure scalability, maintainability, and robustness.
• Proficient in designing application solutions within the Hadoop ecosystem.
• Extensive experience with Hive, HDFS, YARN, Spark, Spark SQL, Scala, and PySpark.
• Ability to manage HDFS file formats including Parquet, ORC, Sequence, along with their respective applications.
• Strong grasp of data warehousing systems.
• Skilled in scripting languages like Shell and Python.
• Familiar with the HortonWorks distribution and SQL engines such as Tez, MR.
• Knowledge of Java, RESTful services, and Maven is advantageous.

Automation & Deployment

• Experience with Control-M development and resource monitoring via Grafana.
• Capable of developing automation scripts in Jenkins for builds, test frameworks, and application settings.
• Practical experience in deploying scalable applications using tools such as Bitbucket, Jenkins, and Azure DevOps (ADO).

Skill Set

Mandatory Skills:

• Big Data Technologies: Hive, HDFS, Spark, Scala, PySpark

Good to Have:

• Schedulers: Control-M
• ETL Tools: Dataiku
• Scripting & OS: Unix/Shell scripting
• Integration Services: FileIT, MQ, and others
• CI/CD Tools: Jenkins, Jira, ADO DevOps suite