Big Data Engineer (Scala, AWS)

Black And White Business Solutions is actively seeking a highly skilled Big Data Engineer with a strong background in Scala and AWS. This role is ideal for a professional with extensive experience in designing, developing, and optimizing scalable big data pipelines using Apache Spark and Scala. You will be crucial in building robust data processing workflows, ensuring data quality, and collaborating with cross-functional teams to deliver high-quality data solutions.

Key Responsibilities

• Design, develop, and optimize scalable big data pipelines using Apache Spark and Scala.
• Build batch and real-time data processing workflows to ingest, transform, and aggregate large datasets efficiently.
• Write high-performance SQL queries to support complex data analysis and reporting requirements.
• Collaborate with data architects, data scientists, and business stakeholders to understand requirements and deliver high-quality, impactful data solutions.
• Ensure data quality, integrity, and governance across all systems and pipelines.
• Participate actively in code reviews and maintain best practices in data engineering throughout the development lifecycle.
• Troubleshoot and optimize performance of Spark jobs and SQL queries to enhance efficiency and reduce processing times.
• Monitor and maintain production data pipelines, performing root cause analysis of data issues to ensure reliability.

Technical Skills

• Overall Experience: 6 to 10 years of overall experience in software/data engineering.
• Apache Spark & Scala: 4+ years of hands-on experience with Apache Spark using Scala.
• Scala Proficiency: Strong proficiency in Scala and functional programming concepts.
• SQL Expertise: Extensive experience with SQL, preferably in distributed databases like Hive, Presto, Snowflake, or BigQuery.
• Hadoop Ecosystem: Experience working in the Hadoop ecosystem (HDFS, Hive, HBase, Oozie, etc.).
• Data Fundamentals: Knowledge of data modeling, data architecture, and ETL frameworks.
• DevOps & Version Control: Familiarity with version control (Git), CI/CD pipelines, and DevOps practices.
• Cloud Platforms: Experience with cloud platforms (AWS, Azure, or GCP) is a significant plus.
• Problem-Solving: Strong analytical and problem-solving skills to tackle complex data challenges.
• Communication: Excellent communication and collaboration abilities to work effectively within a team and with stakeholders.

Qualification

• Any Graduate or Above

Notice Period

• Till 60 days

Contact: SWETHA Black and White Business Solutions Pvt.Ltd. Bangalore, Karnataka, INDIA. Contact Number: 8067432491 Email: [HIDDEN TEXT] | Website: www.blackwhite.in