Key Responsibilities:

Data Architecture: Design and implement scalable storage solutions using Hadoop, NoSQL databases, HBase, and Cassandra.
Big Data Processing: Develop and maintain pipelines using Apache Spark, Apache Storm, Apache Pig, MapReduce, and Oozie.
Scripting & Development: Write and test data processing scripts using Python, Scala, and Java.
System Integration: Integrate big data solutions with existing IT systems and external data sources.
Performance Optimization: Monitor and optimize big data systems for performance, scalability, and availability.
Security & Compliance: Ensure security and privacy of sensitive data.
Collaboration & Mentoring: Work with data scientists and stakeholders; mentor junior engineers and provide technical guidance.
Documentation: Document and communicate technical designs, solutions, and best practices.
Innovation: Stay updated with emerging trends and technologies in big data and data engineering.

Big Data Engineer