SPSparta Systems
Lead Data Engineer
Bangalore ₹2-5 LPA Posted 28 Jul 2025
FULL TIME
snowflake
Hive
Oracle
Postgresql
Sap
+3 more
Job Description
Key Responsibilities:
Solution Architecture & Development:
- Architect and design large enterprise-level analytics solutions, leveraging big data platforms such as Databricks, Snowflake, Hive, Hadoop, and PostgreSQL.
- Develop data pipelines for big data ingestion using tools like Spark, Informatica PowerCenter & BDM, and Apache NiFi/Airflow for continuous data flow automation.
- Work with enterprise systems such as ERPs (SAP, Oracle), RDBMS, CRMs, and logistics tools to create analytics solutions that improve business functions across the organization.
- Ensure scalability, optimization, and automation for all data solutions to meet high-performance and high-availability requirements.
- Develop APIs, microservices, and platform-based solutions for data integration, enabling seamless data interaction across systems and platforms.
Collaboration & Leadership:
- Lead a cross-functional team of data engineers, data architects, data scientists, DevOps, scrum masters, and product owners to build and deploy end-to-end solutions.
- Take ownership of data engineering design and data management best practices, including dimensional modeling, data warehousing, and data mining.
- Collaborate with leadership to understand business needs and propose optimal data-driven solutions.
- Ensure continuous improvement of data processes, focusing on efficiency, automation, and reducing manual work.
Big Data & Cloud Infrastructure:
- Lead the deployment of big data ingestion jobs using Spark and other industry-standard tools on Hadoop, Databricks, and NoSQL platforms.
- Ensure the integration of cloud-based solutions using Azure and AWS platforms, enabling seamless scaling and flexibility.
- Work with IoT data and ensure the integration of data sources for analytics solutions.
Advanced Analytics & Data Science:
- Leverage your expertise to integrate advanced analytics and machine learning capabilities into the platform for predictive and prescriptive analytics.
- Build solutions that harness big data for actionable insights, forecasting, predictive modeling, and other analytics needs.
Process Improvement & Automation:
- Identify opportunities for process optimization and automation, ensuring that all data operations are efficient and reduce the manual effort involved.
- Work towards building a data-driven culture within Honeywell Connected Enterprise, fostering an environment that embraces automation and continuous improvement.
Platform Development:
- Contribute to the creation of a new platform by working on both frontend and backend aspects, including data storage, APIs, microservices, and data orchestration.
- Enable seamless integration between existing and new platforms, ensuring a robust and scalable architecture.
Qualifications:
Must Have:
- Bachelor's/Master's degree in Computer Science, Engineering, Applied Mathematics, or a related field.
- 12+ years of experience in data engineering, data design, and enterprise data management.
- Strong expertise in architecting large-scale analytics projects and big data platforms (Databricks, Snowflake, Hadoop, etc.).
- 4+ years of hands-on experience with tools like Spark, Informatica PowerCenter & BDM, Apache NiFi/Airflow, and cloud platforms like Azure and AWS.
- Experience in dimensional modeling, data warehousing, and data mining.
- Strong knowledge of big data ingestion, processing, and optimization, with a focus on scalability and performance.
- Expertise in cloud-based data solutions (Azure, AWS), with experience in developing, deploying, and managing cloud data pipelines.
- Proven experience in data integration, building APIs, and working with microservices architectures.
Nice to Have:
- Experience with industrial data (IoT, SAP, and other operational systems).
- Proficiency in data science techniques, including predictive modeling, machine learning, and AI-based analytics.
- Knowledge of advanced data analytics platforms such as Power BI or Tableau.
- Ability to work with non-relational databases (Cassandra, NoSQL, etc.) and experience in advanced query optimization.
