Databricks Architect

Job Overview:

• We are in search of a seasoned Databricks Snowflake and PySpark Architect to conceptualize, design, and implement scalable cloud-based data platforms along with cutting-edge analytics solutions.
• The perfect candidate will possess extensive knowledge in data architecture, cloud-native technologies, data engineering best practices, and large-scale data transformation initiatives.
• This role requires collaboration with business stakeholders, solution architects, and engineering teams to construct high-performance, secure, and cost-effective data ecosystems utilizing Databricks, Snowflake, and PySpark.

Core Responsibilities:

• Establish and promote enterprise-scale data architecture and modernization strategies.
• Create comprehensive cloud data platforms utilizing Databricks, Snowflake, and contemporary data engineering frameworks.
• Design scalable batch and real-time data processing pipelines for extensive datasets.
• Oversee the implementation of data lakes, lakehouses, and data warehouses.
• Offer architectural expertise on performance enhancement, scalability, governance, security, and reliability.
• Engage with business stakeholders to grasp analytical and reporting needs.
• Evaluate technical designs, perform architecture assessments, and set best practices.
• Guide and mentor data engineers and technical leads throughout project engagements.
• Direct technology assessments, proof of concepts (POCs), and solution accelerators.
• Facilitate cloud migration and modernization efforts from legacy data architectures.
• Guarantee compliance with enterprise security, governance, and data standards.
• Collaborate with DevOps and platform teams to deploy CI/CD and automate infrastructure for data solutions.

Technical Qualifications:

• Minimum of 11 years of experience in Data Engineering, Analytics, or Data Platform Architecture.
• Deep knowledge of the Databricks Lakehouse Platform.
• Significant hands-on experience with Snowflake Data Cloud.
• Advanced proficiency in PySpark and distributed data processing.
• Extensive experience in designing enterprise-scale data lakes, data warehouses, and lakehouse architectures.
• Practical experience with cloud platforms, including:
• Azure: ADF, ADLS Gen2, Synapse, Azure Databricks
• AWS: S3, Glue, EMR, Redshift
• GCP: BigQuery, DataProc, Cloud Storage
• Strong SQL and data modeling capabilities, including Dimensional Modeling, Data Vault, and Star Schema.
• Experience in developing ETL/ELT frameworks and reusable data engineering accelerators.
• Acquainted with Data Governance, Metadata Management, Data Lineage, and Data Quality frameworks.
• Familiarity with orchestration tools such as Airflow, Azure Data Factory, or similar.
• Expertise in performance tuning and cost optimization within Databricks and Snowflake environments.
• Experience in setting up CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar tools.

Additional Duties:

• Enterprise Data Architecture
• Solution Design Governance
• Technology Strategy Roadmap
• Presales Client Advisory
• Architecture Reviews
• Data Platform Modernization
• Technical Leadership and Mentoring
• Cost Optimization and Non-Functional Requirements
• Stakeholder Engagement

Desired Skills:

Technology->Data Engineering->Databricks