Data Engineer / Data Architect

Job Summary

Design and optimize cloud-based data platforms that enable analytics and data-driven decision making for a global property and casualty insurance organization. Use AWS, PySpark, Python, and Databricks within a hybrid work model to deliver resilient data pipelines, dependable governance, and scalable architectures. Support accurate risk assessment and faster claims processing through high-performing, production-ready data solutions.

Responsibilities

Design and implement scalable data architectures on AWS that support end-to-end data flows for property and casualty insurance underwriting, pricing, claims, and policy administration use cases

Build robust PySpark and Python-based data pipelines in Databricks that ingest, transform, and curate structured and semi-structured insurance data from multiple internal and external sources

Optimize Databricks workflows and Spark jobs to achieve predictable performance, cost efficiency, and reliability for large volumes of policy premium exposure and claims data

Develop reusable data models and canonical data sets that standardize property and casualty insurance entities including policies, coverages, claims, and reinsurance treaties across business lines

Establish and maintain data quality rules, data validation checks, and reconciliation processes that improve accuracy, timeliness, and completeness of critical regulatory and financial reporting datasets

Collaborate with actuaries, underwriters, claims analysts, and product teams to translate analytical and reporting requirements into clearly defined data engineering solutions that deliver measurable business value

Implement secure data access patterns on AWS using appropriate services and configurations that protect sensitive customer and financial information while enabling governed self-service analytics

Create and maintain technical documentation, including data flow diagrams, data dictionaries, and architecture blueprints, to ensure clear traceability from source systems to curated consumption layers

Automate testing, monitoring, and alerting for data pipelines using Python and cloud-native tools to proactively detect data anomalies and pipeline failures before they impact downstream consumers

Partner with enterprise and solution architects to align data engineering solutions with target cloud architecture standards, supporting the modernization of legacy insurance platforms and the decommissioning of redundant assets

Drive continuous improvement by profiling datasets, identifying performance bottlenecks, and applying tuning strategies that shorten data delivery cycles for reporting dashboards and advanced analytics

Support business continuity by implementing backup, recovery, and environment management strategies in AWS and Databricks that reduce operational risk and minimize downtime for critical data services

Mentor peers and junior engineers by sharing best practices in PySpark coding standards, data modeling for insurance, and cloud-based engineering patterns that enhance the overall quality of team deliverables

Qualifications

Possess eight to twelve years of experience in data engineering or data architecture roles with a consistent focus on delivering production-grade solutions in complex enterprise environments

Demonstrate strong hands-on expertise with AWS data and compute services, including data storage, processing, orchestration, and security components used to run modern analytical workloads

Exhibit advanced proficiency in PySpark and Python for building scalable transformation logic, unit tests, and utility frameworks that can be reused across multiple insurance data pipelines

Bring proven experience working with Databricks to develop, optimize, and operate collaborative notebooks, jobs, and clusters tailored for large-scale property and casualty data processing

Show deep functional understanding of property and casualty insurance concepts, including policies, coverages, rating factors, claims lifecycle, and regulatory reporting, and apply this knowledge to data designs

Apply solid knowledge of dimensional and relational data modeling practices to design curated data marts and semantic layers that serve analytics, actuarial studies, and operational reporting

Utilize strong skills in SQL data profiling and data quality assessment to diagnose source system issues and drive corrective actions with business and technology stakeholders

Demonstrate effective communication and collaboration abilities to work in a hybrid model with distributed teams, conveying complex data topics in clear and concise business language

Display familiarity with modern DevOps and DataOps practices such as version control, continuous integration, automated testing, and environment promotion for data engineering solutions

Show commitment to continual learning by staying updated on evolving AWS, Databricks, and open source data engineering capabilities and by actively incorporating relevant innovations into project delivery