Lead Software Engineer

Responsibilities

• Lead the design and delivery of scalable PySpark and SQL pipelines that transform market risk data into analytics-ready datasets.
• Define reusable engineering patterns covering data contracts, semantic definitions, quality checks, lineage, testing, deployment, and operational support.
• Build and optimize data models and views for interactive reporting, visualization, analysis, and business decision-making.
• Partner with stakeholders and risk subject-matter experts to clarify reporting needs and translate requirements into maintainable solutions.
• Guide architecture and implementation across batch processing, lakehouse storage, and reporting or visualization integration.
• Establish responsible AI-assisted development practices with human review and validation of generated code and analysis.
• Apply AI and automation to requirements analysis, pipeline development, test generation, data quality investigation, and reporting workflows where appropriate.
• Develop secure, resilient production code; lead code reviews; and improve testing, observability, deployment, and support practices.
• Mentor engineers through technical coaching, design reviews, pairing, and feedback.
• Resolve performance, scalability, data quality, and operational issues across large data workloads.
• Communicate technical options, delivery risks, and recommendations to business partners, engineers, and senior leaders.

Requirements

• A bachelor’s or advanced degree in computer science, data science, engineering, applied mathematics, or a related technical discipline, or equivalent practical experience.
• Significant experience designing, building, and operating production data-intensive applications, including technical leadership of complex deliveries.
• Expert-level Python and PySpark skills, strong SQL skills, and a solid understanding of distributed data processing and Spark performance.
• Hands-on experience with a modern lakehouse or cloud data platform such as Databricks and Delta Lake, plus cloud object storage such as Amazon S3.
• Strong knowledge of data modeling, analytical data products, data quality, lineage, access controls, and enterprise governance.
• Experience designing reporting, visualization, or analytical data products and translating business definitions into consistent datasets.
• Ability to lead technical design, make pragmatic architecture decisions, and communicate trade-offs to varied audiences.
• Experience with automated testing, code review, production monitoring, operational support, and deployment practices for data pipelines.
• Experience using approved AI-assisted software development tools with disciplined validation of generated code and recommendations.
• Strong collaboration skills and ability to work with business and technical teams in an agile delivery environment.
• Familiarity with financial markets, market risk, or other complex financial data domains, or ability to learn the domain quickly.
• Preferred experience with Databricks workflows or pipelines, Unity Catalog, cloud-native data services, reporting and visualization tools, data product or data mesh practices, LLMs or agentic techniques, APIs or services for governed analytical data, market risk measures, infrastructure as code, data observability, and performance or cost optimization at scale.

Benefits

• Competitive total rewards package with base salary determined by role, experience, skill set, and location.
• Eligible roles may receive commission-based pay and/or discretionary incentive compensation in cash or forfeitable equity.
• Benefits may include comprehensive health care coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching.
• Additional compensation and benefits details are provided during the hiring process.