Sr Data Architect

Job Description

We are seeking a visionary Senior Data Architect to lead our data analytics initiatives and architect the foundation for our next generation of AI capabilities. In this role, you will be the primary technical authority for our Databricks landscape. You will move beyond simple data management to architecting a scalable "Data Intelligence Platform" that powers Generative AI, Machine Learning, and advanced analytics. You will act as a force multiplier—setting the direction for engineering teams, defining architectural standards, and bridging the gap between complex technical execution and high-level business goals.

Key Responsibilities

• Databricks & AI Architecture Strategy: Define and implement the end-to-end architecture for the Lakehouse, ensuring support for high-volume batch processing, real-time streaming, and GenAI/LLM initiatives using Mosaic AI and MLflow.

• Champion Data Health & Integrity: ○ Design solutions that prioritize data quality at the source. ○ Establish and enforce architectural patterns that ensure reliability and trust in our data assets. ○ Implement automated testing and observability frameworks (e.g., Delta Live Tables expectations) to maintain data health.

• Strategic Data Modeling & Business Translation: ○ Bridge the gap between business goals and technical execution. ○ Translate complex business requirements into robust logical and physical data models. ○ Document and maintain clear data flow architectures for applications to ensure alignment between business intent and engineering output.

• Technical Leadership & Team Enablement: ○ Lead virtual Scrum teams and provide deep technical mentorship to data engineers and developers. ○ Advocate for Spark/Databricks best practices, conduct code reviews, and drive the adoption of CI/CD and DevOps principles in data workflows. ○ Note: You will provide the guidance that junior members rely on for success.

• Enterprise Governance & Lifecycle Management: ○ Architect the frameworks for data lineage, security, and lifecycle management (utilizing Unity Catalog). ○ Ensure strict compliance with internal policies and external regulations from the point of data ingestion to final consumption.

• Stakeholder Management & Value Prioritization: ○ Partner directly with business stakeholders to understand their roadmap and priorities. ○ Ensure all data and AI initiatives are strictly aligned with tangible business value. ○ Manage the architectural roadmap to balance immediate business needs with long-term technical scalability

pay varies depending on experience - 70-75/hr

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• Total IT Experience: 8+ years of proven experience in data engineering, software development, or solution architecture.

• Databricks Specialization: 5+ years of hands-on experience specifically architecting and delivering solutions within the Databricks ecosystem (Delta Lake, Unity Catalog, Photon).

• MUST have hands-on experience with strategy, architecture, long-term problem solving.

• Great collaborative personality and communication - must be able to work across multiple domains/teams within the business, natural leader.

• Must have breadth/depth of experience with architecture - different types of architecture and solutioning, experience with configurations, integrations between different applications (understand how integrations work, how cloud works, How to better expose APIs etc. - need to have big picture mindset and experience.

• Cloud Experience: 5+ years of experience with major cloud platforms (AWS, Azure, or GCP)

• Experience within generative AI

• Proficiency in Python (PySpark) and SQL.

• AI/ML Readiness: Experience architecting infrastructure for AI (Vector Search, Feature Stores, Model Serving) and familiarity with LLM constraints.

• Data Modeling: extensive experience with dimensional modeling (Star Schema, Snowflake) and Data Vault methodologies