Data Architect - III Data Architect - III

Data ArchitectArchitect scalable data processing frameworks leveraging Databricks platform for analytical workloads. Implement cost-optimization strategies balancing compute resources, storage tiering, and cluster configurations. Develop and maintain coding standards for PySpark/Python within Databricks notebooks ensuring maintainability and performance. Design and architect ETL/ELT pipelines supporting ingestion, transformation, and distribution of analytic datasets. Design Databricks workspace architectures integrating Unity Catalog for centralized governance, AWS IAM for authentication, and cross-account data access patterns. Design data quality frameworks and validation logic within processing workflows. Understanding of complex SQL queries and Spark transformations for data manipulation, aggregation, and analytical computations. Establish patterns for incremental processing, change data capture, and historical data management. Architect MLOps pipelines for model deployment and monitoring using AWS services, Domino Data Lab, and related platforms. Implement CI/CD workflows using GitLab for data pipeline deployment and version control. Establish infrastructure-as-code practices for environment consistency and disaster recovery. Establish data governance frameworks within centralized data lake environment. Design dimensional data models and semantic layers optimized for Tableau and BI consumption. Explore and implement generative AI capabilities. Partner with product owners, architects, data engineers, analysts, and application teams to translate requirements into technical architectures. Mentor team members on best practices for Spark optimization, data modeling, and cloud-native design patterns. Evaluate emerging technologies and platforms for potential adoption into the data ecosystem. Document architectural decisions, design patterns, and technical standards.