Principal Architect

Job Description

Principal Architect | AWS & Databricks

Experience: 15+ years

Role Overview

We are seeking a Principal Architect with 15+ years of cloud architecture experience and strong hands-on expertise designing and implementing AWS and Databricks solutions. This person will lead the architecture of scalable, secure, and high-performance cloud data platforms while working across infrastructure, data engineering, governance, and DevOps.

Key Responsibilities

\n
• Design scalable, secure, and high-performance AWS and Databricks solutions using AWS Networking, Kinesis, ElastiCache, Redshift, EKS, Cognito, and other PaaS services.
\n
• Build and maintain ETL/ELT pipelines using Python and PySpark within Databricks, including notebooks, jobs, Delta Lake, and Unity Catalog.
\n
• Implement Bronze, Silver, and Gold medallion architecture and optimize Spark jobs for performance, scalability, reliability, and cost.
\n
• Use Terraform to provision and manage Databricks workspaces, clusters, jobs, S3/ADLS storage, networking, IAM, and related infrastructure.
\n
• Develop advanced SQL for data transformation, validation, analytics, and data quality.
\n
• Build CI/CD pipelines using Jenkins and GitHub Actions.
\n
• Integrate data from databases, APIs, streaming platforms, and file-based sources.
\n
• Optimize cloud costs through autoscaling, spot instances, scheduling, and resource management.
\n
• Monitor pipelines, troubleshoot failures, and implement observability and alerting.
\n

Required Skills

\n
• AWS Networking and VPC design
\n
• AWS Kinesis
\n
• AWS ElastiCache
\n
• AWS Redshift
\n
• AWS Lambda, S3, and Glue
\n
• AWS EKS
\n
• AWS Cognito
\n
• Databricks, Delta Lake, Unity Catalog, and Delta Live Tables
\n
• Python and PySpark
\n
• Advanced SQL
\n
• Terraform
\n
• Jenkins and GitHub Actions
\n

Preferred Skills

\n
• Azure Data Lake, Data Factory, Synapse, and Key Vault
\n
• Big data and dimensional modeling
\n
• Lakehouse architecture
\n
• Spark and Databricks performance tuning
\n
• Data governance, lineage, and compliance
\n

Qualifications

\n
• Bachelor's degree in Computer Science, Information Technology, or related field
\n
• AWS Certified Solutions Architect - Professional preferred
\n
• Databricks Certified Data Engineer Professional or Apache Spark certification preferred
\n