Snowflake Architect

Snowflake Architect

Data Lake Architecture and Experience:

• Guide team to migrate from on Prem Cloudera to Azure cloud environment.
• Designed and implemented scalable data lake solutions using Snowflake and Databricks, Developed and optimized data pipelines for ingestion, transformation, and storage.
• Managed data governance, quality, and security across cloud environments and Implemented performance tuning, automation, and CI/CD for data workflows.
• Collaborated with cross-functional teams to support cloud migration activities

Cloudera Cluster Management:

• Install, configure, manage, and monitor Cloudera Hadoop clusters, ensuring high availability, performance, and security. This includes managing HDFS, YARN, and other ecosystem components.

Performance Optimization:

• Tune Hadoop, Hive, and Spark jobs and configurations for optimal performance, efficiency, and resource utilization. This includes optimizing queries, managing partitions, and leveraging in-memory capabilities.

Troubleshooting and Support:

• Diagnose and resolve issues related to Linux servers, networks, cluster health, job failures, and performance bottlenecks. Provide on-call support and collaborate with other teams to ensure smooth operations.

Security, Governance, and Secrets Management:

• Implement and manage security measures within the Cloudera environment, including Kerberos, Apache Ranger, and Atlas, to ensure data governance and compliance.
• Setup and manage Hashi Corp Vault for secure keys and secrets management.
• Utilize CyberArk for privileged access management and secure administrative tasks on the cluster.

Data and Application Migration:

• Migrate Hadoop, Hive, and Spark data and applications to Azure cloud services such as Azure Synapse Analytics, Azure Databricks, or Snowflake. Ensure data integrity, performance tuning, and validation.
• Automation and Scripting: Develop scripts (e.g., shell, Ansible, Python) for automating administrative tasks, deployments, and monitoring. Work with users to develop, debug, optimize Hive/Spark/Python programs that connect to the Cloudera environment.

Documentation:

• Create and maintain documentation for system configurations, operational procedures, and troubleshooting knowledge bases.

Vendor Collaboration:

• Work closely with the Cloudera vendor to stay current with the latest releases, perform upgrades, and address vulnerabilities.