Data Lead

Data Lead

Location: Fremont, CA (Remote)

Mandatory Skills: Azure Data Factory (ADF), Azure Databricks & PySpark, Azure Synapse, Azure SQL, Python, and Spark SQL

Experience Requirements:

• 10 + years of IT experience in the Development/ Architecture Role
• 4+ years of Proven experience in Data Engineering and architecture roles (with Azure Data Factory, Azure Databricks & PySpark, Azure Synapse and Azure SQL).

Qualifications:

Bachelor's or master's degree in computer science, Engineering, or a related field.

Skill Requirements:

• Strong Experience with ETL/ELT tools like ADF, Informatica, Talend etc., and data warehousing technologies like Azure Synapse, Azure SQL, Amazon Redshift, Snowflake, Google Big Query etc.
• Strong hands-on experience with Azure Data Factory (ADF) for data orchestration (for building and managing pipelines), Azure Databricks for big data processing and analytics, and Apache Spark for distributed data processing).
• Adept in with big data tools (Databricks, Spark etc..)
• Experience with Power BI, Tableau/OBIEE etc.
• Proficiency in PySpark, Python, and Spark SQL.
• Experience with CI/CD pipelines for data platforms.
• Solid understanding of data warehouse best practices, development standards, and methodologies.
• Strong expertise in SQL and relational data modeling.
• Experience designing data lakes, data warehouses, or Lakehouse architectures.
• Good understanding of data governance, metadata management, and security best practices.
• Solid understanding of distributed data processing and performance tuning.
• Experience with ETL/ELT patterns and best practices.
• Strong analytical, problem-solving skills and ability to work in a fast-paced, dynamic environment.
• Excellent communication and documentation skills.

Key Responsibilities:

• Design and implement end-to-end data architecture for batch and real-time data processing to implement enterprise-grade data solutions using Azure and Microsoft Fabric.
• Design, build, and optimize ETL/ELT pipelines for ingestion, transformation, and publishing using ADF, Spark, and Databricks.
• Architect and optimize data pipelines using Azure Data Factory (ADF).
• Develop and manage scalable data processing frameworks using Databricks and Apache Spark.
• Develop efficient data transformation logic using PySpark and Spark SQL.
• Build reusable, high-performance data models for analytics and reporting.
• Develop and maintain data ingestion, transformation, and orchestration workflows.
• Ensure data quality, consistency, security, and governance across platforms.
• Define and enforce data engineering best practices, including CI/CD, versioning, testing, and monitoring.
• Mentor and guide data engineers, fostering a culture of innovation and excellence.
• Collaborate with data engineers, analysts, data scientists, and business stakeholders.
• Evaluate and integrate new tools and technologies to enhance data capabilities.
• Own the data lifecycle, from ingestion and transformation to consumption and visualization.
• Support production deployments, monitoring, and troubleshooting of data pipelines.
• Document data architecture, processes, and best practices.

Nice to Have:

• DevOps & CI/CD: Azure DevOps, GitHub Actions, Jenkins
• Real-time streaming (Azure Event Hubs, Kafka), Microsoft Fabric

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.