Data Engineer

Job Overview: Pay Range: $50.00hr - $55.00hr Requirement/Must Have: 5+ years building batch and/or streaming pipelines with Spark. Deep understanding of Spark fundamentals including lazy evaluation, transformations vs. actions, partitions, shuffles, joins, and caching. Experience with Databricks platform on Azure including Unity Catalog (catalogs/schemas/tables, permissions, external locations, storage credentials, lineage). Proficiency with Databricks Jobs/Workflows for task orchestration, cluster policies, retries, and parameters. Experience with Git integration (Repos), CI/CD, and environment promotion. Knowledge of cloud storage and access on Azure, specifically secret management via Databricks secret scopes and Key Vault integration. Experience with Delta Lake including ACID transactions, time travel, MERGE/UPDATE/DELETE, and schema enforcement/evolution. Strong SQL skills including complex SQL with CTEs and window functions. Experience with data modeling and architecture using Medallion (bronze/silver/gold) patterns for data lakes. Ability to design tables for CDC and incremental processing. Experience with large scale data processing including cluster sizing, autoscaling, and cost optimization. Familiarity with observability tools such as Spark UI, metrics, event logs, and Delta logs. Knowledge of software engineering practices including Python packaging, code quality, and testing (pytest, chispa). Strong verbal and written communication skills. Responsibilities: Build and maintain batch and streaming data pipelines using Spark and PySpark. Manage Databricks platform components including Unity Catalog and Workflows. Implement data modeling strategies such as partitioning and file layout to match read patterns. Ensure data governance, security, and compliance within the Databricks environment. Execute CI/CD processes for Databricks using Repos and asset bundles. Collaborate with stakeholders and provide technical documentation. Nice to Have: Performance tuning: broadcast joins, AQE, skew mitigation, and shuffle optimization. Structured Streaming and Auto Loader for incremental ingestion. Declarative pipelines with Delta Live Tables (DLT). ADLS Gen2: ABFS access, ACLs vs. RBAC, and managed identity. OPTIMIZE, Z ORDER, VACUUM and retention policies. Compaction and small files mitigation strategies. Query optimization and execution plan analysis in Databricks SQL. Lakehouse design trade-offs and table constraints. Handling TB scale datasets and throughput vs. latency trade-offs. Privilege model in Unity Catalog; table, row, and column level security patterns. PII handling, masking/anonymization, and auditability. Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, and SCD patterns. Skills: Spark. PySpark. Azure Databricks. Unity Catalog. Delta Lake. SQL. Python. Azure Data Lake Storage (ADLS).

CI/CD.

Data Modeling. Java. Qualification And Education: 10-12 years of professional experience in data engineering or related fields. Extensive experience with the Azure cloud ecosystem. Cynet Systems