Data Engineer

Data Engineer

Onsite (Bellevue -WA)

Mandatory Primary Skills: Hadoop, Python, Databricks, Clickhouse DB, ADLS, SQL + Azure cloud technologies above all with Azure (such as ADF, Databricks, Log Analytics, Logic App, Key Vault, Blob Storage, ADLS, etc.)

Secondary Skills: Snowflake

Detailed JD as below.

Data Engineer

We run a near real-time analytics platform for mobile network data, ingesting billions of rows a day from Azure Data Lake/HDFS into ClickHouse and serving subscriber-level queries back to internal APIs in tens of milliseconds. We are expanding the platform footprint to support additional use cases by adding more datasets.

Data Engineer would work on: extending the Python ingestion framework that loads from Azure Data Lake/HDFS; table design covering partitioning, sort keys, TTL and materialized views; operational diagnosis of merge behaviour, part counts and replication lag; and integration with Snowflake and Databricks Iceberg alongside the primary feeds.

We need strong analytical SQL and experience with a columnar or MPP database — ClickHouse, BigQuery, Redshift or similar. Python sufficient to own a production ingestion framework rather than write scripts. Comfort at hundreds of billions of rows, since the failure modes differ from smaller systems. Cloud object storage and Parquet. Enough Kubernetes to tell whether a problem is the database or the platform underneath it.

Strongly preferred: direct ClickHouse experience, particularly MergeTree engines and aggregate function states, and telecom data such as DNS, IPFIX, etc. ClickHouse specifically is uncommon, and we would rather have a strong engineer from an adjacent system than a weak one with the exact keyword.