Data Scientist

Data Scientist

Indianapolis, IN (One day in office 4 days remote)

Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; ...or

Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR..

Relevant certifications and coursework may substitute for formal education/experience requirements

Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions

ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)

Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT

Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials

Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI

Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control

Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), column store indexes, and partitioning

Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views

Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2

Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)

Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing

ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault

Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles

Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs

Strong written and verbal communication skill

Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Databricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)

Microsoft Certified: Azure Data Engineer Associate (DP-203) if already held

Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance