Databricks Data Developer and Architect (Ref #: 1118803)

Job Description

Job Description

Databricks Data Developer and Architect

Estrada Consulting, Inc. | University research center | Onsite, Clearfield, Utah | $60 to $85 per hour

About the role

ECI is looking for a data developer and architect to build a materials research database from the ground up, onsite. You will get data off lab and manufacturing machines, land it in Azure, and design and build the staging and production layers on Databricks that researchers will use for analytics and data science.

About the project

A university advanced materials research center in northern Utah builds and tests high-temperature carbon fiber composite parts for aerospace. Its 3D printers, ovens, furnaces, microscopes and mechanical and thermal test machines each keep their data on the machine. This six-month engagement builds the center's materials database: pipelines that extract machine logs, sensor readings and microscope images, land them in Azure Blob Storage in an Azure Government account, and stage and model them on Databricks so researchers can analyze their experiments. The team works alongside the center's staff and student interns and hands the system over with training and documentation.

Responsibilities:

• Build extraction from each machine (log files, controller registers, instrument outputs and microscope images) into Azure Blob Storage.
• Create the Databricks staging environment with data types checked and enforced.
• Design and build the production data model for research use, including tables that reference images in blob storage.
• Build orchestration with alerts when loads fail or data is missing or corrupt.
• Support automated reporting and analytics on production data with the center's staff.
• Document the pipelines and data model, and pair with student interns so the center can maintain them.

Required Qualifications:

• Three years or more of data engineering experience.
• Experience building data ingestion pipelines, data models and pipeline monitoring and alerting.
• Strong SQL and Python.
• Cloud experience (Azure, AWS or GCP), including IaaS and PaaS services.
• Data security, permission management and secure networking experience.
• U.S. citizen able to work onsite in Clearfield, Utah, full time.

Preferred Qualifications:

• Databricks, Apache Spark and Delta Lake.
• Microsoft Azure, including Azure Government, Blob Storage and Azure Data Factory or Databricks Workflows.
• Data lake or lakehouse architecture, including storing images in blob storage with references in tables.
• Reading data from Windows servers, log files, PLC or controller registers and lab instruments.
• Experience working with and training student interns.
• Databricks or Azure data engineering certification.

Terms:

Work arrangement: Onsite, full time (40 hours per week) at the research center in Clearfield, Utah. U.S. citizenship is required.

Duration: Six months, possibly renewable, contingent on contract award. The expected start is January 2027.

Rate: $60 to $85 per hour (W2 contract), depending on experience.

Other requirements: The candidate's name and resume will be included in ECI's proposal, with references who can verify data engineering experience.