Data & AI Engineer

Job Title: Data & AI Engineer
Location: Houston, TX 77079 or Austin, TX 78752
Duration: Long-Term Contract
Work Authorization: US Citizen or Green Card Holders

***This is a hybrid position in Austin or Houston***

Data & AI Engineer
SQL Server • Azure Databricks • Data Integration • Vector Databases
Key responsibilities

• Design, build, and maintain data ingestion pipelines from SQL Server into Azure Databricks using CDC-based and batch patterns
• Implement and manage medallion architecture (Bronze, Silver, Gold) using Delta Lake and Unity Catalog
• Write and optimize complex T-SQL queries, stored procedures, and CDC configurations on Microsoft SQL Server
• Develop PySpark and Spark SQL transformations for large-scale data processing and curated analytical layers
• Build and maintain vector database pipelines that transform structured and unstructured source data into embeddings for downstream AI and search applications
• Collaborate with BI and analytics teams to deliver curated data models, dashboards, and AI/BI Genie experiences
• Configure and troubleshoot connectivity between SQL Server, Databricks, and third-party connector applications
• Monitor pipeline health, implement alerting, and resolve data quality and performance issues proactively
• Maintain technical documentation for pipelines, schemas, and architectural decisions

Required qualifications
SQL Server (MSSQL)

• 3+ years of hands-on experience with Microsoft SQL Server (2016 or later)
• Proficiency in T-SQL including CTEs, window functions, dynamic SQL, and query optimization
• Experience configuring and managing Change Data Capture (CDC) for incremental data extraction
• Ability to read execution plans; experience with indexing strategies and statistics management
• Working knowledge of isolation levels, locking behavior, and blocking resolution (RCSI, snapshot isolation)
• Understanding of replication topologies and their impact on downstream pipeline design

Azure Databricks

• 2+ years building notebooks, jobs, and workflows in Azure Databricks
• Hands-on experience implementing Bronze/Silver/Gold medallion architecture using Delta Lake
• Proficiency in PySpark and Spark SQL for large-scale data transformation
• Experience with Unity Catalog for governance and access control
• Experience implementing CDC-based incremental pipelines using watermarks or Delta merge patterns
• Familiarity with cluster configuration, compute management, and job scheduling

Data integration & connectors

• Experience configuring JDBC/ODBC connectivity between SQL Server and cloud compute platforms
• Familiarity with one or more connector/orchestration platforms (Azure Data Factory, Fivetran, dbt, or similar)
• Understanding of Azure networking (VNet peering, NSGs, private endpoints)
• Experience with secret management using Azure Key Vault and Databricks secret scopes

Vector databases & AI infrastructure

• Experience designing and managing vector databases (Pinecone, Weaviate, Chroma, pgvector, or similar)
• Understanding of embedding models and how to generate, store, and query vector representations of data
• Familiarity with similarity search concepts (cosine similarity, ANN indexing such as HNSW or IVF)
• Experience integrating vector stores into retrieval pipelines (RAG patterns, semantic search, or recommendation systems)

General engineering

• Proficiency in Python for pipeline development and automation
• Version control experience using Git
• Strong written communication skills; ability to produce and maintain technical documentation
• Comfortable working within Azure cloud environments

Preferred qualifications

• Experience with Tableau, Power BI, or similar BI tools as a downstream data consumer
• Familiarity with ERP source systems (field service management, inventory, or financial platforms)
• Experience with dbt for transformation layer management
• Knowledge of cloud cost optimization strategies for Databricks compute
• Microsoft Certified: Azure Data Engineer Associate or equivalent certification
• Exposure to LLM orchestration frameworks such as LangChain or LlamaIndex

About INSPYR Solutions
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.


Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions' Privacy Policy and INSPYR Solutions' AI and Automated Employment Decision Tool Policy: By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.