Data Engineer (Databricks, AWS)

Job Description

Job Description

Title: Data Engineer (onsite)
Location: Raleigh, NC
Contract: W2 only, 12-month contract with potential for extension or conversion to full time with the client.
Pay: $70/hour + optional medical, dental, vision, 401(k) match

Overview

The Data Engineer will be responsible for designing, developing, and optimizing large-scale data pipelines and analytical solutions supporting Life Sciences Commercial Operations. This role requires deep expertise in Databricks (including Genie), AWS cloud services, and PySpark, with strong domain knowledge of pharmaceutical/biotech commercial datasets such as IQVIA, Symphony, DDD, NPA, Xponent, Claims, Specialty Pharmacy, and CRM/Sales data. The candidate will partner closely with Commercial Analytics, Data Science, and Business Stakeholders to deliver high-quality, scalable data products that enable sales insights, targeting, forecasting, incentive compensation, and field performance reporting.

Key Responsibilities

• Build, optimize, and maintain data pipelines using Databricks, Delta Lake, and Genie-powered workflows.
• Implement LLM-based automation and query generation using Databricks Genie for data exploration and business-user self-service.
• Develop notebooks, jobs, workflows, and ML-ready datasets within the Databricks environment.
• Design and manage data ingestion, storage, and processing using AWS services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Implement secure, scalable architectures following AWS best practices (IAM, VPC, encryption, monitoring).
• Build distributed data transformation pipelines using PySpark for high-volume commercial datasets.
• Optimize PySpark jobs for performance, cost efficiency, and reliability.
• Implement unit testing, CI/CD, and code versioning using Git-based workflows.
• Ingest, harmonize, and model datasets including IQVIA (Xponent, DDD, NPA, LAAD, NSP), Claims (medical, pharmacy), Specialty Pharmacy data feeds, Sales & CRM (Veeva, Salesforce), Roster, Territory, and Alignment datasets.
• Build commercial data marts supporting sales reporting and dashboards, targeting and segmentation, incentive compensation, forecasting, and field performance insights.
• Work with Commercial Analytics, Data Science, IT, and Business teams to translate requirements into scalable data solutions.
• Ensure data quality, lineage, governance, and compliance with industry standards (HIPAA, GxP, SOC2).

Required Skills

• 5–10+ years of experience in Data Engineering or Big Data Analytics.
• Strong hands-on experience with Databricks, Genie, Delta Lake, and MLflow.
• Advanced proficiency in PySpark, Python, SQL, and distributed computing.
• Deep experience with AWS (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proven experience working with Life Sciences Commercial datasets.
• Strong understanding of data modeling, ETL/ELT, and cloud architecture.
• Experience with CI/CD, Git, DevOps, and automated workflow orchestration.
• Demonstrated knowledge of ingesting, harmonizing, and modeling pharmaceutical/biotech datasets.
• Ability to build scalable, high-performance data pipelines.
• Strong analytical and problem-solving skills.
• Excellent communication and documentation abilities.
• Ability to work in fast-paced, cross-functional environments with high attention to data accuracy.

Required Education

• No Education Requirements

Preferred Skills

• Experience with Databricks Unity Catalog and Lakehouse architecture.
• Familiarity with LLM-based automation or AI-assisted analytics.
• Experience supporting Commercial Operations, Sales Leadership, and Field Teams.
• Knowledge of incentive compensation methodologies and targeting algorithms.
• Experience with BI tools (Tableau, Power BI, Qlik).

Why Should I Apply?

This role offers the opportunity to work on cutting-edge data solutions supporting Life Sciences Commercial Operations, with significant impact on sales insights and forecasting. Join a dynamic team leveraging advanced cloud and big data technologies to transform pharmaceutical data analytics.

About CEI:
As a trusted technology partner, CEI delivers solutions that help our customers transform their business and achieve meaningful results. From strategy and custom application development through application management - our technology and digital experience services are tailored to meet each unique need of our customers. Our staffing solutions bring specialized skills to complement our customers' workforce and project requirements.

#ZR
#INDGEN