Senior AI Data Engineer

Why Karsun? Join Karsun Solutions to grow your career with the company transforming possible for the US Government.At Karsun, collaboration drives our community. We're committed to building an environment where team members from diverse backgrounds can innovate, learn and grow with us. Here at Karsun, the only limit to your potential is the limit of your curiosity.Join Team Karsun, and Find Your Next! Summary We are seeking a highly skilled and motivated Sr. AI Data Engineer with a proven track record in building scalable data platforms and incorporating Generative AI into data engineering workflows. The ideal candidate will have deep expertise in Databricks capabilities-including Delta Lake and Unity Catalog-to power AI and machine learning initiatives. You will play a pivotal role in setting up and operationalizing MLOps directly within Databricks, while seamlessly integrating a variety of open-source tools to enhance data quality, workflow automation, and metadata generation. What You'll Be Doing: * Databricks AI Solutions: Design, build, and maintain scalable data pipelines and workflows using Databricks to directly support AI/ML and analytics workloads. Leverage core capabilities like Delta Lake, Delta Live Tables, and Databricks Workflows to create high-performance data platforms. * MLOps Operationalization: Set up, establish, and operationalize MLOps practices directly within the Databricks environment, including version control, CI/CD for data pipelines, automated testing, and model deployment strategies. * Open-Source Integration: Utilize and integrate open-source tools such as Python, PySpark, and Apache Airflow for distributed data processing and workflow orchestration. * GenAI-Enhanced Workflows: Implement GenAI-enhanced workflows using LLMs to automate metadata generation, create data dictionaries, validate data quality, and track data lineage. * Architecture & Governance: Leverage medallion architecture (Bronze, Silver, Gold layers) following data lakehouse best practices. Integrate Unity Catalog for enterprise data governance and access control. Implement and operationalize best practices. * Data Preparation: Collaborate with AI/ML teams to curate, prepare, and serve high-quality datasets for model training and inference.Required Qualifications: * BA or BS degree in Computer Science, Computer Engineering, Data Science, or a related field (Master's degree is a plus). * Open-Source Proficiency: 5+ years of strong proficiency in open-source languages and frameworks, specifically Python and PySpark, for distributed data processing. Strong knowledge of open-source data orchestration tools like Apache Airflow. * AI/Data Engineering: 5+ years of proven experience building large-scale data platforms, with at least 2+ years incorporating Generative AI into data engineering workflows. * Databricks Expertise: 3+ years of hands-on experience with the Databricks platform, specifically leveraging data engineering and AI features (Delta Lake, DLT, Workflows, Unity Catalog). * MLOps: Proven experience setting up, maintaining, and operationalizing MLOps frameworks within Databricks. * Cloud & Architecture: 3+ years of experience with AWS data services (e.g., S3, Glue, Lambda) and a deep understanding of data lakehouse architecture. * Certifications in Databricks Data Engineer Associate/Professional or AWS Data Analytics.Preferred Qualifications: * Experience with open-source streaming data processing tools like Apache Kafka or Structured Streaming. * Familiarity with open-source data quality and analytics engineering tools such as dbt (data build tool), Great Expectations, or Sweetviz. * Experience with open-source containerization (Docker) and orchestration (Kubernetes) for data applications. * Understanding of vector databases and embedding pipelines for AI/ML applications.Things to Know:Commitment to Non-DiscriminationAll qualified appl