Data Engineer

Job DescriptionJob Description Job Title: Data Engineer Location-Type: NYC Hybrid (7 days onsite/ month) Start Date Is: Mid August Duration: 6 Month Contract Compensation Range: $65-72/hour W2 Benefits: Eligible for Health, Dental, Vision, 401K Must be authorized to work in the U.S. This position is not eligible for sponsorship . Our client is seeking a motivated and detail-oriented Data Engineer to join our growing data team. In this role, you will work closely with senior data engineers, data scientists, and analysts to support the development, maintenance, and optimization of our data infrastructure. You will play a crucial role in ensuring the accuracy, availability, and performance of our data systems, which underpin our research and analysis efforts. Key Responsibilities: 1. Data Pipeline Development: a. Assist in the design, development, and maintenance of scalable data pipelines to process and integrate data from various sources. b. Implement ELT (Extract, Load, Transform) processes to ensure data is collected, cleaned, and stored accurately. 2. Database Management: a. Support the management and optimization of databases, including data warehousing solutions. b. Ensure data integrity, security, and accessibility. 3. Data Integration: a. Collaborate with data scientists and analysts to integrate new data sources and support data-driven projects. b. Assist in the implementation of data integration solutions and APIs. 4. Performance Optimization: a. Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks. b. Implement performance tuning techniques to improve data processing efficiency. 5. Data Quality and Validation: a. Perform data quality checks and validation to ensure the accuracy and consistency of data. b. Develop and maintain documentation for data processes and workflows. 6. LLM experience: a. Prepare quantitative and qualitative data for easy discovery by LLMs. b. Design and manage vector search infrastructure to support Retrieval-Augmented Generation (RAG) pipelines. 7. Collaboration and Support: a. Work collaboratively with cross-functional teams to understand data needs and provide technical support. b. Contribute to the continuous improvement of data engineering practices and processes. Qualifications: Education: o Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field or a Master's degree in Computer Science, Information Technology, Engineering, or a related field with 3-5 years of professional experience. Technical Skills: o Proficiency in SQL and Python. o Understanding of ELT processes and data pipeline development. o Experience with cloud platforms (Google Cloud, BigQuery) is a plus. o Prior experience with dbt and Air Flow is preferred. o Familiarity with CI/CD (Docker and Terraform) and concepts of data integrity. o Familiarity with data visualization tools (e.g., d3, plotly or Carto preferred). Soft Skills: o Strong analytical and problem-solving skills. o Excellent communication and teamwork abilities. o Detail-oriented with a commitment to accuracy