Principal Data Engineer
InterSources Inc
Role - Principal Data Engineer
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
- ...Title: Data Architect Engineer Duration: 6 month contract (potential to extend or convert) Work Location: Charlotte, NC preferred. If remote, would need to come onsite once per month for 2 days once converted to full time. Job Description The Data Architect Engineer will...SuggestedFull timeContract workLocal areaRemote work
INSPYR Solutions
Fort Mill, York County, SC3 days ago - ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...SuggestedFull timeLocal area
Capgemini
Fort Mill, York County, SC2 days ago - ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World IncSuggested
- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...SuggestedFull timeLocal area
- # Data Engineer**Job Duties:**Design and implement efficient large-scale data solutions in areas data ingestion, data curation, data quality, data visualization and data distribution using Informatica, Snaplogic, Tableau and MicroStrategy. Manage and query data in data...SuggestedFull timeRelocation
Pantar Solutions
Weddington, NC1 day ago $85.68k - $117.81k
...Job Summary Associate Data Science to work with the existing Data Science team to develop models and analyze data. Position Responsibilities... ...or a related field 1–3 years of experience in data science, ML engineering, or a related role (including internships) Proficiency in...Temporary workInternshipFlexible hoursNight shiftWeekend workSunbelt Rentals West Chester
Fort Mill, York County, SC2 days ago$103.2k - $129k
...Principal Mechanical Engineer Are you interested in being part of an innovative team that supports Westinghouse's mission to provide clean energy solutions? At Westinghouse, we recognize that our employees are our most valuable asset and we seek to identify, attract...Temporary workWork at officeRemote workRelocation packageFlexible hoursWestinghouse Electric Company
Rock Hill, SC6 hours ago$103.2k - $129k
...talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Civil Structural Engineer you will support civil and structural design activities like structural systems evaluation for steel members,...Flexible hoursWestinghouse Electric Company
Rock Hill, SC2 days ago- ...Principal Electrical Engineer Are you interested in being part of an innovative team that supports Westinghouse's mission to provide clean energy solutions? At Westinghouse, we recognize that our employees are our most valuable asset and we seek to identify, attract...Work at officeRemote workFlexible hours
Westinghouse Electric Company
Rock Hill, SC3 days ago - ...frontend frameworks. ~ Hands-on experience developing and deploying AI/ML solutions. ~ Experience with LLM integration, prompt engineering, embeddings, and RAG architecture. ~ Strong knowledge of SQL Server, PostgreSQL, Cosmos DB, or MongoDB. ~ Experience with...Full time
Qode
Fort Mill, York County, SC19 hours ago - ...Scholarships and more. Basic Job Functions: Nucor Business Technology (NBT) is seeking qualified applicants for the position of Data Engineer. This position will be responsible for designing and maintaining robust data models that support business intelligence and...Full timeWork experience placement
$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Welding Engineer you will support the development, implementation, and troubleshooting of welding-related requirements and activities for...Contract workFlexible hours2 days per weekWestinghouse Electric Company, LLC
Rock Hill, SC3 days ago- ...Position: Engineering Manager: Utility Structures Location: Rock Hill, SC, United States Our client, a leading manufacturer of utility steel structures is seeking an experienced Engineering Manager to lead a team responsible for the design, estimation, and detailing of...Relocation
Global Edge Group
Rock Hill, SC4 days ago - Operations Manager Surface Experts of SW Charlotte & Rock Hill Lead the Team. Elevate the Standard. Build Something Great. Are you the type of leader who enjoys coaching people, improving processes, and making sure every customer receives an exceptional experience? Surface...Work at office
SURFACE EXPERTS, INC.
Fort Mill, York County, SC7 hours ago - Greystar Worldwide, LLC in Fort Mill, SC seeks a Property Manager to lead day-to-day operations at Fox Hunt Farms. You will manage budgeting, leasing, staff supervision, and tenant relations to achieve occupancy and revenue goals. You will oversee maintenance coordination...Worldwide
Greystar Worldwide, LLC
Fort Mill, York County, SC7 hours ago $20 - $23 per hour
Description Our client is seeking detail-oriented Operations Processing Specialists to support a high-volume operations team. This role is focused on processing transactions and requests, ensuring accurate and timely completion of work within established service level...Contract workTemporary workWork at officeTEKsystems
Fort Mill, York County, SC1 day ago- Ulta Beauty in Fort Mill, South Carolina, is seeking a Task Associate to support operational excellence and enhance guest experience. The role involves maintaining store standards, ensuring compliance with company policies, and contributing to team performance. Candidates...Work experience placementFlexible hours
Ulta Beauty
Fort Mill, York County, SC5 days ago $22.48 - $37.44 per hour
Job Overview The Premier Operations Specialist will provide comprehensive administrative and operational support to the LPL Wealth Advisors Group (WealthAG), a group of corporate employee advisors who provide investment solutions to end‑investors virtually and over the...Work from homeLPL Financial
Fort Mill, York County, SC2 days ago- ...supporting a leading utility in their search for a Project Manager II to join their team onsite in the Southeast. This role provides engineering project management expertise and technical direction for small to medium-scale electric substation, transmission, distribution,...Full timeContract workFor subcontractorWork at office
- ...TO THE ROLE ~ Bachelor’s degree in Supply Chain, Industrial Engineering or related degree ~7+ years of Supply Chain Management or related... ...experience ~ Manufacturing related experience ~ Strong data analysis and reporting skills ~ Spanish or Portuguese speaking...Temporary workFor contractorsImmediate startWorldwideRelocationVisa sponsorshipWork visa
Continental
Fort Mill, York County, SC4 days ago - ...activities for the regional and local office organization within Pike Engineering. The incumbent will support client interaction up to and... ...Maintain records, files and databases of projects and operational data and develop reports, charts, graphs, etc. as required. Provide...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC3 days ago - ...AI Infrastructure Engineer Salary: 150K Gross Location: Fort Mill, SC Onsite – 5... ...security — VPC, PrivateLink, encryption, data residency Compliance — SOC 2, HIPAA, FedRAMP... ...— LangSmith, Arize, OpenTelemetry Principal FDE (future / escalation profile): Not...Temporary work
Diverse Lynx
Fort Mill, York County, SC2 days ago - Take on the role of a Warehouse Operations Coordinator and play a vital role in supporting warehouse leadership. Utilize your organizational and communication skills to ensure compliance with DOT requirements and efficient operations. Grow your caree
US Foods
Fort Mill, York County, SC4 days ago - ...AI Infrastructure Engineer (Forward Deployed AI Engineer) Location: Fort Mill, SC (Onsite – 5 Days/Week) Tax Term (W2, C2C): W2 /... ...Dynatrace, Grafana). Cloud security (VPC, PrivateLink, encryption, data residency). Security automation, vulnerability management,...Permanent employmentFull timeContract work
Apolis
Fort Mill, York County, SC2 days ago - ...activities for the regional and local office organization within Pike Engineering. This role supports client interaction, time and expense... ...Maintain records, files, and databases of projects and operational data; develop reports, charts, and graphs as required. Coordinate...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC2 days ago - F. Schumacher & Co. is looking for an HR Assistant to provide essential HR support to local employees. This role includes recruiting, facilitating hiring processes, and managing employee communication and logistics for meetings. The ideal candidate will have 2-3 years of...Local area
- F.-Schumacher- in Fort Mill, South Carolina, is seeking an HR Assistant to support local employees and collaborate with the HR department to cultivate a high-performance culture. Responsibilities include recruitment, onboarding, and managing HR processes. The ideal candidate...Local area
- ...Agentic AI Engineer – API Development Focus Location: Fort Mill, SC Duration: 12Months+ Extension Hourly Rate: Depending on Experience (DOE) Job Description We are seeking a highly skilled Agentic AI Engineer with a strong background in API development to join our innovative...Hourly payPermanent employmentContract workLocal area
ETHEREUM TECHNOLOGIES LLC
Fort Mill, York County, SC3 days ago $107.66k - $161.7k
...working on our Poe product. About the Team and Role: Our small engineering team works on challenging problems every day. We have a culture... ...end ownership of Machine Learning systems -- from prototyping, data pipelines and training, to realtime LLM application at scale...Remote jobFull timeWork experience placementInternshipQuora
Rock Hill, SC1 day ago- ...approach to where we work. About the Role: As a Machine Learning Engineer, you will have the opportunity to collaborate closely with... ...problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine learning...Permanent employmentFull timeWork at officeRemote workWork from homeFlexible hours
Instacart
Rock Hill, SC1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- data modeling Fort Mill, York County, SC
- clinical data coordinator remote Fort Mill, York County, SC
- data Fort Mill, York County, SC
- sap master data Fort Mill, York County, SC
- ai data Fort Mill, York County, SC
- data capturer Fort Mill, York County, SC
- data tech Fort Mill, York County, SC
- data network cabling Fort Mill, York County, SC
- education data Fort Mill, York County, SC
- health data Fort Mill, York County, SC