Principal Data Engineer
InterSources Inc
Role - Principal Data Engineer
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...SuggestedFull timeLocal area
Capgemini
Fort Mill, York County, SC1 day ago - ...Title: Data Architect Engineer Duration: 6 month contract (potential to extend or convert) Work Location: Charlotte, NC preferred. If remote, would need to come onsite once per month for 2 days once converted to full time. Job Description The Data Architect Engineer will...SuggestedFull timeContract workLocal areaRemote work
INSPYR Solutions
Fort Mill, York County, SC2 days ago - ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World IncSuggested
- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...SuggestedFull timeLocal area
- # Data Engineer**Job Duties:**Design and implement efficient large-scale data solutions in areas data ingestion, data curation, data quality, data visualization and data distribution using Informatica, Snaplogic, Tableau and MicroStrategy. Manage and query data in data...SuggestedFull timeRelocation
Pantar Solutions
Weddington, NC5 days ago $85.68k - $117.81k
...Job Summary Associate Data Science to work with the existing Data Science team to develop models and analyze data. Position Responsibilities... ...or a related field 1–3 years of experience in data science, ML engineering, or a related role (including internships) Proficiency in...Temporary workInternshipFlexible hoursNight shiftWeekend workSunbelt Rentals West Chester
Fort Mill, York County, SC1 day ago$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role As a Principal Civil Structural Engineer you will support civil and structural design activities such as structural systems evaluation for steel members,...Flexible hoursWestinghouse Electric Company
Rock Hill, SC3 days ago$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Mechanical Engineer you will lead others in the design and analyses of the AP1000 and AP300 power plants. You will report to the...Permanent employmentFor subcontractorFlexible hours2 days per weekWestinghouse Electric Company
Rock Hill, SC5 days ago- ...Principal Electrical Engineer Are you interested in being part of an innovative team that supports Westinghouse's mission to provide clean energy solutions? At Westinghouse, we recognize that our employees are our most valuable asset and we seek to identify, attract...Work at officeRemote workFlexible hours
Westinghouse Electric Company
Rock Hill, SC2 days ago - ...frontend frameworks. ~ Hands-on experience developing and deploying AI/ML solutions. ~ Experience with LLM integration, prompt engineering, embeddings, and RAG architecture. ~ Strong knowledge of SQL Server, PostgreSQL, Cosmos DB, or MongoDB. ~ Experience with...Full time
Qode
Fort Mill, York County, SC9 hours ago - ...Scholarships and more. Basic Job Functions: Nucor Business Technology (NBT) is seeking qualified applicants for the position of Data Engineer. This position will be responsible for designing and maintaining robust data models that support business intelligence and...Full timeWork experience placement
$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Welding Engineer you will support the development, implementation, and troubleshooting of welding-related requirements and activities for...Contract workFlexible hours2 days per weekWestinghouse Electric Company, LLC
Rock Hill, SC2 days ago$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role: As a Principal Mechanical Engineer you will develop and support nuclear power plant design modifications, engineering evaluations, and technical issue resolution...Full timeTemporary workWork at officeRemote workRelocation packageFlexible hours- ...Position: Engineering Manager: Utility Structures Location: Rock Hill, SC, United States Our client, a leading manufacturer of utility steel structures is seeking an experienced Engineering Manager to lead a team responsible for the design, estimation, and detailing of...Relocation
Global Edge Group
Rock Hill, SC3 days ago $20 - $23 per hour
Position Summary Our client is seeking detail-oriented Operations Processing Specialists to support a high-volume operations team. This role focuses on processing transactions and requests, ensuring accurate and timely completion of work within established service level...Hourly payContract workTemporary workWork at officeTEKsystems
Fort Mill, York County, SC5 days ago- Ulta Beauty in Fort Mill, South Carolina, is seeking a Task Associate to support operational excellence and enhance guest experience. The role involves maintaining store standards, ensuring compliance with company policies, and contributing to team performance. Candidates...Work experience placementFlexible hours
Ulta Beauty
Fort Mill, York County, SC4 days ago - Operations Manager Surface Experts of SW Charlotte & Rock Hill Lead the Team. Elevate the Standard. Build Something Great. Are you the type of leader who enjoys coaching people, improving processes, and making sure every customer receives an exceptional experience? Surface...Work at office
$22.48 - $37.44 per hour
...difference by turning life's aspirations into financial realities. Please log in or create an account to apply to this position. Principals only. EOE. Information on Interviews: LPL will only communicate with a job applicant directly from an @lplfinancial.com...Work from homeLPL Financial Holdings, Inc.
Fort Mill, York County, SC1 day ago- ...supporting a leading utility in their search for a Project Manager II to join their team onsite in the Southeast. This role provides engineering project management expertise and technical direction for small to medium-scale electric substation, transmission, distribution,...Full timeContract workFor subcontractorWork at office
- ...TO THE ROLE ~ Bachelor’s degree in Supply Chain, Industrial Engineering or related degree ~7+ years of Supply Chain Management or related... ...experience ~ Manufacturing related experience ~ Strong data analysis and reporting skills ~ Spanish or Portuguese speaking...Temporary workFor contractorsImmediate startWorldwideRelocationVisa sponsorshipWork visa
Continental
Fort Mill, York County, SC3 days ago - Greystar Worldwide, LLC in Fort Mill, SC seeks a Property Manager to lead day-to-day operations at Fox Hunt Farms. You will manage budgeting, leasing, staff supervision, and tenant relations to achieve occupancy and revenue goals. You will oversee maintenance coordination...Worldwide
- ...activities for the regional and local office organization within Pike Engineering. The incumbent will support client interaction up to and... ...Maintain records, files and databases of projects and operational data and develop reports, charts, graphs, etc. as required. Provide...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC2 days ago - ...AI Infrastructure Engineer Location: Fort Mill, SC Forward Deployed AI Engineer... ...security - VPC, PrivateLink, encryption, data residency Compliance - SOC 2, HIPAA,... ...Tracing - LangSmith, Arize, OpenTelemetry Principal FDE (future / escalation profile) Not...Temporary work
Diverse Lynx
Fort Mill, York County, SC1 day ago - ...AI Infrastructure Engineer (Forward Deployed AI Engineer) Location: Fort Mill, SC (Onsite – 5 Days/Week) Tax Term (W2, C2C): W2 /... ...Dynatrace, Grafana). Cloud security (VPC, PrivateLink, encryption, data residency). Security automation, vulnerability management,...Permanent employmentFull timeContract work
Apolis
Fort Mill, York County, SC1 day ago - Take on the role of a Warehouse Operations Coordinator and play a vital role in supporting warehouse leadership. Utilize your organizational and communication skills to ensure compliance with DOT requirements and efficient operations. Grow your caree
US Foods
Fort Mill, York County, SC3 days ago - ...activities for the regional and local office organization within Pike Engineering. This role supports client interaction, time and expense... ...Maintain records, files, and databases of projects and operational data; develop reports, charts, and graphs as required. Coordinate...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC1 day ago - F.-Schumacher- in Fort Mill, South Carolina, is seeking an HR Assistant to support local employees and collaborate with the HR department to cultivate a high-performance culture. Responsibilities include recruitment, onboarding, and managing HR processes. The ideal candidate...Local area
- F. Schumacher & Co. is looking for an HR Assistant to provide essential HR support to local employees. This role includes recruiting, facilitating hiring processes, and managing employee communication and logistics for meetings. The ideal candidate will have 2-3 years of...Local area
- ...Agentic AI Engineer – API Development Focus Location: Fort Mill, SC Duration: 12Months+ Extension Hourly Rate: Depending on Experience (DOE) Job Description We are seeking a highly skilled Agentic AI Engineer with a strong background in API development to join our innovative...Hourly payPermanent employmentContract workLocal area
ETHEREUM TECHNOLOGIES LLC
Fort Mill, York County, SC2 days ago - ...approach to where we work. About the Role: As a Machine Learning Engineer, you will have the opportunity to collaborate closely with... ...problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine learning...Permanent employmentFull timeWork at officeRemote workWork from homeFlexible hours
Instacart
Rock Hill, SC14 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- data modeling Fort Mill, York County, SC
- clinical data coordinator remote Fort Mill, York County, SC
- data Fort Mill, York County, SC
- ai data Fort Mill, York County, SC
- data capturer Fort Mill, York County, SC
- data tech Fort Mill, York County, SC
- data network cabling Fort Mill, York County, SC
- education data Fort Mill, York County, SC
- health data Fort Mill, York County, SC
- data cabling installation Fort Mill, York County, SC