Principal Data Engineer
InterSources Inc
Role - Principal Data Engineer
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Vacancy posted 18 hours ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
- ...brands helps customers create products and capture moments that reflect who they uniquely are. Shutterfly is looking for a Principal Data Engineer to join their team! As part of the Data Engineering team you will tackle the scalability, performance and distributed...SuggestedRemote work
Shutterfly Career Site
Fort Mill, York County, SC4 days ago - ...Principal Data Engineer Location: Fort Mill or Austin (Hybrid) We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines...Suggested
United IT
Fort Mill, York County, SC4 days ago - ...Job Description • Lead a team of 2-5 Data Engineers with varied experience level to design and build data pipelines from various data sources to a target data warehouse using real-time and batch data load strategies utilizing cutting edge cloud technologies. • Serve...Suggested
Insight Global
Fort Mill, York County, SC2 days ago - ...Position Summary We are seeking an experienced Data Engineer to design, build, and enhance data ingestion pipelines and metadata frameworks that support data lineage, modernization, and analytics enablement. The ideal candidate has hands-on experience with AWS...Suggested
Euclid Innovations
Fort Mill, York County, SC3 days ago - ...AWS Data Engineer Location: Austin Tx, Fort Mill or NY – hybrid Contract Visa: USC & GC Job Description Architect and implement data pipelines using AWS Glue for ETL processes. Design event-driven workflows using AWS Lambda and Step Functions. Ensure...SuggestedContract work
AceStack LLC
Fort Mill, York County, SC1 day ago $95k - $120k
...Data Engineer The Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers,...Work at officeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC4 days ago- ...Role Name: Data Engineer (Python, AWS Glue, Airflow, Lambda) Location: Fort Mill, SC - 100% onsite JOB DESCRIPTION: Local Grade : C2 Location: Fortmill, SC Role Overview The Senior API Engineer (Python / AWS) is a critical member of the Data Modernization...Contract workLocal area
eTeam
Fort Mill, York County, SC3 days ago - Role Data Engineer (Snowflake, AWS) - Location: Fort Mill, SC / ONSITE - Duration: Long Term Project - Experience: 10+ Job Role and Responsibilities Responsible for designing, developing and maintaining scalable data pipelines and ETL processes using AWS Glue. This role...
- ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World Inc
$39.86 - $62.29 per hour
Data Engineer (AWS Glue, Airflow, Lambda, Postgres) We are seeking a skilled Data Engineer with strong hands‑on experience in AWS‑native data services, particularly Glue, Lambda, EventBridge, and workflow orchestration using Airflow. The role focuses on building scalable...Contract work$39.86 - $62.29 per hour
Capgemini is seeking a skilled Data Engineer (AWS Glue, Airflow, Lambda, Postgres) in Fort Mill, South Carolina. The role involves designing and building scalable data pipelines and performing analytics on AWS and Postgres platforms. The ideal candidate should have 6+ years...$39.86 - $62.29 per hour
Overview We are seeking an experienced Developer with 10+ years of IT experience and deep expertise in data engineering and cloud-based data platforms (Snowflake/AWS). The ideal candidate will have strong hands-on experience in designing, implementing, and optimizing scalable...Contract work- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...Full timeLocal area
- Capgemini is seeking a Data Engineer in Fort Mill, SC, to design and maintain scalable data pipelines using AWS and Snowflake. The ideal candidate will have extensive experience with data engineering, including ETL processes, and proficiency in Python. Responsibilities...
- We are seeking a Mid-Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines...
- Tech Mirrors in Fort Mill, SC is seeking a Data Engineer specializing in Snowflake and AWS. The ideal candidate will design, develop, and maintain scalable data pipelines and ETL processes using AWS Glue, and collaborate with data professionals to ensure data quality. Responsibilities...
- A tech recruitment firm is searching for an ETL/Data Tester in Fort Mills, SC with a hybrid work model. The ideal candidate should have a minimum of 8 years of experience in ETL/Data Testing and 12+ years overall. Key skills required include Snowflake, Automation using...Contract work
- ...seeking an AWS ETL Technical Lead to design and develop scalable data pipelines and warehouse solutions. This role involves leading... ...familiarity with CI/CD practices. Preferred certifications include Solutions Architect and Data Engineer. #J-18808-Ljbffr Signature IT World Inc
- ...Intelligence Developer in Fort Mill, SC. This position focuses on empowering business leaders through quality information and robust data analysis, supporting key initiatives, and driving business impact. The ideal candidate will possess a Bachelor's degree in a...
$83.43k - $222.48k
...helping to simplify health care one person, one family and one community at a time. ? Position Summary We're seeking a Sr. Data Engineer to design and implement data pipelines that power analytical capabilities. This hands-on role requires an understanding of data...Hourly payFull timeTemporary workWork experience placementLocal areaCVS Health
Rock Hill, SC2 days ago- Capgemini is looking for an experienced Developer based in South Carolina to design, implement, and optimize data engineering solutions. With over 10 years of IT experience, the ideal candidate will have deep expertise in Snowflake and cloud platforms including AWS and...
$119.07k - $198.38k
Job Overview The AVP, Data Reporting Manager leads the strategy, governance, and delivery of LPL’s enterprise Business Intelligence capabilities. This role is responsible for advancing Tableau Cloud, enabling AI‑driven analytics, and ensuring scalable, reliable, and governed...$50.37k - $62.96k
...Description We are hiring a Chief Engineer. Are you looking for a great opportunity to join a company where you can use your engineering background at one of North America’s fastest growing hospitality companies? Our Chief Engineering role will provide you a venue to...Full timeWork experience placementConcord Hospitality
Fort Mill, York County, SC3 days ago- We are seeking a skilled Quality Engineer with hands-on experience in Behavior-Driven Development (BDD) to ensure the quality and reliability of our data ingestion and pipeline solutions. The ideal candidate will collaborate with cross-functional teams to define, implement...
- Concord Hospitality Enterprises is seeking a Chief Engineer for their Fort Mill location. This role offers a unique opportunity to leverage your engineering expertise in a dynamic environment at one of North America’s fastest growing hospitality companies. The Chief Engineer...Full time
$50 per hour
...maintain web applications using React and Python. This role requires a minimum of 5 years of enterprise experience and familiarity with data manipulation libraries. The ideal candidate will demonstrate strong skills in front-end development and have proficiency in SQL...Hourly payWork at office- A technology company in Fort Mill, SC is looking for a skilled Quality Engineer to improve data ingestion and pipeline solutions. The ideal candidate will have a Bachelor's degree in a relevant field and at least 5 years of experience in software quality assurance. Responsibilities...
$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Mechanical Engineer you will lead others in the design and analyses of the AP1000 and AP300 power plants. You will report to the...For subcontractorFlexible hours2 days per week$149.35k - $248.85k
...direct people management, this is your seat. Job Overview: As the Principal Engineer, Cloud Strategy Lead, you set and drive the multi-year... ...generation, automated PR review, MCP-backed agents for internal data, and conversational interfaces that let internal customers...Full timeWork experience placementWork from home$103.2k - $129k
Select how often (in days) to receive an alert: Principal Engineer - New Plant Commissioning Support Requisition Number: 30774 Company: Westinghouse Electric Co Location: Rock Hill, SC, US, 29730 Cranberry Township, PA About the role As a Principal Engineer (APx Commissioning...Flexible hours2 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- principal engineer Fort Mill, York County, SC
- director software engineering Fort Mill, York County, SC
- general engineer Fort Mill, York County, SC
- principal developer Fort Mill, York County, SC
- data center chief engineer Fort Mill, York County, SC
- finance data engineer Fort Mill, York County, SC
- data developer Fort Mill, York County, SC
- senior cloud data engineer Fort Mill, York County, SC
- data engineer machine learning Fort Mill, York County, SC
- senior data center engineer Fort Mill, York County, SC