Principal Data Engineer
United IT
Principal Data Engineer
Location: Fort Mill or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
- Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
- Optimize data ingestion, transformation, and storage processes for high-performance data processing.
- Work with structured and unstructured data, ensuring data integrity, quality, and governance.
- Develop SQL queries to extract and manipulate data efficiently from relational databases.
- Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
- Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
- Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
- Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
- Experience in Data Engineering or a related field.
- Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
- Hands-on experience with Python for data processing and automation.
- Expertise in Apache Spark for distributed data processing.
- Solid understanding of ETL pipeline design and data warehousing concepts.
- Proficiency in SQL for querying and managing relational databases.
- Experience writing unit and integration tests using Pytest.
- Familiarity with CI/CD pipelines and version control systems (e.g., Git).
- Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
- Experience with Terraform, Docker, or Kubernetes.
- Knowledge of big data tools such as Apache Kafka or Airflow.
- Exposure to data governance and security best practices.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
- ...brands helps customers create products and capture moments that reflect who they uniquely are. Shutterfly is looking for a Principal Data Engineer to join their team! As part of the Data Engineering team you will tackle the scalability, performance and distributed...Suggested
Shutterfly Career Site
Fort Mill, York County, SC3 days ago - ...Description • Translate business strategy into technical execution, ensuring data platforms enable both real-time and batch analytics at scale. • Lead, mentor, and grow teams of data engineers and engineering managers with diverse experience levels. • Acting as a...Suggested
Insight Global
Fort Mill, York County, SC3 days ago - ...Job Description • Lead a team of 2-5 Data Engineers with varied experience level to design and build data pipelines from various data sources to a target data warehouse using real-time and batch data load strategies utilizing cutting edge cloud technologies. • Serve...Suggested
Insight Global
Fort Mill, York County, SC14 hours ago - ...Position Summary We are seeking an experienced Data Engineer to design, build, and enhance data ingestion pipelines and metadata frameworks that support data lineage, modernization, and analytics enablement. The ideal candidate has hands-on experience with AWS...Suggested
Euclid Innovations
Fort Mill, York County, SC2 days ago - ...Role: AWS Data Engineer Location: Fort Mill, SC Duration: Longterm Contract Note: We need candidate to work onsite from Day 1 (Hybrid) Key Responsibilities Design, develop, and maintain scalable data pipelines and data integrations across...SuggestedContract work
E-Solutions
Fort Mill, York County, SC1 day ago - ...EDM Developer Technical SME with deep expertise in IHS Markit EDM and strong experience in modern cloud-native event-driven and data engineering ecosystems. This role goes beyond traditional EDM development and requires ownership of end-to-end data mastering architecture...
eTeam
Fort Mill, York County, SC2 days ago $95k - $120k
...Data Engineer The Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers,...Work at officeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC3 days ago- Role Data Engineer (Snowflake, AWS) - Location: Fort Mill, SC / ONSITE - Duration: Long Term Project - Experience: 10+ Job Role and Responsibilities Responsible for designing, developing and maintaining scalable data pipelines and ETL processes using AWS Glue. This role...
- ...Services Microsoft Dynamics 365 Power BI INDUSTRIES PRODUCTS OneXcel IoT Solutions RESOURCES Whitepapers CAREERS Contact Us Sr. Data Engineer (Mid-Senior Level) - AWS & Streaming Full Time Fort Mill, SC (3 days hybrid) Posted 3 months ago eone Infotech Job...Full time
- Capgemini in Fort Mill, SC, is seeking an experienced Developer with over 10 years of IT experience, particularly in data engineering and cloud platforms like Snowflake and AWS. The ideal candidate will design, implement, and optimize scalable data solutions, manage database...
$39.86 - $62.29 per hour
Capgemini is seeking a skilled Data Engineer (AWS Glue, Airflow, Lambda, Postgres) in Fort Mill, South Carolina. The role involves designing and building scalable data pipelines and performing analytics on AWS and Postgres platforms. The ideal candidate should have 6+ years...$39.86 - $62.29 per hour
Overview We are seeking an experienced Developer with 10+ years of IT experience and deep expertise in data engineering and cloud-based data platforms (Snowflake/AWS). The ideal candidate will have strong hands-on experience in designing, implementing, and optimizing scalable...Contract work- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...Full timeLocal area
- Capgemini is seeking a Data Engineer in Fort Mill, SC, to design and maintain scalable data pipelines using AWS and Snowflake. The ideal candidate will have extensive experience with data engineering, including ETL processes, and proficiency in Python. Responsibilities...
$39.86 - $62.29 per hour
Capgemini in Fort Mill, SC is looking for a skilled Data Engineer. The role focuses on building scalable data pipelines using AWS services like Glue, Lambda, and EventBridge. Candidates should have strong hands-on experience with these technologies and proficiency in SQL...Hourly pay- We are seeking a Mid-Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines...
- ...Intelligence Developer in Fort Mill, SC. This position focuses on empowering business leaders through quality information and robust data analysis, supporting key initiatives, and driving business impact. The ideal candidate will possess a Bachelor's degree in a...
BROAD RIVER RETAIL
Fort Mill, York County, SC5 days ago - Tech Mirrors in Fort Mill, SC is seeking a Data Engineer specializing in Snowflake and AWS. The ideal candidate will design, develop, and maintain scalable data pipelines and ETL processes using AWS Glue, and collaborate with data professionals to ensure data quality. Responsibilities...
- A tech recruitment firm is searching for an ETL/Data Tester in Fort Mills, SC with a hybrid work model. The ideal candidate should have a minimum of 8 years of experience in ETL/Data Testing and 12+ years overall. Key skills required include Snowflake, Automation using...Contract work
- Capgemini is looking for an experienced Developer based in South Carolina to design, implement, and optimize data engineering solutions. With over 10 years of IT experience, the ideal candidate will have deep expertise in Snowflake and cloud platforms including AWS and...
$50.37k - $62.96k
...Description We are hiring a Chief Engineer. Are you looking for a great opportunity to join a company where you can use your engineering background at one of North America’s fastest growing hospitality companies? Our Chief Engineering role will provide you a venue to...Full timeWork experience placementConcord Hospitality
Fort Mill, York County, SC2 days ago- We are seeking a skilled Quality Engineer with hands-on experience in Behavior-Driven Development (BDD) to ensure the quality and reliability of our data ingestion and pipeline solutions. The ideal candidate will collaborate with cross-functional teams to define, implement...
- Concord Hospitality Enterprises is seeking a Chief Engineer for their Fort Mill location. This role offers a unique opportunity to leverage your engineering expertise in a dynamic environment at one of North America’s fastest growing hospitality companies. The Chief Engineer...Full time
- Job Category: Engineering Requisition Number: CHIEF061337 Full-Time On-site Location Courtyard by Marriott Fort Mill, 1385 Broadcloth St., Fort Mill, SC 29715, USA Description We are hiring a Chief Engineer at one of North America’s fastest growing hospitality companies...Full timeTemporary work
- A technology company in Fort Mill, SC is looking for a skilled Quality Engineer to improve data ingestion and pipeline solutions. The ideal candidate will have a Bachelor's degree in a relevant field and at least 5 years of experience in software quality assurance. Responsibilities...
$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Licensing Engineer you will support the deployment of advanced nuclear technology, with a focus on Advanced Plant reactor projects. You...Full timeWork at officeRemote workFlexible hours- London Stock Exchange Group in Fort Mill is seeking a Lead Software Engineer with a focus on AI. You'll design and develop Java and Python applications, lead engineering projects, and help build innovative solutions for financial markets. The ideal candidate has at least...
- Job Overview The BOSH Supervisor will support the BOSH Team and, by extension, our customers by providing real‑time support and development to the team. The Supervisor will also interface with other departments such as Customer Service, Verifications, FIS and EFG. Responsibilities...Full timePart timeWork at officeLocal areaFlexible hours
OneMain Financial
Fort Mill, York County, SC5 days ago $100k - $172.5k
...Description: We are searching for the best talent for a Principal Product Security Engineer to be located in Danvers, MA or Raritan, NJ. Remote work... ...including security plans, architecture diagrams, data flow diagrams, threat models, security requirements, Design...Full timeTemporary workWork at officeLocal areaImmediate startRemote work3 days per weekJohnson & Johnson
Rock Hill, SC4 days ago- ...We are seeking a talented and experienced Python Developer to join our data engineering team, contributing to a major system modernization initiative aimed at enhancing the investor experience. This role involves close collaboration with cross-functional teams to design...
Compunnel
Fort Mill, York County, SC5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- engineering director Fort Mill, York County, SC
- principal engineer Fort Mill, York County, SC
- director software engineering Fort Mill, York County, SC
- general engineer Fort Mill, York County, SC
- principal developer Fort Mill, York County, SC
- data center chief engineer Fort Mill, York County, SC
- chief engineer Fort Mill, York County, SC
- hotel chief engineer Fort Mill, York County, SC
- data engineer Fort Mill, York County, SC
- finance data engineer Fort Mill, York County, SC