Principal Data Engineer
United IT
Principal Data Engineer
Location: Fort Mill or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
- Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
- Optimize data ingestion, transformation, and storage processes for high-performance data processing.
- Work with structured and unstructured data, ensuring data integrity, quality, and governance.
- Develop SQL queries to extract and manipulate data efficiently from relational databases.
- Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
- Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
- Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
- Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
- Experience in Data Engineering or a related field.
- Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
- Hands-on experience with Python for data processing and automation.
- Expertise in Apache Spark for distributed data processing.
- Solid understanding of ETL pipeline design and data warehousing concepts.
- Proficiency in SQL for querying and managing relational databases.
- Experience writing unit and integration tests using Pytest.
- Familiarity with CI/CD pipelines and version control systems (e.g., Git).
- Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
- Experience with Terraform, Docker, or Kubernetes.
- Knowledge of big data tools such as Apache Kafka or Airflow.
- Exposure to data governance and security best practices.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
- ...Principal Data EngineerWe are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing...Suggested
InterSources
Fort Mill, York County, SC10 hours ago - ...Our client is looking Snowflake Engineer for Contract project in Fort mill SC below is the detailed requirements. Job Title... ..., or a related field. Strong experience in software/data engineering experience preferred. Must have hands on experience...SuggestedContract work
Lorven Technologies
Fort Mill, York County, SC4 days ago - ...Apache Airflow, AWS Glue, Amazon S3, SQL Server/T-SQL, Informatica Location: Fort Mill SC Job Description: We at Coforge are hiring Data Engineer with the following skill sets. Strong hands-on experience in Snowflake, DBT, Python, Apache Airflow, AWS Glue, Amazon S3, SQL...Suggested
COFORGE Marketing
Fort Mill, York County, SC4 days ago $113.34k - $155.85k
...Key Responsibilities Develop and maintain data ingestion and transformation pipelines... ...implement fixes Collaborate with senior engineers, analysts, and data scientists to understand... ...ADAAA. The above description covers the principal duties and responsibilities of the job....SuggestedFull timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC4 days ago$130.32k - $179.18k
...Design, build, and maintain end-to-end IoT data pipelines from edge devices into... ...Structured Streaming Define and enforce data engineering best practices, including pipeline... ...ADAAA. The above description covers the principal duties and responsibilities of the job....SuggestedFull timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC4 days ago- ...AWS Data Engineer Location: Fort Mill, SC Duration: Longterm Contract Note: We need candidate to work onsite from Day 1 (Hybrid) Key Responsibilities: Design, develop, and maintain scalable data pipelines and data integrations across systems Implement data...Contract work
E-Solutions
Fort Mill, York County, SC2 days ago $95k - $120k
...Data EngineerThe Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers, quality...Full timeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC1 day ago$95k - $120k
...Data Engineer The Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers,...Full timeWork at officeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC4 days ago- ...We are seeking a Lead Data Engineer with strong experience in delivering enterprise-scale cloud data and analytics solutions. The ideal candidate will have expertise in Google Cloud Platform (GCP), technical project leadership, and end-to-end delivery of modern data platforms...
Techvilla Solutions
Fort Mill, York County, SC5 days ago $39.86 - $62.29 per hour
Capgemini is seeking a skilled Data Engineer (AWS Glue, Airflow, Lambda, Postgres) in Fort Mill, South Carolina. The role involves designing and building scalable data pipelines and performing analytics on AWS and Postgres platforms. The ideal candidate should have 6+ years...- We are seeking a Mid-Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines...
$39.86 - $62.29 per hour
Overview We are seeking an experienced Developer with 10+ years of IT experience and deep expertise in data engineering and cloud-based data platforms (Snowflake/AWS). The ideal candidate will have strong hands-on experience in designing, implementing, and optimizing scalable...Contract work- Incedo Inc. is a high-growth Digital, Data and AI Transformation Specialist firm headquartered in New Jersey. We are a long-term strategy... ...Incedo delivers ROI from AI @ Scale through the “Power of 3”: Engineering & Operations excellence. Job Description:: We are seeking an...
$39.86 - $62.29 per hour
Data Engineer (AWS Glue, Airflow, Lambda, Postgres) We are seeking a skilled Data Engineer with strong hands‑on experience in AWS‑native data services, particularly Glue, Lambda, EventBridge, and workflow orchestration using Airflow. The role focuses on building scalable...Contract work- Role Data Engineer (Snowflake, AWS) - Location: Fort Mill, SC / ONSITE - Duration: Long Term Project - Experience: 10+ Job Role and Responsibilities Responsible for designing, developing and maintaining scalable data pipelines and ETL processes using AWS Glue. This role...
- ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World Inc
- Incedo Inc. is seeking an experienced Data Engineer with strong AWS OpenSearch expertise to design and maintain scalable data pipelines and search solutions. The role emphasizes Python, SQL, and cloud-native architectures to support analytics and real-time search across...
- Sunbelt Rentals, Inc. is seeking a Senior Data Engineer to design, build, and maintain end-to-end IoT data pipelines into Databricks, focusing on streaming ingestion, data quality, and Delta Lake operations. The role requires strong Python/SQL, Spark, and Kafka experience...
- A tech recruitment firm is searching for an ETL/Data Tester in Fort Mills, SC with a hybrid work model. The ideal candidate should have a minimum of 8 years of experience in ETL/Data Testing and 12+ years overall. Key skills required include Snowflake, Automation using...Contract work
- Tech Mirrors in Fort Mill, SC is seeking a Data Engineer specializing in Snowflake and AWS. The ideal candidate will design, develop, and maintain scalable data pipelines and ETL processes using AWS Glue, and collaborate with data professionals to ensure data quality. Responsibilities...
- ...seeking an AWS ETL Technical Lead to design and develop scalable data pipelines and warehouse solutions. This role involves leading... ...familiarity with CI/CD practices. Preferred certifications include Solutions Architect and Data Engineer. #J-18808-Ljbffr Signature IT World Inc
- ...Must Have Skills Strong experience in AWS services (hands-on experience) Data engineering skills (Glue, ECS, Data pipeline etc,) primarily serverless, databases, storage services, container services, schedulers, and batch services. Experience in Snowflake and Data...
2T Consulting
Pineville, NC10 hours ago - Capgemini is looking for an experienced Developer based in South Carolina to design, implement, and optimize data engineering solutions. With over 10 years of IT experience, the ideal candidate will have deep expertise in Snowflake and cloud platforms including AWS and...
$130.32k - $179.18k
...stakeholders What You'll Bring ~5+ years of experience in data science or quantitative analytics, with demonstrated impact on... ...to comply with ADA/ADAAA. The above description covers the principal duties and responsibilities of the job. The description shall...Full timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC4 days ago$130.32k - $179.18k
...technical stakeholdersWhat You'll Bring5+ years of experience in data science or quantitative analytics, with demonstrated impact on... ...to comply with ADA/ADAAA. The above description covers the principal duties and responsibilities of the job. The description shall not...Full timeFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC4 days ago- Ebara Pumps Americas Corporation is seeking a Product Data Engineer responsible for the creation, maintenance, and governance of product master data across ERP systems, including part numbers, BOMs, routings, and item attributes. This role ensures data accuracy, integrity...
$103.2k - $129k
...most qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Mechanical Engineer you will lead others in the design and analyses of the AP1000 and AP300 power plants.You will report to the Engineering...Permanent employmentFor subcontractorFlexible hours2 days per week$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Civil Structural Engineer you will support civil and structural design activities like structural systems evaluation for steel members,...Flexible hours$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Commissioning Engineer - Mechanical Testing,you will will be responsible for leading development of Westinghouse products in support of the APx...Flexible hours2 days per week$155.3k - $258.8k
...seeking a strategic and innovative technology leader to lead engineering teams responsible for Digital Experience, Cash & Liquidity solutions... ...log in or create an account to apply to this position. Principals only. EOE.Information on Interviews:LPL will only communicate...Full timeWork from home
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- director software engineering Fort Mill, York County, SC
- data center engineer Fort Mill, York County, SC
- senior cloud data engineer Fort Mill, York County, SC
- etl data engineer Fort Mill, York County, SC
- data engineer analytics Fort Mill, York County, SC
- data engineer machine learning Fort Mill, York County, SC
- finance data engineer Fort Mill, York County, SC
- data engineer Fort Mill, York County, SC
- data developer Fort Mill, York County, SC
- senior data center engineer Fort Mill, York County, SC