Principal Data Engineer
United IT
Principal Data Engineer
Location: Fort Mill or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
- Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
- Optimize data ingestion, transformation, and storage processes for high-performance data processing.
- Work with structured and unstructured data, ensuring data integrity, quality, and governance.
- Develop SQL queries to extract and manipulate data efficiently from relational databases.
- Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
- Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
- Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
- Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
- Experience in Data Engineering or a related field.
- Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
- Hands-on experience with Python for data processing and automation.
- Expertise in Apache Spark for distributed data processing.
- Solid understanding of ETL pipeline design and data warehousing concepts.
- Proficiency in SQL for querying and managing relational databases.
- Experience writing unit and integration tests using Pytest.
- Familiarity with CI/CD pipelines and version control systems (e.g., Git).
- Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
- Experience with Terraform, Docker, or Kubernetes.
- Knowledge of big data tools such as Apache Kafka or Airflow.
- Exposure to data governance and security best practices.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
$130k - $190k
...or TN visa, nor is it available to work corp-to-corp. The Red Ventures Home Client Services group is looking for a talented Data Engineering Manager to join our team! As a Data Engineering Manager at Red Ventures, you will have the opportunity to build technology...SuggestedFull timeTemporary workWork at officeRemote workMonday to FridayFlexible hoursRed Ventures
Fort Mill, York County, SC3 days ago$95k - $120k
...Data Engineer The Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers,...SuggestedFull timeWork at officeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC2 days ago- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...SuggestedFull timeLocal area
Capgemini
Fort Mill, York County, SC22 hours ago - Title: Data Architect Engineer Duration: 6 month contract (potential to extend or convert) Work Location: Charlotte, NC preferred. If remote, would need to come onsite once per month for 2 days once converted to full time. Job Description: The Data Architect...SuggestedFull timeContract workLocal areaRemote workFlexible hours
INSPYR Solutions
Fort Mill, York County, SC4 days ago - ...AWS Data Engineer Location: Fort Mill, SC Duration: Longterm Contract Note: We need candidate to work onsite from Day 1 (Hybrid) Key Responsibilities: Design, develop, and maintain scalable data pipelines and data integrations across systems Implement data...SuggestedContract work
E-Solutions
Fort Mill, York County, SC8 hours ago - ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World Inc
- ...locationIndustriesChoose a partner with intimate knowledge of your industry and first-hand experience of defining its future.# Data Engineer (AWS & Snowflake)AtlantaChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way...Full timeLocal area
- ...: Nucor Business Technology (NBT)is seeking qualified applicants for the position ofData Engineer. This position will be responsible for designing and maintaining robust data models that support business intelligence and analytics initiatives, while also contributing...Work experience placement
Nucor Steel Auburn
Rock Hill, SC2 days ago - # Data Engineer**Job Duties:**Design and implement efficient large-scale data solutions in areas data ingestion, data curation, data quality, data visualization and data distribution using Informatica, Snaplogic, Tableau and MicroStrategy. Manage and query data in data...Full timeRelocation
Pantar Solutions
Weddington, NC4 days ago $103.2k - $129k
...talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Civil Structural Engineer you will support civil and structural design activities like structural systems evaluation for steel members,...Flexible hoursWestinghouse Electric Company, LLC
Rock Hill, SC9 hours ago- ...frontend frameworks. ~ Hands-on experience developing and deploying AI/ML solutions. ~ Experience with LLM integration, prompt engineering, embeddings, and RAG architecture. ~ Strong knowledge of SQL Server, PostgreSQL, Cosmos DB, or MongoDB. ~ Experience with...Full time
Qode
Fort Mill, York County, SC22 hours ago $103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Welding Engineer you will support the development, implementation, and troubleshooting of welding-related requirements and activities for...Contract workFlexible hours2 days per weekWestinghouse Electric Company, LLC
Rock Hill, SC1 day ago- Operations Manager Surface Experts of SW Charlotte & Rock Hill Lead the Team. Elevate the Standard. Build Something Great. Are you the type of leader who enjoys coaching people, improving processes, and making sure every customer receives an exceptional experience? Surface...Work at office
SURFACE EXPERTS, INC.
Fort Mill, York County, SC4 days ago $20 - $23 per hour
Position Summary Our client is seeking detail-oriented Operations Processing Specialists to support a high-volume operations team. This role focuses on processing transactions and requests, ensuring accurate and timely completion of work within established service level...Hourly payContract workTemporary workWork at officeTEKsystems
Fort Mill, York County, SC4 days ago$22.48 - $37.44 per hour
...difference by turning life's aspirations into financial realities. Please log in or create an account to apply to this position. Principals only. EOE. Information on Interviews: LPL will only communicate with a job applicant directly from an @lplfinancial.com email...Work from homeLPL Financial
Fort Mill, York County, SC2 days ago- Alt Ops/ Corporate Actions Specialist As an Alt Ops/ Corporate Actions Specialist, you will support key processes that connect us with its clients, vendors, and partners. You'll play a critical part in ensuring accurate and timely processing across Alternative Investments...
Apex Systems
Fort Mill, York County, SC1 day ago - Ulta Beauty in Fort Mill, South Carolina, is seeking a Task Associate to support operational excellence and enhance guest experience. The role involves maintaining store standards, ensuring compliance with company policies, and contributing to team performance. Candidates...Work experience placementFlexible hours
Ulta Beauty
Fort Mill, York County, SC3 days ago - ...supporting a leading utility in their search for a Project Manager II to join their team onsite in the Southeast. This role provides engineering project management expertise and technical direction for small to medium-scale electric substation, transmission, distribution,...Full timeContract workFor subcontractorWork at office
- Greystar Worldwide, LLC in Fort Mill, SC seeks a Property Manager to lead day-to-day operations at Fox Hunt Farms. You will manage budgeting, leasing, staff supervision, and tenant relations to achieve occupancy and revenue goals. You will oversee maintenance coordination...Worldwide
- Take on the role of a Warehouse Operations Coordinator and play a vital role in supporting warehouse leadership. Utilize your organizational and communication skills to ensure compliance with DOT requirements and efficient operations. Grow your caree
US Foods
Fort Mill, York County, SC2 days ago - ...AI Infrastructure Engineer Salary: 150K Gross Location: Fort Mill, SC Onsite – 5... ...security — VPC, PrivateLink, encryption, data residency Compliance — SOC 2, HIPAA, FedRAMP... ...— LangSmith, Arize, OpenTelemetry Principal FDE (future / escalation profile): Not...Temporary work
Diverse Lynx
Fort Mill, York County, SC22 hours ago - ...AI Infrastructure Engineer (Forward Deployed AI Engineer) Location: Fort Mill, SC (Onsite – 5 Days/Week) Tax Term (W2, C2C): W2 /... ...Dynatrace, Grafana). Cloud security (VPC, PrivateLink, encryption, data residency). Security automation, vulnerability management,...Permanent employmentFull timeContract work
Apolis
Fort Mill, York County, SC5 days ago - ...activities for the regional and local office organization within Pike Engineering. The incumbent will support client interaction up to and... ...Maintain records, files and databases of projects and operational data and develop reports, charts, graphs, etc. as required. Provide...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC1 day ago - F. Schumacher & Co. is looking for an HR Assistant to provide essential HR support to local employees. This role includes recruiting, facilitating hiring processes, and managing employee communication and logistics for meetings. The ideal candidate will have 2-3 years ...Local area
F. Schumacher
Fort Mill, York County, SC1 day ago - ...activities for the regional and local office organization within Pike Engineering. This role supports client interaction, time and expense... ...Maintain records, files, and databases of projects and operational data; develop reports, charts, and graphs as required. Coordinate...Work at officeLocal area
Pike Corporation
Fort Mill, York County, SC22 hours ago - F.-Schumacher- in Fort Mill, South Carolina, is seeking an HR Assistant to support local employees and collaborate with the HR department to cultivate a high-performance culture. Responsibilities include recruitment, onboarding, and managing HR processes. The ideal candidate...Local area
- Agentic AI Engineer – API Development Focus Location: Fort Mill,SC Duration: 12Months + Extension Hourly Rate: Depending on Experience (DOE) Work Authorization: Job Description: We are seeking a highly skilled Agentic AI Engineer with a strong background in API development...Hourly payPermanent employmentContract workLocal area
Cloud Hybrid Technologies LLC
Fort Mill, York County, SC11 hours ago $107.66k - $161.7k
...working on our Poe product. About the Team and Role: Our small engineering team works on challenging problems every day. We have a culture... ...end ownership of Machine Learning systems -- from prototyping, data pipelines and training, to realtime LLM application at scale...Remote jobFull timeWork experience placementInternshipQuora
Rock Hill, SC4 days ago- ...approach to where we work. About the Role: As a Machine Learning Engineer, you will have the opportunity to collaborate closely with... ...problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine learning...Permanent employmentFull timeWork at officeRemote workWork from homeFlexible hours
Instacart
Rock Hill, SC4 days ago - Operations Manager – GGP Location: Carolina Place – 11025 Carolina Place Parkway Business Headquartered in Chicago, IL, GGP is a leading retail real estate company with a portfolio of 110 high-performing retail assets, encompassing over 103 million square feet of space...For contractorsWork at officeLocal areaAfternoon shift
GGP
Pineville, NC11 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- data loss prevention analyst Fort Mill, York County, SC
- data analysis part time Fort Mill, York County, SC
- data Fort Mill, York County, SC
- clinical data coordinator remote Fort Mill, York County, SC
- data visualization Fort Mill, York County, SC
- sap master data Fort Mill, York County, SC
- health data Fort Mill, York County, SC
- data collection researcher Fort Mill, York County, SC
- data network cabling Fort Mill, York County, SC
- clinical data Fort Mill, York County, SC