Principal Data Engineer
InterSources Inc
Role - Principal Data Engineer
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Location - Fort Mill SC or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Fort Mill, York County, SC vacancy
$130.32k - $179.18k
...Design, build, and maintain end-to-end IoT data pipelines from edge devices into... ...Structured Streaming Define and enforce data engineering best practices, including pipeline... ...ADAAA. The above description covers the principal duties and responsibilities of the job....SuggestedFull timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC1 day ago$95k - $120k
...Data Engineer The Data Engineer will be responsible for a wide variety of tasks which include all aspects of the development and delivery of data solutions and maintaining the enterprise data warehouse. You will work closely with business analysts, product managers,...SuggestedFull timeWork at officeFlexible hoursTwo Harbors Investment Corp
Fort Mill, York County, SC2 days ago$113.34k - $155.85k
...Responsibilities Develop and maintain data ingestion and transformation pipelines... ...fixes Collaborate with senior engineers, analysts, and data scientists to understand... ...ADAAA. The above description covers the principal duties and responsibilities of the job....SuggestedFull timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC2 days ago$100k - $170k
Senior Data Engineer We are seeking a Senior Data Engineer to provide technical leadership for building and enhancing on-premises data warehouse solutions and accelerating our migration to AWS, including Amazon Redshift. In this role you will be an integral part of an exciting...SuggestedWork experience placementWork from homeVisa sponsorshipWork visaMonday to FridayShift work- ...role focuses on designing, developing, and automating end-to-end data pipelines to deliver scalable data integration and data... ...communication and leadership skills. Preferred AWS certifications (Solutions Architect, Data Engineer). #J-18808-Ljbffr Signature IT World IncSuggested
- Data Engineer - Fort Mill, SC - Contract - W2 - $44-54 Per HR Description Engage in platform administration and data engineering within a cloud environment. Collaborate with teams to enhance the data ecosystem and ensure operational excellence. Design scalable data solutions...Contract work
- Position Summary We are seeking an experienced Data Engineer to design, build, and enhance data ingestion pipelines and metadata frameworks that support data lineage, modernization, and analytics enablement. The ideal candidate has hands-on experience with AWS Glue , Python...
- Must Have Skills Strong experience in AWS services (hands-on experience) Data engineering skills (Glue, ECS, Data pipeline etc,) primarily serverless, databases, storage services, container services, schedulers, and batch services. Experience in Snowflake and Data Build...
- ...supporting a leading utility in their search for a Project Manager II to join their team onsite in the Southeast. This role provides engineering project management expertise and technical direction for small to medium-scale electric substation, transmission, distribution,...Full timeContract workFor subcontractorWork at office
FindTalent
Fort Mill, York County, SC4 days ago $130.32k - $179.18k
...stakeholders What You'll Bring ~5+ years of experience in data science or quantitative analytics, with demonstrated impact on... ...to comply with ADA/ADAAA. The above description covers the principal duties and responsibilities of the job. The description shall not...Full timeTemporary workFlexible hoursNight shiftWeekend workSunbelt Rentals
Fort Mill, York County, SC2 days ago$103.2k - $129k
...most qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Mechanical Engineer you will lead others in the design and analyses of the AP1000 and AP300 power plants.You will report to the Engineering...Permanent employmentFor subcontractorFlexible hours2 days per week$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Civil Structural Engineer you will support civil and structural design activities like structural systems evaluation for steel members,...Flexible hours$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace. About the role: As a Principal Mechanical Engineer you will develop and support nuclear power plant design modifications, engineering evaluations, and technical issue...Temporary workWork at officeRemote workRelocation packageFlexible hoursWestinghouse Electric Company
Rock Hill, SC4 days ago$155.3k - $258.8k
...seeking a strategic and innovative technology leader to lead engineering teams responsible for Digital Experience, Cash & Liquidity solutions... ...log in or create an account to apply to this position. Principals only. EOE.Information on Interviews:LPL will only communicate...Full timeWork from home$167.99k - $279.95k
...Financial is seeking an experienced and visionary VP, Software Engineering Manager - Kong Platform, & OneELK to lead the strategic... ...Please log in or create an account to apply to this position. Principals only. EOE.Information on Interviews:LPL will only communicate...Full timeWork from home$103.2k - $129k
...qualified talent while recognizing and encouraging the value of diversity in the global workplace.About the role:As a Principal Commissioning Engineer - Mechanical Testing,you will will be responsible for leading development of Westinghouse products in support of the APx...Flexible hours2 days per week- ...frontend frameworks. ~ Hands-on experience developing and deploying AI/ML solutions. ~ Experience with LLM integration, prompt engineering, embeddings, and RAG architecture. ~ Strong knowledge of SQL Server, PostgreSQL, Cosmos DB, or MongoDB. ~ Experience with...Full time
Qode
Fort Mill, York County, SCmore than 2 months ago $79 - $89.94 per hour
...AI/ML Engineer Location: Fort Mill, SC Onsite Flexibility: Onsite Contract Details Position Type: Contract Contract... ..., traceable, and aligned with enterprise governance and data standards. Participate in architecture discussions, code reviews...Contract workWorldwideWork visaGlobal Technical Talent
Fort Mill, York County, SC3 days ago- AI/ML Engineer (Hands On Exp In AWS Bedrock) Location: Austin, TX and South Carolina 6-month contract to hire Key Responsibilities Define... ...workflow linking code-graph analysis, LLM evaluation, dispositioning, data migration, and productionization. Review outputs across the pod...Contract work
$25.25 - $35.25 per hour
...assessor of talent; consistent) Patient Advocate (flexible and adaptive; empathetic; passionate; ethical Process Focused Operator (data driven decision-maker; detailed; organized and structured; comprehensive knowledge of all operational processes; computer proficient...Hourly payTraineeshipWork experience placementWork at officeLocal areaFlexible hoursPDS Health
Fort Mill, York County, SC2 days ago- ...with Customer Support, Sales, Billing, Product Management, and Engineering teams to ensure customer satisfaction and business success. Monthly... ..., and billable devices. Provide operational files and system data required to support monthly billing activities. Resolve...Local area
AMETEK
Fort Mill, York County, SC2 days ago - What You’ll Do Review and process submitted forms and documentation with a high level of accuracy and attention to detail. Reconcile, report, process and validate cash and securities transactions. Research and resolve discrepancies between DTCC, transfer agents, and contra...
Apex Systems
Fort Mill, York County, SC1 day ago $22.48 - $37.44 per hour
Operations Specialist At LPL Financial, we empower professionals to shape their success while helping clients pursue their financial goals with confidence. The Operations Specialist will work with the Wealth Advisors Group (Wealth AG) and will be part of a team providing...Work from homeLPL Financial
Fort Mill, York County, SC3 days ago- ...end and Python backend experience, responsible for developing and helping maintain our AWS hosted web application. Collaborate with data modelers, analysts, and business users to ensure ongoing development with different product teams. Skills & Experience: • Shown experience...
- Our values start with our people, join a team that values you! Bring your talents to Ross, our leading off-price retail chain with over 2,200 stores, and a strong track record of success and growth. Our focus has always been bringing our customers a constant stream of ...Hourly payFull timeSeasonal workWork at officeLocal areaImmediate startShift work
- ...technical accounting skills, we're looking for someone who embodies these attributes: Ownership Mentality: You own the operational engine of accounting. When invoices aren't processed, cash doesn't reconcile, or inventory is off, you feel it personally. You don't point...Casual workBank staff
Shoe Station Group, Inc.
Fort Mill, York County, SC5 days ago $23 - $27 per hour
...documentation, and other operational issues. Maintain organized and accurate files, records, and documentation. Support internal teams with data gathering, reporting, and special projects. Identify opportunities to improve operational processes, communication, and efficiency....Hourly payWeekly payPermanent employmentFull timeTemporary workWork at officeLocal area- ...Description Summary Target Department: Sustaining & Modification Engineering (Conveyance Systems) Primary Focus: Team leadership,... ...skills Analytical mindset with the ability to interpret data and make informed decisions Resilient and adaptable in a fast...Temporary workLocal areaWork from home
Rexroth
Pineville, NC2 days ago - ...AI Infrastructure Engineer (Forward Deployed AI Engineer) Location: Fort Mill, SC (Onsite – 5 Days/Week) Tax Term (W2, C2C): W2 /... ...Dynatrace, Grafana). Cloud security (VPC, PrivateLink, encryption, data residency). Security automation, vulnerability management,...Permanent employmentFull timeContract work
Apolis
Fort Mill, York County, SC5 days ago - ...merchandise inventory, budgets, vendor spend, and customer/dealer engagement. Develop reports and translate marketing and operational data into actionable insights, recommendations, and program improvements. Collaborate with Sales, Marketing, Operations, dealers, vendors...Work at office
Acro
Fort Mill, York County, SC13 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- data loss prevention engineer Fort Mill, York County, SC
- data Fort Mill, York County, SC
- test data management Fort Mill, York County, SC
- data cabling installation Fort Mill, York County, SC
- health data Fort Mill, York County, SC
- data loss prevention analyst Fort Mill, York County, SC
- education data Fort Mill, York County, SC
- data intern Fort Mill, York County, SC
- data capturer Fort Mill, York County, SC
- data collection Fort Mill, York County, SC