Principal Data Engineer
United IT
Principal Data Engineer
Location: Fort Mill or Austin (Hybrid)
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
- Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
- Optimize data ingestion, transformation, and storage processes for high-performance data processing.
- Work with structured and unstructured data, ensuring data integrity, quality, and governance.
- Develop SQL queries to extract and manipulate data efficiently from relational databases.
- Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
- Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
- Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
- Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
- Experience in Data Engineering or a related field.
- Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
- Hands-on experience with Python for data processing and automation.
- Expertise in Apache Spark for distributed data processing.
- Solid understanding of ETL pipeline design and data warehousing concepts.
- Proficiency in SQL for querying and managing relational databases.
- Experience writing unit and integration tests using Pytest.
- Familiarity with CI/CD pipelines and version control systems (e.g., Git).
- Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
- Experience with Terraform, Docker, or Kubernetes.
- Knowledge of big data tools such as Apache Kafka or Airflow.
- Exposure to data governance and security best practices.
Vacancy posted 10 hours ago
Similar jobs that could be interesting for youBased on the Principal Data Engineer in Austin, TX vacancy
- ...connections, and building confidence through expert guidance.We have an exciting opportunity for a motivated and energetic Principal Data Engineer to shape and lead our technical strategy, a role critical to driving the next phase of growth and innovation within our Data...SuggestedWork at officeLocal area
$174k - $226k
...that power infrastructure, solutions, and data management our customers need most. Our... ...and reimagining how work gets done. Engineers define intent, author precise specifications... ...at MyWellatDell.comMechanical Senior Principal Engineer-Data Center IT Rack Feasibility...Suggested- Job DescriptionAbout the RoleThe Data Engineering Manager will lead a team of data engineers focused on building and operating Customer data platforms supporting OnStar, Connected Services, Customer, Marketing, Personalization, AI, and Advanced Analytics.This is a highly...SuggestedFull timeLocal areaWork from homeRelocation package
$152k - $209k
...that power infrastructure, solutions, and data management our customers need most. Our... ...and reimagining how work gets done. Engineers define intent, author precise specifications... ...of enterprise technology. Mechanical Principal Engineer-Data Center Custom SolutionsMechanical...SuggestedRelocation package$145k - $187k
...that power infrastructure, solutions, and data management our customers need most. Our... ...and reimagining how work gets done. Engineers define intent, author precise specifications... ...technology in Austin, Texas.Mechanical Principal Engineer Why Work with UsDell Technologies...Suggested- ...selected candidate for this role to work on site in the specified location(s): Westlake or Austin, TX Our OpportunityThe Senior Data Engineer is a senior technical contributor within DAFgiving360’s Data organization responsible for designing, building, operating, and evolving...Full timeWork at office
$151.2k - $204.6k
...we’re the people who keep the cloud running. We support all AWS data centers and all of the servers, storage, networking, power, and... ...You’ll join a diverse team of software, hardware, and network engineers, supply chain specialists, security experts, operations managers...Flexible hours$124k - $242k
...that power infrastructure, solutions, and data management our customers need most. Our... ...and reimagining how work gets done. Engineers define intent, author precise specifications... ...future of enterprise technology. Senior Principal Thermal Engineer-Data Center Solutions...Relocation package$132.4k - $238.4k
...world's top brands, offering comprehensive engineering, supply chain, and manufacturing... ...seeking a highly experienced and innovative Principal Controls Engineer to lead the development... ...critical systems, with a strong emphasis on data center infrastructure, power/energy...Full timeTemporary workWork at officeLocal areaRemote workWorldwide- ...world's top brands, offering comprehensive engineering, supply chain, and manufacturing... ...experienced and technically proficient Principal Electrical Engineer to engage in the design... ...and integration of electrical systems for data center power and cooling products, as well...Full timeContract workWork at officeLocal areaRemote workWorldwide
$170k - $225k
...Data Engineering Manager Self Financial is a venture-backed, high-growth FinTech company with a mission to increase economic inclusion and financial resilience by empowering people to build credit and build savings. Our team is passionate about challenging the status...Casual workRemote workFlexible hoursSELF
Austin, TX4 days ago$114.5k - $154.58k
...re doing! Job Description Summary: We're looking for a Principal Sales Engineer who is not only passionate about technology but thrives on engaging... ...: Engage with prospective clients to understand their data integration challenges, especially around migrating from...Full timeRemote workWorldwideRocket Software
Austin, TX17 hours ago- ...Data Engineering ManagerHusch Blackwell LLP is a full-service litigation and business law firm with multiple locations across the United... ...requirements; and more.Please include a cover letter and resume when applying.EOE/Minority/Female/Disabled/Vet. Principal Applicants Only....Full timeTemporary workRemote workFlexible hours
Husch Blackwell
Austin, TX3 days ago - ...an exciting opportunity for a highly experienced and technically proficient Principal Electrical Engineer to engage in the design, fabrication, and integration of electrical systems for data center power and cooling products, as well as factory enablement projects. This...Contract workRemote work
JABIL CIRCUIT, INC
Austin, TX17 hours ago - ...adventures. We are BUILT FOR THE WILD.About the RoleWe are actively searching for a Senior Director, Enterprise Integrations and Data Engineering to oversee YETI’s Data Warehouse, Analytics Platforms, MDM, and Integration Platforms. This role is responsible for defining...Full timeContract workLocal areaFlexible hours
- As the Manager of Data Engineering & Analytics , you will report to the Vice President of Business Intelligence and lead The Helper Bees' Data Engineering and Analytics team, responsible for both the technical execution and strategic direction of data across the company...Work experience placement
- Texas Department of Family and Protective Services is seeking a Data Engineering Manager V to lead the Data Infrastructure & Tech (DIT) division in Austin. The role directs data engineering, reporting, and analytics teams, guiding program delivery and implementing data...Remote job
$8,333.33 - $9,000 per month
...could help you pay off federal student loans faster. You can see all the details here: ERS recruitment brochure Functional Title: Data Engineering Manager V Job Title: Manager V Agency: Dept of Family & Protectve Svc Department: DATA INFRASTRUCTURE & TECH/DIT Posting...Full timeTemporary workPart timeWork experience placementWork at officeLocal areaRemote workFlexible hoursShift workDay shift- Charles Schwab in Austin is seeking an experienced ETL/Big Data Engineer to develop, maintain, and support large-scale data pipelines and applications. You will work with platform architects to ensure technical integrity, proactively troubleshoot production issues, and...
- A leading engineering firm is seeking a Principal Mechanical Integration Engineer to lead mechanical design for advanced data center infrastructure. The ideal candidate will have at least 8 years of experience in mechanical design and integration for mission-critical systems...Remote work
- ...work for a company that values your contributions and supports your growth, we would like to meet you.## Key Responsibilities* **Data Engineering team leadership**. Lead, mentor, and develop a team of data engineers supporting reporting, analytics, data science, AI, and...Permanent employmentTemporary workWork at officeFlexible hours
- ...maintain, and support large-scale and complex ETL and distributed Big Data applications. Work with platform architects to ensure technical... ...: Requires Bachelor's in Computer Science, Computer Engineering, Mechanical Engineering, or a related field and 60 months of progressive...Remote work
$134.5k - $265.1k
...combining Deloitte's sports industry experience with proprietary data, AI-native decision intelligence, and reusable Converge... ...organizations. The team works across Product Management, Data Engineering, Data Science, Software Engineering, Forward Deployed Engineering...Local area- The Helper Bees in Austin, TX is seeking a strategic leader for Manager of Data Engineering & Analytics to drive data strategy, governance, and delivery across the organization. You will lead a diverse team of data engineers and analysts, own intake/prioritization in Jira...
- ...IAM, Cloud Monitoring/Logging)Expert Python for production-grade data and backend engineeringStrong SQL and data modeling for... ...least privilege, auditability)Strong observability and reliability engineering skills (monitoring, alerting, incident response, SLAs/SLOs)Fundamental...Full timeWork at office
- ...Austin, TXEmployment Type: Full TimeIndustry: Computer SoftwareClient: WiproContact: Meghana GorusuCompany: SRI Tech SolutionsJob Title: Data EngineerLocation: Austin, TXJob Description: Panel is available daily 2-4 PM cSTSubmit with skill...
$132.1k - $178.8k
...list of customers that are building mission-critical applications on top of AWS services.The AWS Marketplace team is looking for a Data Engineer to play a key role in building insights from our large and ever-growing sets of partner and customer data. Are you passionate...Flexible hours$132.1k - $178.8k
Join the OIS/CXI Analytics team to build strategic data infrastructure powering Amazon's Operations Technology ecosystem. Our team provides... ...Fresh, Prime Now, Lockers, Pantry, and Amazon Campus. As a Data Engineer, you will build and maintain scalable data pipelines and ML-...Full timeTemporary workSeasonal workWorldwideFlexible hours$133k - $188.6k
...transformation, curation, and delivery of manufacturing and enterprise data.Build reusable Bronze, Silver, and Gold data pipelines, data... ...and operationalize ML workflows for data preparation, feature engineering, model training, evaluation, deployment, batch scoring,...Full timeH1bLocal areaWork from homeRelocation packageFlexible hours- ...AreOverhaul is a supply chain integrity solutions company that allows shippers to connect disparate sources of data into the first fully transparent situational analysis engine designed for the logistics industry. Data that is transformed into critical insights can instantly...Local areaRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Data Engineer. Be the first to apply!
Related searches
- principal network engineer Austin, TX
- principal application developer Austin, TX
- civil engineer project manager Austin, TX
- principal security engineer Austin, TX
- principal engineer Austin, TX
- principal battery engineer Austin, TX
- principal infrastructure engineer Austin, TX
- director quality engineering Austin, TX
- senior civil engineer project manager Austin, TX
- senior chief engineer Austin, TX