Data Engineer with DevOps

System One

Job Description

Job Title: Senior Data Engineer with DevOps

Duration : Full Time Location : Pittsburgh, PA, Cleveland, OH, or Dallas, TX

Work Mode : 5 Days Onsite Years Of Exp : 8+ Yrs Job Description: We are seeking a Data Engineer with 5 years of experience to design and maintain scalable data pipeline supporting analytics, reporting, and operational needs. The role involves collaborating with cross functional teams to ensure data alignment with business requirements and enterprise standards. Duties and Responsibilities:

Design and build scalable data pipelines aligned with business needs
Process large dataset (batch + sometimes near Realtime)
Ensure data quality, consistency, and governance standards across systems
Support data integration and transformation efforts for analytics and reporting platforms
Maintain data dictionaries, metadata, and documentation
Participate in data architecture reviews and model validation processes
Support analytics reporting and risk platforms.

Required Qualifications
5+ years of experience in data engineering and big data processing
Strong expertise in Apache Spark (Spark Core, Spark SQL) and PySpark for large scale batch processing
Experience working with structured and semi structured data, including complex transformations and performance tuning
Proficiency in data ingestion and integration from sources like Oracle, SQL Server, Hive, HDFS, and S3; transform data into ‘curated data models'
Experience writing data to Hive tables, Data Lakes (Iceberg), and downstream reporting systems
Strong knowledge of SQL and data modeling concepts
Hands on experience with Apache Airflow for workflow orchestration (DAG design, scheduling expectations, monitoring)
Proficiency in shell scripting for job automation, file validation, dependency handling, and logging. Trigger Spark Jobs, perform file checks and validation; Archive & purge data; mange job dependency, logging & error handling
Strong understanding of batch processing and batch job scheduling frameworks
Experience migrating from CA7/Control M Airflow (daily, hourly, weekly schedules) CI/CD for data pipelines
Fundamentals in Linux and Networking
Docker, OCP containerization / Kubernetes

Knowledge of CI/CD pipeline tools: Tools commonly include Jenkins, GitHub Actions, Azure DevOps, GitLab Cl, Maven, and Gradle
Automate operational tasks using Python, Bash/Shell, and PowerShell
Implement monitoring and alerting, Application Insights. Enable centralized logging with tools such as ELK.
Experience ensuring data quality, reliability, and compliance in regulated environments
Good communication and documentation skillsGood communication and collaboration skills in cross-functional teams
Agile/Safe methodologies

Must Have Skills

Airflow
Containerization
DevOps
Elastic Stack & Elasticsearch
GitHub
Hadoop Hive
Jenkins
JSON Web Token (JWT)
Kubernetes
OpenShift
Oracle
Python
Shell Script
SQLite

Ref: #404-IT Pittsburgh

Apply

Vacancy posted 7 days ago

Similar jobs that could be interesting for youBased on the Data Engineer with DevOps in Dallas, TX vacancy

Data Engineer
...Purple DriveJob Title: Data EngineerLocation: Dallas, TX / Plano, TX (Onsite)Job SummaryWe are seeking a Data Engineer with strong experience in designing, building, and maintaining... ...with corporate standards.Work closely with DevOps teams to automate deployments using CI/...
Devops
Purple Drive
Dallas, TX
3 days ago
Data Engineer
...into the tech industry by providing them with the in-demand technical skills, hands-on... ...Java Software ProgrammersData Analysts, Data ScientistsData Engineers & AI/Machine Learning EngineersPreferred Skills For Java/Full Stack/DevOps Roles:Bachelor's or Master's degree in...
Devops
Full time
SynergisticIT
Dallas, TX
2 days ago
Data Engineer
...office, following Genpact locations are fine with client Atlanta, GA Jacksonville, FL... ...Training and small amount of hands-onBig Data/Hadoop - Working experience with Hadoop stack... ...versioning tools such as GitHub or BitBucket.DevOps/CICD - Jenkins, Docker, and Kubernetes - 5...
Devops
Work experience placement
Work at office
Remote work
Staffing the Universe
Richardson, TX
2 days ago
Senior Data Engineer
...Senior Data EngineerLocation - Dallas, TX and San Ramon, CA and... ...Integrate application components with CI/CD pipelines and GitOps... ...Conduct code reviews and enforce engineering, security, performance, and maintainability... ...analysts, data engineers, DevOps engineers, and other...
Devops
Contract work
Remote work
Argyle Infotech
Dallas, TX
21 hours ago
Sr. Data Engineer
$90 - $95 per hour
...organization in the technology and data solutions industry, is seeking a Sr. Data Engineer to join their team. As a Sr.... ...primarily AWS to GCP. Collaborate with cross-functional teams to deliver... ...Terraform. Knowledge of DevOps practices, CI/CD pipelines, and...
Devops
Hourly pay
Weekly pay
Temporary work
Flexible hours
3 days per week
Manpower Group Inc.
Dallas, TX
9 hours ago
Snowflake Data Engineer
...Snowflake Data EngineerLocation: Dallas TXKey Responsibilities:Design... ..., data vault) optimized with micro-partitioning, clustering... ...deployment using GitHub Actions, Azure DevOps, or dbt Cloud.Qualifications:... ...'s in Computer Science, Data Engineering, or related field.Experience:...
Devops
Yantran LLC
Dallas, TX
2 days ago
Senior Data Engineer
...and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device... ..., and on-call activities. Work closely with Product, Application Engineering, Quality Assurance, DevOps/Site Reliability Engineering, Analytics, and globally...
Devops
Highbrow
Syracuse, NY
4 days ago
AWS Data Engineer
$150k - $170k
...References to be submitted Role: Senior Data Engineer Salary: $150,000-170,000 Fully... ...working on our data systems and services with other data engineers to deliver projects... ..., and security across platforms. DevOps & Automation Implement CI/CD pipelines...
Devops
Local area
Remote work
LHH
Dallas, TX
3 days ago
Data Engineer
$95k - $135k
...difference. Ready to fuse your passion with purpose? Let us shape the future of healthcare... ...SHARE program. JOB SUMMARY: The Data Engineer designs, builds, and maintains the data... ...Experience with GitHub and/or Azure DevOps (ADO) for version control and collaborative...
Devops
Permanent employment
H1b
Healthfuse
Syracuse, NY
4 days ago
Databricks Data Engineer
...scale. For more information, please visit Role: Databricks Data Engineer with AI Data Platform Location: Irving TX Technical Skills... ...Development REST APIs and Integration Frameworks Git CICD DevOps Preferred Skills Mosaic AI Vector Search Feature...
Devops
Temporary work
Local area
LTM
Irving, TX
9 hours ago
Data Engineer
...Data EngineerGeneral Purpose The Data Engineer will be responsible for designing, developing, and maintaining data pipelines... ...techniques within pipelines. Work with business users and analysts to... ...(Terraform, ARM templates), and DevOps practices.Understanding of data governance...
Devops
Local area
Coca-Cola Southwest Beverages
Dallas, TX
1 day ago
Senior Data Engineer
$112.71k - $183.14k
...innovation here – we make it happen, with our customers, where we work... ...worldwide, our teams use data, technology, advanced analytics... ...You Will Do:As a Senior Data Engineer on the Data Engineering team,... ...Jenkins, Github Actions, Azure Devops etc.Experience with AWS components...
Devops
Part time
Worldwide
Relocation package
Flexible hours
Caterpillar
Dallas, TX
2 days ago
Sr. Data Engineer
...Sr. Data Engineer Opportunity At Orion180At Orion180, we don't just stand out for what we do;... ...governance frameworks. You will collaborate with data scientists, analysts, and business... ...for all data assets using Azure DevOps, ensuring automated testing, validation,...
Devops
Work at office
Orion180
Irving, TX
21 hours ago
Data Acquisition Engineer
$136k - $221k
...Data Acquisition Engineer Fully Remote (U.S. only) | Full-time | $136,000 – $221,000 + performance bonus... ...commerce. You'll work closely with a Senior Backend Engineer powering the... ...GitHub Actions), testing frameworks, and DevOps reliability, with full observability...
Devops
Full time
Remote work
Flexible hours
Molecular Surroundings
Dallas, TX
4 days ago
Principal Data Engineer
...We are seeking a Principal Data Engineer to lead the architecture, design, and delivery of modern... ...deep data‑engineering expertise with strong consulting instincts: they can translate... ...controls. ~ Working knowledge of DevOps and DataOps practices, including Git‑based...
Devops
CG Infinity
Dallas, TX
1 day ago
Senior Data Engineer
$120k - $140k
...Title: Senior Data Engineer Location: Coppell, TX Salary Range: $120K-$140K We are... ...Reporting" teams but will be primarily tasked with designing, building, maintaining, and... ...environment (Azure and/or AWS), Azure DevOps (ADO), SQL (ETL, SSIS packages), and PySpark...
Devops
Temporary work
Addison Group
Dallas, TX
2 days ago
GCP Data Engineer
...Purple DriveGCP Data Engineer Share Contractual Jersey City, NJ | Dallas, TX PDT - 9229 10-12... ...seeking an experienced GCP Data Engineer with a strong background in data engineering,... ...and automation of data processes.Apply DevOps principles for continuous integration and...
Devops
Purple Drive
Dallas, TX
3 days ago
AWS Data Engineer
...Opportunity for advancement Job Title: AWS Data Engineer... ...seeking an experienced AWS Data Engineer with strong expertise in ETL testing, cloud migration... ...data during cloud migrations, applying DevOps and CI/CD practices where needed Qualifications...
Devops
Work at office
Local area
Work from home
Flexible hours
3 days per week
Select Minds LLC
Dallas, TX
a month ago
Snowflake Data Engineer
$150k
...Our Addison, TX-based client is seeking a Data Engineer to build and maintain data pipelines for... .... The role is initially fully onsite, with potential for two remote days per week after... ...Silver/Gold layer management, and Azure DevOps/CI/CD for pipeline deployment....
Devops
Full time
Contract work
Remote work
2 days per week
Business Centric Technology
Addison, TX
1 day ago
Data Engineer
...for an Analytics / Reporting Engineer to join the Enterprise Reporting... ...You'll build and maintain the data pipelines, dashboards, and... ...design reports, and work directly with stakeholders who depend on... ...processing Experience with Azure DevOps for version control and...
Devops
Temporary work
HumCap, Inc.
Dallas, TX
a month ago
Data Operations Engineer
$125k - $155k
...means to work at a national law firm. With more than 1,400 attorneys across 46 offices... ...thrive professionally. Summary Data Operations Engineer to manage, optimize, and support... ...for production systems. Automation & DevOps Automate operational tasks using PowerShell...
Devops
Full time
Wilson Elser
Dallas, TX
4 days ago
GCP Data Engineer
...GCP Senior Big Data Engineer/ArchitectLooking for a Senior Big Data Engineer/Architect on Google... ...exhibiting right negotiating posture with customer and program teams to make the right... ...Software development life cycle (SDLC), DevOps, CI/CD. At least 2-year experience in...
Devops
Samprasoft
Irving, TX
2 days ago
Senior Data Platform Engineer
$107.12k - $160.68k
...a highly skilled and motivated Senior Data Platform Engineer, AVP (Python/PySpark, Microsoft SSIS, SQL... ...SSIS, and large-scale data platforms, with experience troubleshooting production data... ..., and data‑processing components. DevOps: Use Git-based source control and...
Devops
Full time
Citi
Irving, TX
2 days ago
Data Platform Engineer
...Take2 is hiring an AWS Lakehouse Data Engineer who is eligible to be sponsored for a... ...development, test, and production environments with consistent configuration and controlled... ...orchestration services. Experience with modern DevOps practices and tools such as Git,...
Devops
Remote work
Take2 Consulting LLC
Syracuse, NY
4 days ago
Data Engineer/Senior Data Engineer/Lead/Architect
...Data Engineer / Senior Data Engineer / Lead / Architect Location: Irving, TX / Chicago, IL... ...Engineering ~5+ years of hands-on experience with Azure Data technologies ~2+ years of... ...Power BI FHIR / HL7 Azure DevOps & Infrastructure as Code (IaC) AI/ML...
Devops
Long term contract
3 days per week
E-Solutions INC
Irving, TX
1 day ago
Principal Consultant - Data Engineering
$130k - $170k
...challenge of turning complex data into actionable insights? Do you... ...? If you're ready to lead with purpose and bring data to life... ...you. We are looking for data engineers and technical leaders who don'... ...tools like GitHub Actions, Azure DevOps, or Jenkins Exposure to...
Devops
Visa sponsorship
2 days per week
3 days per week
Torq
Dallas, TX
22 days ago
Data Platform Engineer
...care platform connecting people with the best care when they need... ...lanterncare.com. Lantern’s data organization is moving from a... ...one where domain teams (Data Engineering, Data Ops, Analytics Engineering... ...with GitHub Actions and/or Azure DevOps. Proficiency with Python or...
Devops
Temporary work
Flexible hours
Shift work
3 days per week
Lantern
Dallas, TX
4 days ago
Lead Data Platform Engineer
...we process high-volume payment data for some of the industrys... ...deliberately small and senior with executive-level support and minimal... ...is our highest-utilization engineering role on the project central to... ...Senior Technical Lead Senior DevOps engineer and mid-level developer...
Devops
Full time
Monday to Friday
Flexible hours
Aperia
Dallas, TX
1 day ago
Staff Engineer, Data Analysis (R5290)
...venture-backed defense-tech company with the mission of protecting service members... ...: We’re looking for a Staff Engineer to lead V-BAT’s fleet data analysis efforts. The V-BAT system produces... .... You will work closely with DevOps, Flight Software, Flight Test, Production...
Devops
Full time
Temporary work
Part time
Worldwide
Shield AI
Dallas, TX
2 days ago
ML Data Engineer
$125k - $140k
...Data Engineer -MLWe are looking for a highly skilled and passionate Data Engineer with strong Python/PySpark experience and understanding of AI/ML models to implement AI/ML models in OnPrem and Cloud environment. In this role, you will be responsible for implementing...
Tata Consultancy Services
Irving, TX
3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Data Engineer with DevOps. Be the first to apply!