Big Data Engineer

Guru Schools

Irving, TX

Overview:

Job Title : Big Data Engineer

Location : Irving TX (hybrid role) (Only local candidates)

Duration : 12 Months

Roles and responsibilities:

Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
Perform data migration and conversion validation activities on different applications and platforms.
Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
Assist Manager in project planning, validation strategy development
Provide support in User acceptance testing and production validation activities.
Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
Evaluate existing methodologies and processes and recommend improvements.
Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.

Must have :

8+ years of Software development and testing experience.
4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
4+ years of programming experience in Scala, Java or Python
Experience in technical leading and mentoring the teams
Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
Strong knowledge in shell scripting Experience in Web Services - API development and testing.
Experience with development and automated framework in a CI/CD environment.
Experience with cloud environments - AWS or GCP is a plus.
Knowledge of GIT/Jenkins and pipeline automation is a must.
A solid understanding of common software development practices and tools.
Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
Good organizational skills and strong written and verbal communication skills.

Nice to have :

Working experience on large migration Projects is a big plus.
Working experience on Google Cloud platform is a big plus
Development experience for tools and utilities for monitoring and alert set etc.
Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool

Skills:

ETL,,Spark SQL,

Apply

Vacancy posted 2 days ago

Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy

AIA COMMS -BIG DATA ENGINEER
...Job title COGNIZANT LOOKING FOR BIG DATA ENGINEER Job summary BigData Engineer Primary Skill Set • 4+ years of experience working on data engineering teams related to BigData and Teradata • Expert knowledge of SQL NoSQL and Python • Good experience with Apache Spark ecosystem...
Big data
Shift work
Avani Technology Solutions
Irving, TX
a month ago
Big Data Engineer in Irving TX
...Role: Big Data Engineer Location: Irving TX Duration: 5+ Months Job Description: Minimum 5+ years of experience in IT industry with 2+ years of experience in Big Data implementing complete Hadoop solutions along with Scala Knowledge on complete Hadoop...
Big data
Work experience placement
Avani Technology Solutions
Irving, TX
9 days ago
Data Engineer
...Big Data Engineer Location: Irving, TX (Day 1 Onsite-Hybrid) Hexaware Primary Skills: Hadoop, Hive & Spark Framework Responsibilities: Experience with big data processing and distributed computing systems like Spark. Implement ETL pipelines and data...
Big data
InterSources
Irving, TX
15 hours ago
PySpark Data Engineer
$120k - $140k
...Job : PySpark Data Engineer Location : Irving, TX(Onsite) Fulltime Only Salary Range: $120000 to $140000/Annum + Full Time Benefits... ...data solutions for data generation, collection, and processing in Big Data environment using predominantly PySpark/Python. Your...
Big data
Full time
Work experience placement
Diverse Lynx
Irving, TX
15 hours ago
Data Engineer
...Job Description Job Description Job Title: Data Engineer – MEM SQL Location: New Jersey / Irving, TX / Tampa, FL Job Description:... ...Experience with cloud platforms (AWS / Azure / GCP) Knowledge of Big Data technologies (Spark, Kafka, etc.) Telecom domain...
Big data
Qode
Irving, TX
3 days ago
Data Engineer
..., Impala, HBase spark scala batch pipelines GCP, AWS - data pipelines Oozie, Airflow, Composer - scheduling tools... ...preferred. Job Description ~ Experience in data engineer, Data Analytics ~ Big Data Technologies Expert in Python ~(Developer level Expertise...
Big data
Omni Inclusive
Irving, TX
2 days ago
IAM Data Lake -Data Engineer
...deliverables within Information Security Engineering and contribute to large-scale planning related... ...This position is focused on IAM Data Lake -Data Engineering responsibilities.... ...Data Lake in Google Cloud Platform using big data tools and technologies. Experience with...
Big data
Mindlance
Irving, TX
3 days ago
Data engineer with MemSQL
...Data engineer with MemSQL Long Term Contract Irving, TX Job Summary: s a Data Engineer specializing in... ...MemSQL, data warehousing concepts and design. Knowledge of big data technologies ( Hadoop, Spark, etc.) is a plus....
Big data
Long term contract
Abode Tech Zone
Irving, TX
15 hours ago
Lead Data Engineer
...Job Title: Lead Data Engineer Location: Irving, TX - Hybrid Must have: ~ Expert in Databricks, PySpark and Azure data services... ...technologies such as Terraform or Pulumi. ~ Exposure to big data tools (Spark, Hadoop, Kafka) preferred. ~ Experience with...
Big data
Jobs via Dice
Irving, TX
4 days ago
Sr. Pyspark Data Engineer
...Role: Sr. PySpark Data Engineer - Fulltime Location: Irving, TX Job Description: We are seeking a skilled PySpark... ...to join our team. The ideal candidate will have expertise in big data processing, ETL pipeline development, and cloud-based data...
Big data
Full time
SARIAN Co
Irving, TX
1 day ago
Data Engineer
$43.59 - $51.59 per hour
...Genesis10 is currently seeking a Data Engineer for a contract position with a Global Financial Institution located in Columbus, OH or Las Colinas... ...experience building Data Lakes in Google Cloud Platform using big data tools and technologies. This position is focused on IAM...
Big data
Hourly pay
Permanent employment
Contract work
Genesis10
Irving, TX
2 days ago
Sr Data Ops Engineer
...client, a leading organization in the technology and data solutions industry, is seeking a Sr Data Ops Engineer to join their team. As a Sr Data Ops Engineer, you... ...ingest, transform, and distribute data for various big data applications Leverage AI/Agentic frameworks...
Big data
Flexible hours
Experis/Manpower Group
Irving, TX
3 days ago
Senior Data Ops Engineer
...SPOTLIGHT - 3.30 MUST HAVES Work in a large data environment Senior Data Ops Engineer (Lead Information Security Engineer P4) Lead... ..., transform, and distribute data for a variety of big data apps ? Leverage AI/Agentic frameworks to automate...
Big data
Mindlance
Irving, TX
2 days ago
Data Developer / Princ Engr - Data Engineering (ETL)
...Principal Engineer - Data Engineering (ETL) Location: Irving, TX 75038 (Hybrid – 3 days in office) Duration: 24 months (possible extension... ...the organization. Desired Skills: Experience with big data technologies (e.g., Hadoop, Spark, Clickhouse), NoSQL...
Big data
Work at office
Local area
TriOptus LLC
Irving, TX
15 hours ago
Data Engineer - Python/PySpark
$125k
...Data Engineer - Python/PySpark Location: Irving TX (3 Days onsite/week) Fulltime Salary: $125K (Including QPLC) Job Description: •... ...(transformations, shuffles, optimization). • Experience with big data frameworks such as Hadoop and Spark. • Proficiency with...
Big data
Full time
3 days per week
Diverse Lynx
Irving, TX
23 hours ago
Principal Data Engineer
...Principal Data Engineer We are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer... ...to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...
Big data
Local area
TXU Energy
Irving, TX
15 hours ago
Data Engineer
$77 - $82 per hour
...Provide technical guidance, mentorship, and code reviews to junior data engineers, fostering a culture of excellence and continuous improvement.... ...Python programming skills. o Proven experience with Big Data ecosystems, including Cloudera and/or Data Bricks. o Hands...
Big data
Contract work
Work at office
Immediate start
Remote work
Flexible hours
Axelon
Irving, TX
1 day ago
Analytics Developer
...Responsibilities: Responsible for developing strategies for effective data analysis and reporting. Selects, configures, and implements... ...: ~ Senior Developer for development of IT project using Big Data/Hadoop technologies (Kafka, Storm, Hive, Elastic search)....
Big data
Contract work
Immediate start
Pyramid Consulting
Irving, TX
10 days ago
Data Engineer - IAM Data Lake (GCP)
$55 - $61 per hour
...A client of Innova Solutions is immediately hiring for a Data Engineer - IAM Data Lake (GCP). Position type: Contract (12 months w/ extension... ...Build and operate IAM Data Lake solutions on GCP using big data tools. Design and implement batch and streaming ingestion...
Big data
Hourly pay
Contract work
Temporary work
Work experience placement
Immediate start
Remote work
Worldwide
Flexible hours
Innova Solutions
Irving, TX
4 days ago
Sr Data Engineer
...Company that Empowers you to Build your Future As a Senior Data Engineer, you are responsible for analyzing large amounts of business... ...load data from a wide variety of data sources using SQL and AWS big data technologies Explore and learn the latest AWS technologies...
Big data
Live in
Work at office
Local area
Lennar
Irving, TX
15 hours ago
Data Engineer
$99.6k - $156.5k
...Data Engineer Imagine your future with us! Since 1971, Paychex has been at the forefront of simplifying HR, payroll, and benefits for American... ...in Working with SQL, Snowflake, Databricks, Spark and/or other Big Data technologies. ~4 years of experience in Experience...
Big data
Permanent employment
Full time
Contract work
Part time
Internship
Work at office
Local area
Paycor
Irving, TX
15 hours ago
Data Engineer
...expanding our efforts into complementary data technologies for decision support in areas... ...execution. Other data sets support search engine-based analytics. Exploration and deployment... ..., and writing skills. Understanding of big data platforms and architectures, data stream...
Big data
Work at office
Flexible hours
2 days per week
GMAC Financial Services
Irving, TX
2 days ago
Senior Data Engineer - Vice President
...Senior Data Engineer - Vice President Citi is seeking a highly skilled and experienced Senior Data Engineer to join our dynamic and innovative... ...services for data storage, processing, and analytics. Big Data Technologies: Work extensively with big data frameworks and...
Big data
Citi
Irving, TX
4 days ago
Starburst Senior Data Engineer- Vice President
...Starburst Senior Data Engineer- Vice President Working at Citi is far more than just a job. A career with us means joining a team of more... ...hands-on experience in data engineering, data warehousing, or big data analytics projects. ~ Minimum of 3 years of direct experience...
Big data
Citi
Irving, TX
15 hours ago
GCP Data Engineer
...GCP Data Engineer Location: Irving, TX (Hybrid) Duration: 12 Month Contract Overview: This position is focused on IAM Data... ...experience building Data Lake in Google Cloud Platform using big data tools and technologies. Experience with Hadoop/HDFS is highly...
Big data
Contract work
Strategic Staffing Solutions
Irving, TX
15 hours ago
Lead Data Engineer
...Lead Data Engineer Software Guidance & Assistance, Inc., (SGA), is searching for a Lead Data Engineer for a CONTRACT/RIGHT TO HIRE assignment... ...advanced Python programming skills. Proven experience with Big Data ecosystems, including Cloudera and/or Data Bricks....
Big data
Contract work
Flexible hours
SGA
Irving, TX
2 days ago
Applied Analytics Engineer
...to join our team as an Applied Analytics Engineer . The successful candidate will be responsible... ..., optimizing, and maintaining robust data pipelines that ingest, clean, and transform... ...Azure, GCP) ~ Practical experience with big data processing frameworks such as Spark...
Big data
Full time
Vulcan Materials
Irving, TX
4 days ago
Big Data Developer
...Position: Big Data Developer Location: Irving, TX Duration: Long Term Responsibilities: 1). Develop software using Hadoop technologies like Spark, Scala, Python, Hbase, Hive, Cloudera. 2). Fine tune application performance...
Big data
3B Staffing LLC
Irving, TX
3 days ago
Lead Data Engineer/Data Architect
...Lead Data Engineer/Data Architect Location: Onsite Irving, TX (Local ONLY – No Relo) Duration: 9 Months MoI: Video Banking Domain exp... ...As a Lead Data Engineer/Architect, you will be responsible for big data/analytics projects that gather and integrate large volumes...
Big data
Work experience placement
Local area
Shift work
InterSources
Irving, TX
15 hours ago
GCP Data Engineer with AI/ML
...GCP Data Engineer with AI/ML Integration & MLOps We are seeking a GCP Data Engineer with deep, hands-on architectural and development experience in Google Cloud Platform’s big data ecosystem. You will be responsible for designing, building, and optimizing a modern...
Big data
United IT
Irving, TX
15 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!