Big Data Engineer
Guru Schools
Overview: Job Title : Big Data Engineer Location : Irving TX (hybrid role) (Only local candidates) Duration : 12 Months Roles and responsibilities:
- Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
- Perform data migration and conversion validation activities on different applications and platforms.
- Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
- Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
- Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
- Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
- Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
- Assist Manager in project planning, validation strategy development
- Provide support in User acceptance testing and production validation activities.
- Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
- Evaluate existing methodologies and processes and recommend improvements.
- Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.
- 8+ years of Software development and testing experience.
- 4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
- 4+ years of programming experience in Scala, Java or Python
- Experience in technical leading and mentoring the teams
- Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
- Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
- Strong knowledge in shell scripting Experience in Web Services - API development and testing.
- Experience with development and automated framework in a CI/CD environment.
- Experience with cloud environments - AWS or GCP is a plus.
- Knowledge of GIT/Jenkins and pipeline automation is a must.
- A solid understanding of common software development practices and tools.
- Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
- Good organizational skills and strong written and verbal communication skills.
- Working experience on large migration Projects is a big plus.
- Working experience on Google Cloud platform is a big plus
- Development experience for tools and utilities for monitoring and alert set etc.
- Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool
Vacancy posted 4 hours ago
Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy
- ...Job Title: ML Data Engineer Location: Irving, TX Full Time Job Description Roles & Responsibilities Job Title... ...with ML Ops/LLM Ops and distributed systems. Experience with Big Data platform like Cloudera Hadoop and Could platforms like AWS...Big dataFull time
Acestack
Irving, TX2 days ago $125k - $140k
...Roles & Responsibilities Job Title: Data Engineer Job Description: We are seeking a highly skilled and motivated Data Engineer... ...and efficient data solutions. Qualifications: • Big Data Frameworks Expertise: Demonstrated high proficiency in Apache...Big dataTata Consultancy Services
Irving, TX2 days ago- Job Title: Data Engineer - MEM SQL Location: New Jersey / Irving, TX / Tampa, FL Job Description: We are looking for an experienced Data Engineer... ...with cloud platforms (AWS / Azure / GCP) Knowledge of Big Data technologies (Spark, Kafka, etc.) Telecom domain experience...Big data
- Job Titile: Data Engineer Work Location; Irving, Texas Duration: 8+ Months Job Description: Specific Skills Experience or knowledge of relational... ...Pipeline, Apache Airflow, Argo. Experience or knowledge of big data tools: i.e., Hadoop, Spark, Kafka. Experience or knowledge...Big data
- Role description Job Title: Sr. DATA Engineer Work Location Irving TX Job Description: Mandatory Karat Interview Role Overview We are... ...AWS ecosystem The ideal candidate will have strong expertise in big data processing using PySpark and a deep understanding of data...Big data
- ...Summary We are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer, you will be... ...to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...Big dataFull timeLocal area
Vistra
Irving, TX3 days ago - Job Title: Sr. DATA Engineer Work Location Irving TX Job Description Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize big scalable data processing solutions within the Databricks Spark SQL Scala Python...Big dataTemporary workLocal area
- Lead AWS Data Engineer Job Title: Lead AWS Data Engineer Location: Miami, Florida or Irving, Texas (Hybrid) Duration: Contract to Hire A Career... ...simultaneously. Expertise in cloud data platforms (AWS) and big data ecosystems, with a focus on optimizing cost, performance,...Big dataContract work
$55 - $60 per hour
AI-Enabled Data Engineer (GCP) Duration: 12+ Months Contract Location: Chandler, AZ | Charlotte, NC | Minneapolis, MN | Las Colinas, TX (Hybrid... ...to ingest, transform, and distribute data for a variety of big data apps Leverage AI/Agentic frameworks to automate data management...Big dataHourly payContract work- ...expanding our efforts into complementary data technologies for decision support in areas... ...execution. Other data sets support search engine-based analytics. Exploration and deployment... ...Journey Analytics solutions Understanding of big data platforms and architectures, data...Big dataWork experience placement
$125.76k - $188.64k
...autonomous individuals. We are seeking a Senior Data Developer /Lead to join one of our small,... ...will possess deep technical expertise in big data technologies, including real-time... ...diverse data requirements from engineers and scientists into detailed technical specifications...Big dataFull time- ...the region is known for. We are actively recruiting for a Sr. Data Engineer to join our growing team! We believe analytics isn't a back-office... ...flows, and parameterizing pipelines in Azure Data Factory. Big Data Compute: Proven experience processing large-scale data using...Big dataWork at office
$107.12k - $160.68k
...DevelopmentCompany: CitiWe are seeking a highly skilled and motivated Senior Data Platform Engineer, AVP (Python/PySpark, Microsoft SSIS, SQL Server, ETL &... ...using scheduling/orchestration platforms such as AutoSys.Big Data Platforms: Work with Hadoop ecosystem technologies...Big dataFull timeCitigroup
Irving, TX2 days ago- ...are currently available and apply online.Job SummaryAs a Lead Data Engineer, you will be responsible for leading the design and implementation... ...tools such as Tableau or Power BI.•Experience with big data technologies such as Hadoop, Spark, or Kafka. •Experience...Big dataLocal area
- Data Platform Engineer We are seeking a world-class Data Platform Engineer to take complete operational and architectural ownership of one of the... ..., Unix Linux systems administration, and distributed big data platforms. You will design, scale, and automate massive...Big data
- We are looking for a Senior Data Engineer to help shape and expand a modern enterprise data ecosystem in Dallas, Texas. This role is ideal for... ...Databricks, Spark, PySpark, Python, Delta Lake, and related big data technologies.• Create and maintain layered lakehouse data...Big data
Robert Half
Dallas, TX2 days ago $195.3k - $199.2k
...will design, develop, and maintain scalable data pipelines and systems using Databricks on... .... You will be leading and executing data engineering projects from inception to completion,... ...engineering, with a strong background in big data technologiesProficiency in Databricks...Big dataApprenticeshipEasy workMcKinsey & Company
Dallas, TX1 day ago- ...Data Engineer Senior Design and Implement scalable solutions for ever-increasing data volumes, using big data/cloud technologies like Pyspark, Kafka, etc. Collaborate with cross-functional teams to understand data requirements and provide effective solutions. Implement...Big data
E-Solutions
Dallas, TX2 days ago - ...including but not limited to, Excel, Word, and Access. Proficiency in programming languages—preferably JavaScript Understanding of big data analytics and visualization tools Prior experience working with Robotic Process Automation (RPA) applications. Knowledgeable...Big data
Judit Inc
Coppell, TX7 days ago - ...Job Description Job Description Job Title: Senior Data Engineer with DevOps Duration : Full Time Location : Pittsburgh, PA, Cleveland... ...~5+ years of experience in data engineering and big data processing ~ Strong expertise in Apache Spark (Spark Core...Big dataHourly payFull time
System One
Dallas, TXa month ago - ...Data Engineer Loc: Dallas, TX Client: CVS Responsibilities: # Contribute to Mail Intelligence Mission: Support the development of... ...science, data engineering, or a related field, with a focus on big data technologies and cloud platforms. Technical Skills: You...Big data
Argyle Infotech
Dallas, TX3 days ago - ...Sr. Data Engineer Dallas, Texas, United States About the Job Role: Senior Data Engineer Required Qualifications: ~ Strong... ...platforms (i.e., Azure, GPC, AWS) ~ Experience building automated big data pipelines ~ Experience performing data analysis and data...Big data
Inizio Partners
Dallas, TX4 hours ago - EXL is seeking an experienced Data Engineering Lead to drive architecture decisions and mentor distributed teams. You will own end-to-end data pipelines, focusing on on-prem Hadoop ecosystems, Spark processing, and effective scheduling with CA7/Control-M. The role requires...Big data
- TEK-Engineers is a Texas-based software solutions company delivering automated and robust software engineering solutions, IT services, and data-driven products. This position focuses on real-time big data initiatives in a fully remote environment with Dallas metro preference...Big dataRemote job
- ...Job Description Job Description Data Engineer Who we are Iapetus Infrastructure oversees a portfolio of specialized field services... ...a wide variety of data sources using SQL, Fabric, and Azure big data technologies. Keep our data separated and secure....Big dataFull timeContract workWork at office
Iapetus Infrastructure Services
Dallas, TXa month ago - ...Purple Drive Job Title: Data Engineer Location: Dallas, TX / Plano, TX (Onsite) Job Summary We are seeking a Data Engineer with... ...warehousing concepts (Kimball, star schema, SCD). ~ Hands-on with big data frameworks (Spark, Hadoop) and NoSQL databases (MongoDB,...Big data
Purple Drive
Dallas, TX4 days ago - ...Description:We are seeking a highly motivated and self-driven Data Engineer with a strong background in building scalable batch and streaming... ...lifecycle methodologies (e.g., BMAD, Spec Kit)Experience with big data and distributed processing technologies such as Hadoop, Spark...Big dataFull timeWork experience placementLocal areaFlexible hours
CoreLogic
Dallas, TX14 hours ago - Data Engineer Required Skills • Strong skills in Python Programming • Strong experience in PySpark • Experience with data pipeline and workflow... ...with a wide variety of data sources using: SQL Eligible to workP big data technologies Skills: Digital : Python~MySQL Diverse LynxBig data
$44 per hour
...Payrate: $44.00 - $44.05/hr. Summary We are seeking a Data Engineer to join our team for a 12-month assignment at our Coppell, TX... ...languages, preferably JavaScript. Understanding of big data analytics and visualization tools. Prior experience with...Big dataHourly payFull timeLocal areaFlexible hoursAditi Consulting
Coppell, TX2 days ago$100 - $110 per hour
...Remote Lead Data Engineer Dallas, Texas, United States $ 100.00 - 110.00 (US Dollar) Remote Lead Data Engineer needs 7+ years experience... .... Implement data ingestion and processing with the help of Big Data technologies Data processing/transformation using various...Big dataRemote workGlobal Channel Management
Dallas, TX4 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!
Related searches
- senior data center engineer Irving, TX
- senior cloud data engineer Irving, TX
- senior data integration developer Irving, TX
- data developer Irving, TX
- data engineer Irving, TX
- big data cloud engineer Irving, TX
- data engineering intern summer Irving, TX
- data engineer analytics Irving, TX
- big data devops engineer Irving, TX
- junior big data engineer Irving, TX