Big Data Engineer
Guru Schools
Overview: Job Title : Big Data Engineer Location : Irving TX (hybrid role) (Only local candidates) Duration : 12 Months Roles and responsibilities:
- Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
- Perform data migration and conversion validation activities on different applications and platforms.
- Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
- Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
- Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
- Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
- Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
- Assist Manager in project planning, validation strategy development
- Provide support in User acceptance testing and production validation activities.
- Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
- Evaluate existing methodologies and processes and recommend improvements.
- Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.
- 8+ years of Software development and testing experience.
- 4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
- 4+ years of programming experience in Scala, Java or Python
- Experience in technical leading and mentoring the teams
- Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
- Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
- Strong knowledge in shell scripting Experience in Web Services - API development and testing.
- Experience with development and automated framework in a CI/CD environment.
- Experience with cloud environments - AWS or GCP is a plus.
- Knowledge of GIT/Jenkins and pipeline automation is a must.
- A solid understanding of common software development practices and tools.
- Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
- Good organizational skills and strong written and verbal communication skills.
- Working experience on large migration Projects is a big plus.
- Working experience on Google Cloud platform is a big plus
- Development experience for tools and utilities for monitoring and alert set etc.
- Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy
- ...Role description Job Title: Big DATA Developer with Python Work Location : Tampa,FL Job Description: Seeking a Senior Python Data Engineer with 3 to 5 years of experience skilled in Apache Spark and Python to design and implement scalable data processing...Big data
LTM
Irving, TX4 days ago - ...Job Title: Data Engineer – MEM SQL Location: New Jersey / Irving, TX / Tampa, FL Job Description: We are looking for an experienced... ...Experience with cloud platforms (AWS / Azure / GCP) Knowledge of Big Data technologies (Spark, Kafka, etc.) Telecom domain...Big data
Qode
Irving, TX1 day ago $59.53 - $67.53 per hour
...Data Scientist / Machine Learning Engineer - Hybrid Genesis10 is currently seeking a Data Scientist / Machine Learning Engineer - Hybrid position with... ...pipelines, and model deployment frameworks Exposure to big data technologies, distributed computing, or real-time...Big dataHourly payContract work3 days per weekGenesis10
Irving, TX2 days ago- ...Data Engineer With MemSQL Long Term Contract Irving, TX Job Summary: As a Data Engineer specializing in MemSQL, you will design... ...MemSQL, data warehousing concepts and design. Knowledge of big data technologies (Hadoop, Spark, etc.) is a plus. Strong communication...Big dataLong term contract
Abode Techzone LLC
Irving, TX2 days ago - ...Role: Sr. PySpark Data Engineer - Fulltime Location: Irving, TX Job Description: We are seeking a skilled PySpark... ...to join our team. The ideal candidate will have expertise in big data processing, ETL pipeline development, and cloud-based data...Big dataFull time
SARIAN Co
Irving, TX1 day ago - ...Data Engineer - IAM Data Lake (Google Cloud Platform) Location: Irving, TX (Preferred) or Ohio (Hybrid) Duration: 12 Month Contract... ...large-scale data processing initiatives using modern cloud and big data technologies. The ideal candidate will have hands-on...Big dataContract work
Strategic Staffing Solutions
Irving, TX3 days ago $125k - $140k
...Roles & Responsibilities Job Title: AWS Certified Data Engineer Job Description: We are seeking a highly skilled... ...) ecosystem. The ideal candidate will have strong expertise in big data processing using PySpark and a deep understanding of data...Big dataTata Consultancy Services
Irving, TX4 days ago$125k - $140k
...Roles & Responsibilities Job Title: Databricks Data Engineer Job Description: We are seeking a highly skilled and motivated... ...Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). • Big Data Concepts: Deep understanding of distributed computing,...Big dataTata Consultancy Services
Irving, TX3 days ago- ...Principal Data Engineer We are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer... ...to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...Big dataLocal area
TXU Energy
Irving, TX20 hours ago - ...to join our team as an Applied Analytics Engineer . The successful candidate will be responsible... ..., optimizing, and maintaining robust data pipelines that ingest, clean, and transform... ...Azure, GCP) ~ Practical experience with big data processing frameworks such as Spark...Big dataFull time
Vulcan Materials
Irving, TX4 days ago - ...PySpark Data Engineer We are seeking a skilled PySpark Data Engineer to join our team and drive the development of robust data processing... ...candidate possesses strong PySpark development skills, experience with big data technologies, and the ability to work in a fast-paced,...Big data
Photon
Dallas, TX20 hours ago $115k - $129k
PDF Solutions is hiring a Data Engineer in Dallas, Texas to enhance their Big Data Analytics platform. The role involves architecting scalable NoSQL databases, maintaining high performance of Cassandra clusters, and utilizing DevOps practices for infrastructure automation...Big data- McKinsey & Company is seeking a skilled Data Engineer to join their team in Dallas, Texas. The ideal candidate will have 6+ years of experience in data engineering with a focus on Databricks, Azure, and big data technologies. You will lead the development of scalable data...Big data
- NMC2 is seeking a Lead Data Engineer in Dallas, Texas, to advance its big data engineering and analytics capabilities. This role is pivotal in designing and optimizing scalable data platforms, real-time data pipelines, and data lake architectures across cloud environments...Big data
- A leading investment firm in Dallas, TX is seeking a Lead Data Engineer to advance big data engineering and analytics capabilities. The role involves designing scalable data platforms, implementing data mesh architecture, and ensuring data quality. Candidates should have...Big data
- ...Job Description (Posting). : We are seeking a talented Data Engineer with 3+ years of experience to join our team in the telecom industry... ...like Python and Scala for data processing. Familiarity with big data frameworks such as Apache Spark, Hadoop, or similar....Big data
Damco Solutions
Dallas, TX4 days ago - ...Data Engineer APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client... ...Relevant certifications (e.g., Snowflake, Azure Cloud, and Big Data) are a plus We are committed to fostering a diverse, inclusive...Big dataWork at officeRemote work1 day per week
APN Consulting
Dallas, TX20 hours ago - ...DEPARTMENT: Product: Engineering POSITION: Data Engineer REPORTS TO: Director, Engineering SUPERVISES... ...queuing, stream processing, and highly scalable ‘big data’ data stores Ability to implement ETL using SSIS...Big dataWork at officeRemote workWork from home
Wondr Health
Dallas, TX1 day ago - ...Job Title-Senior Data Engineer Location: Remote Duration: 6 Months Contract to Hire (Quarterly Onsite for a few days in Dallas... ...and advanced analytics. We specialize in cutting-edge Big Data, Machine Learning, and Custom Software strategy, analysis,...Big dataContract workRemote work
Saviance
Dallas, TX4 days ago - ...Senior Data Engineer Dallas, TX - Hybrid (3x in office/week) About Lantern Lantern is the specialty care platform connecting people... ...environment and you take the time to celebrate both the small & big wins. INCLUSION is a core tenant of your personal beliefs. A...Big dataTemporary workWork at office
EmployerDirect Healthcare
Dallas, TX4 days ago - ...Data Engineer Employment Type: Full-Time, Mid-level CGS is seeking a passionate and driven Data Engineer to support a rapidly growing... ...manipulate data (Python, R, SQL, SAS). ~ Strong knowledge of big data analysis and storage tools and technologies. ~ Strong understanding...Big dataFull timeFlexible hours
Contact Government Services LLC
Dallas, TX5 days ago - ...Consultant (Data Engineer Skills Et) Do you want to support innovative clients at the intersection of cutting-edge technologies and impactful... ...team. Responsibilities: Lead the design of scalable big data platforms/solutions used to ingest and process accurately...Big dataDay shift
evolv consulting
Dallas, TX20 hours ago $195.3k - $199.2k
Consulting Senior Data Engineer - Periscope Job ID: 108653 Atlanta Boston Dallas New Jersey Apply NowApply Later Do you... ...of experience in data engineering, with a strong background in big data technologies ~ Proficiency in Databricks on Azure, Spark...Big dataHourly payApprenticeshipWork at officeEasy workMcKinsey & Company
Dallas, TX5 days ago- ...breakthroughs that shape the industries of tomorrow. Its engineers build critical infrastructure to eliminate... .... The Position We are seeking a Lead Data Engineer to play a critical role in advancing NMC²'s big data engineering and analytics capabilities. This...Big dataTemporary workFlexible hours
NorthMark Strategies
Dallas, TX4 days ago - ...General Purpose This role oversees and manages the work of Data Engineer who design and develop ETL pipelines and processes in our Cloud... ...engineering, such as Python, Java, Scala, or SQL. ~ Proficiency with big data technologies and platforms, such as Hadoop, Spark, Apache...Big dataLocal area
Coca-Cola Southwest Beverages
Dallas, TX20 hours ago $106.9k - $176.5k
...working world. We are seeking a highly skilled Senior Consultant Data Engineer with expertise in cloud data engineering, specifically... ...solutions. ~ Expertise in Databricks and experience with Spark for big data processing. ~ Proven experience in at least two end-to-...Big dataSummer holidayFlexible hoursEY
Dallas, TX3 days ago- NorthMark Compute and Cloud LLC is seeking a Lead Data Engineer to enhance its big data capabilities. The candidate will design and optimize scalable data platforms and real-time pipelines within the organization's Big Data Engineering and Analytics team. This role demands...Big data
- ...Job Title: Principal Engineer I - Senior Data Engineer Location: Block 23 What you'll do: As the Principal Data Engineer, you... .... Implement methods to include unstructured data and big data. What you'll need: ~8+ years' experience with data...Big data
Western Alliance Bank
Dallas, TX7 days ago - Alkami Technology, based in Dallas, is seeking a Data Engineering Manager to lead the big data engineering team and drive data strategy. The role involves overseeing the design and maintenance of scalable data systems, mentoring a high-performing team, and collaborating...Big dataRemote job
- A technology solutions firm in Dallas is seeking a Big Data Infrastructure DevOps/Support Administrator. This role involves supporting and developing the Enterprise Infrastructure Big Data Service using Java, Python, and Hadoop. Ideal candidates will have a strong background...Big data
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!
Related searches
- senior data integration developer Irving, TX
- data engineer contract Irving, TX
- data science developer Irving, TX
- senior data center engineer Irving, TX
- data developer Irving, TX
- remote data engineer Irving, TX
- sr data engineer Irving, TX
- big data cloud engineer Irving, TX
- big data devops engineer Irving, TX
- etl data engineer Irving, TX