Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Big Data Developer

Techvilla Solutions

We are looking for an experienced Big Data Developer with strong expertise in big data technologies, distributed data processing, and large-scale data analysis. The ideal candidate should have hands-on experience with Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala, along with performance tuning and troubleshooting in the Cloudera Big Data ecosystem.

Key Responsibilities
  • Develop, maintain, and optimize big data applications and data processing pipelines.
  • Work with Kafka and Apache Spark for real-time and batch data processing.
  • Perform data analysis, data wrangling, and processing of large volumes of structured and unstructured data.
  • Write and optimize complex SQL queries using Hive and Impala.
  • Monitor, troubleshoot, and resolve performance issues in the Cloudera Big Data ecosystem.
  • Use Splunk for log analysis, monitoring, and troubleshooting.
  • Manage and monitor batch workflows and job scheduling using AutoSys.
  • Work in Linux-based environments for development, deployment, and operational support.
  • Collaborate with business users and cross-functional teams to deliver high-quality data solutions.
  • Ensure data quality, reliability, scalability, and performance of big data applications.
Required Technical Skills
  • Strong hands-on experience with Kafka, Spark, Scala, and SQL.
  • Experience with Hive, Impala, Hadoop, and the Cloudera ecosystem.
  • Proficiency in data analysis and data wrangling for large datasets.
  • Experience in performance analysis, tuning, troubleshooting, and issue resolution.
  • Hands-on experience with Splunk and AutoSys.
  • Strong knowledge of Linux-based infrastructures.
  • Experience in distributed data processing and big data technologies.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Big Data Developer in Brighton, NY vacancy
  • Position: Data Platform Engineer Location: Rochester, NY Duration: 12+ Months Hadoop and...  ..., development and maintenance of the big data platform. This platform hosts structured...  ...when technical support is not sufficient.- Develops standards, policies and procedures for the... 
    Big data
    Temporary work
    Work experience placement
    Internship

    3B Staffing LLC

    Rochester, NY
    1 day ago
  •  ...Job Summary: Location : Rochester, NY The Data Center Hands and Feet Support Engineer provide on-site physical support in the data center environment. This role acts as the remote hands for engineering teams, assisting with equipment installations, troubleshooting... 
    Suggested
    Local area
    Remote work
    Shift work
    Night shift

    HCLTech

    Rochester, NY
    1 day ago
  • $24 - $26 per hour

     ...with the technical operations department on developing future MKS products Reports issues...  ...role At MKS, it's all about courage, big ideas, and a serious passion for innovation...  ...intelligence, cloud technology, and big data. We're the behind-the-scenes support for... 
    Big data
    Permanent employment
    Full time
    Monday to Friday

    MKS Instruments

    Rochester, NY
    12 hours ago
  • $19 - $22 per hour

     ...title and location of the role At MKS, it's all about courage, big ideas, and a serious passion for innovation. Winning here is...  ...energy & storage, artificial intelligence, cloud technology, and big data. We're the behind-the-scenes support for some amazing technology.... 
    Big data
    Permanent employment
    Full time

    MKS Instruments

    Rochester, NY
    1 day ago
  • $110.7k - $218.3k

     ...marketing, commerce, sales, and service. We are a team of strategists, data scientists, operators, creatives, designers, engineers, and...  ...at ****@*****.***. Recruiting tips From developing a stand out resume to putting your best foot forward in the... 
    Suggested
    Local area

    Deloitte

    Rochester, NY
    3 days ago
  • $180k - $210k

     ...entrepreneurial corner of the agency world. We're hiring a Senior Manager, Data Engineering to lead and grow our Data Engineering team within Data & Analytics. This is a true player‑coach role. You'll develop a team of engineers, including early‑career talent, and still get... 
    Permanent employment
    Work experience placement
    Local area
    Shift work

    OUTFRONT Media

    Rochester, NY
    3 days ago
  • Position Summary Data Engineer-Consultant AI&EAre you an experienced, passionate pioneer in technology who wants to work in a...  ...stakeholder collaboration.Technical lead, responsible for Design, develop, and operationalize scalable healthcare data platformsDesign,... 

    Deloitte

    Rochester, NY
    3 days ago
  •  ...engineers and product leaders as part of your team. Together, you'll develop and enhance Instacart's marketplace systems. You will use...  ...foundational problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine... 
    Permanent employment
    Full time
    Work at office
    Remote work
    Work from home
    Flexible hours

    Instacart

    Rochester, NY
    4 days ago
  • $107.66k - $161.7k

     ...and our engineers are encouraged to think big and experiment with new ideas. Using...  ...product. You will also play a key role in developing tools and abstractions that our other developers...  ...Learning systems -- from prototyping, data pipelines and training, to realtime LLM... 
    Remote job
    Full time
    Work experience placement
    Internship

    Quora

    Rochester, NY
    4 days ago
  • $105k - $155k

    Mindex is looking for a Data Engineering Solution Lead to head a focused team migrating Informatica pipelines into Maia (Matillion) and...  ..., escalating risks and explaining trade-offs early. Coach and develop data engineers, which may include direct reports, and evaluate new... 
    Remote job
    Work at office
    Visa sponsorship
    Work visa
    Flexible hours

    Mindex Technologies

    Rochester, NY
    1 day ago
  •  ...We are seeking an experienced AI/ML Developer with strong expertise in Machine Learning, Python, Natural Language Processing (NLP), and banking Large Language Models (LLMs). The ideal candidate will develop and implement AI/ML solutions, analyze complex business problems... 

    2T Consulting

    Brighton, NY
    3 days ago
  • $90k - $140k

     ...Mindex is looking for a Data Engineer to build the data pipelines, models, and warehouse structures behind SchoolTool’s interactive...  ...warehouse structures for SchoolTool’s dashboards and analytics. Develop analytics solutions with Python, Apache Spark, Snowflake, SQL... 
    Full time
    Work at office
    Visa sponsorship
    Work visa
    Flexible hours

    Mindex

    Rochester, NY
    2 days ago
  •  ...Who We’re Looking For Megazone Cloud is looking for a Data Engineer Co-op who is a "builder" at heart and eager to apply technical skills to real-world customer problems. We need students who are curious, driven, and comfortable asking "what if?" to solve complex... 
    Full time
    Immediate start

    megazone

    Rochester, NY
    7 days ago
  •  ...Asia's largest cloud provider, building the next generation of data solutions from the ground up. We're a team of builders, thinkers...  ...powerful, efficient data models. Write and review critical code: Develop clean, optimized, and automated solutions using Python and SQL.... 
    Full time
    Immediate start
    Flexible hours

    megazone

    Rochester, NY
    7 days ago
  •  ...payroll and benefits administration, Paylocity goes further by developing tools that HR and businesses need to compete for talent and deliver...  ...team develops infrastructure and tooling that enable data-driven decisions and AI/ML experiences at scale for millions of... 
    Full time
    Internship
    Work at office
    Worldwide

    Paylocity

    Rochester, NY
    2 days ago
  •  ...Data Scientist I This is a Hybrid position based out of the Fairport, New York Office Position Summary: The Data Scientist will assist in developing, implementing, and automating predictive analytical models leveraging the Company’s data assets and statistical... 
    Permanent employment
    Work experience placement
    Work at office
    Remote work

    ConServe

    Fairport, NY
    1 day ago
  • $95k - $154k

     ...confidence, and familiarity with tools used in real software or data teams. If your applications are not converting, your profile...  ...looking for entry-level software programmers, Java Full stack developers, Python/Java developers, Data analysts/Data Engineers/ Data Scientists... 
    Remote job
    Full time

    SynergisticIT

    Rochester, NY
    2 days ago
  • $140k

     ...The Senior Agentic AI Engineer is the operational owner of the Constituent Data Layer (CDL) and the safe, secure rollout of AI agents across CaterTrax® constituent surfaces. Reporting to the Director, Enterprise Architecture as a member of the Data Layer group, this... 
    Full time
    Work at office

    CaterTrax

    Rochester, NY
    2 days ago
  • $124k - $280k

     ...solutions for clients within our Advisory practice. In this role, you will bring together data engineering, analytical modeling, prompt engineering, and retrieval-augmented generation to develop AI applications grounded in useful information. Working with cloud and product... 
    Full time
    H1b

    PwC

    Rochester, NY
    3 days ago
  • Known for being a great place to work and build a career, KPMG provides audit, tax and advisory services for organizations in today's most important industries. Our growth is driven by delivering real results for our clients. It's also enabled by our culture, which encourages...
    H1b
    Local area

    KPMG

    Rochester, NY
    1 day ago
  • $77k - $202k

     ...OpportunityAs a Senior Associate - Azure AI Engineer you will design and develop generative AI solutions for clients within our Advisory practice...  ...generation and vector databases to ground responses in client data. You will also develop agent workflows with LangGraph,... 
    Full time
    H1b

    PwC

    Rochester, NY
    3 days ago
  • $127.2k - $201.3k

     ...automated controls and guardrails across the NIST CSF functions an OWASP Top 10 frameworks, such as mitigating prompt injection, preventing data leakage, and establishing secure model- and tool-interaction patternsLead complex security engineering initiatives end-to-end with... 
    H1b
    Local area

    KPMG

    Rochester, NY
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Developer. Be the first to apply!