Big Data Developer
Techvilla Solutions
We are looking for an experienced Big Data Developer with strong expertise in big data technologies, distributed data processing, and large-scale data analysis. The ideal candidate should have hands-on experience with Kafka, Spark, Scala, SQL, Hadoop, Hive, and Impala, along with performance tuning and troubleshooting in the Cloudera Big Data ecosystem.
Key Responsibilities
- Develop, maintain, and optimize big data applications and data processing pipelines.
- Work with Kafka and Apache Spark for real-time and batch data processing.
- Perform data analysis, data wrangling, and processing of large volumes of structured and unstructured data.
- Write and optimize complex SQL queries using Hive and Impala.
- Monitor, troubleshoot, and resolve performance issues in the Cloudera Big Data ecosystem.
- Use Splunk for log analysis, monitoring, and troubleshooting.
- Manage and monitor batch workflows and job scheduling using AutoSys.
- Work in Linux-based environments for development, deployment, and operational support.
- Collaborate with business users and cross-functional teams to deliver high-quality data solutions.
- Ensure data quality, reliability, scalability, and performance of big data applications.
Required Technical Skills
- Strong hands-on experience with Kafka, Spark, Scala, and SQL.
- Experience with Hive, Impala, Hadoop, and the Cloudera ecosystem.
- Proficiency in data analysis and data wrangling for large datasets.
- Experience in performance analysis, tuning, troubleshooting, and issue resolution.
- Hands-on experience with Splunk and AutoSys.
- Strong knowledge of Linux-based infrastructures.
- Experience in distributed data processing and big data technologies.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Big Data Developer in Brighton, NY vacancy
- Position: Data Platform Engineer Location: Rochester, NY Duration: 12+ Months Hadoop and... ..., development and maintenance of the big data platform. This platform hosts structured... ...when technical support is not sufficient.- Develops standards, policies and procedures for the...Big dataTemporary workWork experience placementInternship
- ...Job Summary: Location : Rochester, NY The Data Center Hands and Feet Support Engineer provide on-site physical support in the data center environment. This role acts as the remote hands for engineering teams, assisting with equipment installations, troubleshooting...SuggestedLocal areaRemote workShift workNight shift
$24 - $26 per hour
...with the technical operations department on developing future MKS products Reports issues... ...role At MKS, it's all about courage, big ideas, and a serious passion for innovation... ...intelligence, cloud technology, and big data. We're the behind-the-scenes support for...Big dataPermanent employmentFull timeMonday to Friday$19 - $22 per hour
...title and location of the role At MKS, it's all about courage, big ideas, and a serious passion for innovation. Winning here is... ...energy & storage, artificial intelligence, cloud technology, and big data. We're the behind-the-scenes support for some amazing technology....Big dataPermanent employmentFull time$110.7k - $218.3k
...marketing, commerce, sales, and service. We are a team of strategists, data scientists, operators, creatives, designers, engineers, and... ...at ****@*****.***. Recruiting tips From developing a stand out resume to putting your best foot forward in the...SuggestedLocal area$180k - $210k
...entrepreneurial corner of the agency world. We're hiring a Senior Manager, Data Engineering to lead and grow our Data Engineering team within Data & Analytics. This is a true player‑coach role. You'll develop a team of engineers, including early‑career talent, and still get...Permanent employmentWork experience placementLocal areaShift work- Position Summary Data Engineer-Consultant AI&EAre you an experienced, passionate pioneer in technology who wants to work in a... ...stakeholder collaboration.Technical lead, responsible for Design, develop, and operationalize scalable healthcare data platformsDesign,...
- ...engineers and product leaders as part of your team. Together, you'll develop and enhance Instacart's marketplace systems. You will use... ...foundational problems executing on a high throughput system with dynamic data. About the Job: Design, develop, and deploy machine...Permanent employmentFull timeWork at officeRemote workWork from homeFlexible hours
$107.66k - $161.7k
...and our engineers are encouraged to think big and experiment with new ideas. Using... ...product. You will also play a key role in developing tools and abstractions that our other developers... ...Learning systems -- from prototyping, data pipelines and training, to realtime LLM...Remote jobFull timeWork experience placementInternship$105k - $155k
Mindex is looking for a Data Engineering Solution Lead to head a focused team migrating Informatica pipelines into Maia (Matillion) and... ..., escalating risks and explaining trade-offs early. Coach and develop data engineers, which may include direct reports, and evaluate new...Remote jobWork at officeVisa sponsorshipWork visaFlexible hours- ...We are seeking an experienced AI/ML Developer with strong expertise in Machine Learning, Python, Natural Language Processing (NLP), and banking Large Language Models (LLMs). The ideal candidate will develop and implement AI/ML solutions, analyze complex business problems...
$90k - $140k
...Mindex is looking for a Data Engineer to build the data pipelines, models, and warehouse structures behind SchoolTool’s interactive... ...warehouse structures for SchoolTool’s dashboards and analytics. Develop analytics solutions with Python, Apache Spark, Snowflake, SQL...Full timeWork at officeVisa sponsorshipWork visaFlexible hours- ...Who We’re Looking For Megazone Cloud is looking for a Data Engineer Co-op who is a "builder" at heart and eager to apply technical skills to real-world customer problems. We need students who are curious, driven, and comfortable asking "what if?" to solve complex...Full timeImmediate start
- ...Asia's largest cloud provider, building the next generation of data solutions from the ground up. We're a team of builders, thinkers... ...powerful, efficient data models. Write and review critical code: Develop clean, optimized, and automated solutions using Python and SQL....Full timeImmediate startFlexible hours
- ...payroll and benefits administration, Paylocity goes further by developing tools that HR and businesses need to compete for talent and deliver... ...team develops infrastructure and tooling that enable data-driven decisions and AI/ML experiences at scale for millions of...Full timeInternshipWork at officeWorldwide
- ...Data Scientist I This is a Hybrid position based out of the Fairport, New York Office Position Summary: The Data Scientist will assist in developing, implementing, and automating predictive analytical models leveraging the Company’s data assets and statistical...Permanent employmentWork experience placementWork at officeRemote work
$95k - $154k
...confidence, and familiarity with tools used in real software or data teams. If your applications are not converting, your profile... ...looking for entry-level software programmers, Java Full stack developers, Python/Java developers, Data analysts/Data Engineers/ Data Scientists...Remote jobFull time$140k
...The Senior Agentic AI Engineer is the operational owner of the Constituent Data Layer (CDL) and the safe, secure rollout of AI agents across CaterTrax® constituent surfaces. Reporting to the Director, Enterprise Architecture as a member of the Data Layer group, this...Full timeWork at office$124k - $280k
...solutions for clients within our Advisory practice. In this role, you will bring together data engineering, analytical modeling, prompt engineering, and retrieval-augmented generation to develop AI applications grounded in useful information. Working with cloud and product...Full timeH1b- Known for being a great place to work and build a career, KPMG provides audit, tax and advisory services for organizations in today's most important industries. Our growth is driven by delivering real results for our clients. It's also enabled by our culture, which encourages...H1bLocal area
$77k - $202k
...OpportunityAs a Senior Associate - Azure AI Engineer you will design and develop generative AI solutions for clients within our Advisory practice... ...generation and vector databases to ground responses in client data. You will also develop agent workflows with LangGraph,...Full timeH1b$127.2k - $201.3k
...automated controls and guardrails across the NIST CSF functions an OWASP Top 10 frameworks, such as mitigating prompt injection, preventing data leakage, and establishing secure model- and tool-interaction patternsLead complex security engineering initiatives end-to-end with...H1bLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Developer. Be the first to apply!





