Senior Staff Engineer

Big Data EngineerWe are looking for an experienced big data engineer to design and build our data pipeline and ETL processes, which are scalable, highly performant and able to handle diverse datasets including IoT, leveraging the latest big data technologies such as Hadoop/HDFS, Kafka, and HBase etc.ResponsibilitiesBuild data pipelines and ETL from heterogeneous sources including IoT using Kafka, Flume, Sqoop, Spark Streaming and custom code etc. Your solutions will support both real-time or near real-time applicationsPartner with product owners, data analyst and data scientists, to better understand their requirements, find bottlenecks and come up with the best resolution, etc.Expand and grow data platform capabilities to solve new data problems and challengesYou will work closely with development and QA teams both local and oversea to design ingestion pipelinesYou will also need to mentor other team membersDesired Skills & Experience6+ years of software development experience with minimum 2-year hands-on experience in Hadoop ecosystem and other Big Data technologies (Kafka, Cassandra, MongoDB and Elastic search)Expert level software development experience using Java and/or PythonHands-on experience with building big data ingestion pipelines using Hadoop, HBase, Kafka, Elastic Search etcKnowledge and hands-on experience with relational databases (Greenplum and Oracle MySQL/Postgres etc) and database schema designExperience with Unix/Linux and shell scriptingKnowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes and testingAbility to take a project from scoping requirements through actual launch of the projectBS/MS Degree in Computer Science or related fieldExcellent inter-personal and teamwork skills, having experiences working with oversea teams is a plus