Hadoop Data Engineer Job Description Template
Our company is looking for a Hadoop Data Engineer to join our team.
Responsibilities:
- Assemble large, complex data sets that meet functional / non-functional business requirements;
- Create and maintain optimal data pipeline architecture;
- Keep our data separated and secure across national boundaries through multiple data centers and AWS regions;
- Work with data and analytics experts to strive for greater functionality in our data systems.
Requirements:
- Experience with Hortonworks preferred;
- 3-5+ years of experience handling Hadoop clusters;
- Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala;
- Experience with Big Data ML toolkits, such as TensorFlow or SparkML;
- Knowledge of Lambda Architecture, as well as its advantages and drawbacks;
- Knowledge of various ETL techniques and frameworks, such as Flume;
- Strong experience with Hadoop, MapReduce, HDFS;
- Experience with various messaging systems, such as Kafka or RabbitMQ;
- Strong knowledge of distributed computing principles;
- Spark & Scala expertise required;
- Expert with big data tools: Hadoop, Spark, Kafka, etc;
- Expert with relational SQL and NoSQL databases, including Postgres and Cassandra;
- Strong analytic skills related to working with unstructured datasets;
- Expert with AWS cloud services: EC2, EMR, RDS, Redshift;
- Bachelor’s Degree in Computer Science, Computer Engineering or a closely related field.