Junior Data Engineer Job Description Template
Our company is looking for a Junior Data Engineer to join our team.
Responsibilities:
- Assisting in quality control of quantitative and qualitative research projects;
- Designing, building and support data processing pipelines to transform data using Hadoop technologies;
- Working with business analysts to understand business requirements and use cases;
- Developing and executing quality assurance and test scripts;
- Building survey and voter file data pipelines;
- Designing, building data assets in HIVE;
- Designing and developing tools for internal team use to simplify complex task flows;
- Designing schemas, data models and data architecture for Hadoop and HBase environments;
- Implementing data flow scripts using Unix / Hive QL / Oozie scripting;
- Creating scripts and workflows to automate repeated data processing tasks;
- Working with data team to identify opportunities for task automation and helpful tools.
Requirements:
- Understanding of best practices for building Data Lake and analytical architecture on Hadoop;
- Familiarity with MapR distribution of Hadoop;
- Scripting / programming with UNIX, Java, Python, Scala etc;
- Experience with Graph database;
- Experience in working in large environments such as RDBMS, EDW, NoSQL, etc;
- Knowledge in schema design, developing data models and proven ability to work with complex data;
- Understanding Hadoop file format and compressions;
- Hands-on experience with Hadoop, MapReduce, Hive, Oozie, Airflow, ElasticSearch;
- Knowledge in real time data ingestion into Hadoop;
- Experience with Test Driven Code Development, SCM tools such as GIT, Jenkins i.