Data Engineer

Data Engineer

Location: McLean, VA (Tuesday and Wednesday onsite)

Duration: 6+ Months contract (with possible conversion)

Job Description:

• Cleanse, manipulate and analyze large datasets (Structured and Unstructured data – XMLs, JSONs, PDFs) using Hadoop platform.
• Develop Python, Py Spark, Spark scripts to filter/cleanse/map/aggregate data.
• Manage and implement data processes (Data Quality reports)
• Develop data profiling, deduping logic, matching logic for analysis
• Programming Languages experience in Python, Hive Query Language, Py Spark and Spark for data ingestion
• Programming experience in RDBMS and Bigdata using Hive, Hadoop platform
• Present ideas and recommendations on Hadoop and other technologies best use to management
• Good understanding of Functional programming and Object-oriented programming
• Detail oriented. Excellent communication skills (verbal and written)
• Must be able to manage multiple priorities and meet deadlines

Required Skills:

• 5+ years of experience in processing large volumes and variety of data (Structured and unstructured data, writing code for parallel processing, XMLS, JSONs, PDFs) - Mandatory
• Programming experience in Python, Spark, SQL, HQL for data processing and analysis
• Strong SQL experience
• 3+ years of experience – using Hadoop platform and performing analysis. Familiarity with Hadoop cluster environment and configurations for resource management for analysis work

Desired Skills:

• 2+ years of experience with containerization and orchestration. Hands on experience with GIT, AWS, Kubernetes, Docker etc.