Data engineer

Data Engineer


Primary Responsibilities
• Support Data Science team by applying data mining techniques, feature engineering, algorithm optimization and scaling machine learning models in big data, cloud environment
• Support data aggregation jobs in a production environment as well as tune and optimize existing code base for multiple data products
• Process structured and unstructured data, validate data quality, and help design data quality tests in a big data environment
• Facilitate end to end solution design reviews with other members of the team that include communicating design, requirements, feature sets, functionality and limitations of systems/applications
• Build data movement pipelines leveraging SQL, Python and Spark that meet all functional & non-functional requirements
• Develop efficient feature capabilities to support proper data extraction & curation for analytical purposes
• Develop self-service analytics to enable users to better understand and assess impacts of key business drivers on financial results.


Minimum Qualifications
• 5+ years working as data engineer / database developer (Hadoop, cloud experience preferred)
• 5+ years of experience in data warehousing / big data environment, preferably in wireless industry and/or finance function
• Experience building & optimizing data pipelines on cloud solutions such as AWS & Azure
• Extensive experience in Spark, Python, PySpark, Scala, SQL/Hive
• Familiar with Spark MLlib, SParkSQL
• Writing complex SQL statements and performing data discovery
• Proven experience with data modeling, complex data structures, data processing, data quality, and data lifecycle
• Experience in API design, SDK architecture, and mobile software lifecycle development practices
• Strong organizational and time management skills, ability to prioritize a large group of diverse projects simultaneously, and deliver on multiple threads of work
• Experience working in agile environments on multiple projects simultaneously
• Excellent written and verbal communication skills
• Bachelor's degree in computer science or related technical field