Senior Data Engineer

OverviewJob PurposeICE Mortgage Technology, a subsidiary of Intercontinental Exchange, Inc. (ICE) is seeking a full-time Development Lead to join a team focused on delivering large-scale, connected property and mortgage centric datasets. These datasets support high profile clients in the real estate, mortgage and financial services industries.The ideal candidate will have a strong background in both micro and big data analytics, with expertise in data manipulation and advanced analytic techniques. This role requires a deep understanding of the entire software development lifecycle-from requirements gathering and design to coding, unit testing, debugging, and production support.As a Development Lead, candidate will provide strategic direction and hands-on guidance to a team of developers, ensuring high quality data delivery and continuous improvement. The successful candidate will join a growing data and operational team that plays a critical role in ICE Mortgage Technology’s long-term strategy. The ideal candidate combines technical proficiency and problem-solving skills with a proactive attitude, professionalism, and excellent communication abilities. Strong written and oral communication is required as this role involves frequent collaboration with stakeholders across the real estate and financial sectors to drive business outcome.ResponsibilitiesEffectively interact with business and data fulfillment teams for solving design and operational challenges.Meet with internal departments to explain data structures, discuss changes in data delivery, walk through best processes for data processing, and assist in determining solutionsAnalyze requirements and propose technical solutions in delivering projects within expected parameters of cost, time, and qualityData format conversion to make it more appropriate for product needsCreate experimental and new frameworks to collect and deliver data as well as build tools to automate data collection and data delivery.Create solutions to remove corrupted data and fixing coding errors and related problemsSearch through large sets of data for meaningful informationCreate meaningful reports that represents the data and presentations for business usesUse statistical tools to identify, analyze, and interpret patterns and trends in complex data sets that could be helpful for the diagnosis and prediction of data issues and trendsDevelop and maintain databases, data systems – reorganizing data in a readable formatDesign and support operational processes for data ingestion, loading, and replicationWork with project managers to compile a work breakdown structure of tasks for a given deliverable and provide realistic estimates for completing the assignmentDevelop and unit test all functions created or modified according to software development best practicesCollaborate with other Developers and QA Engineers to execute full cycle integration, functional and regression testing.Continuously advocate to improve the system performance and propose better solutions.Brainstorm identifying ML/AI/NLP innovations to apply in quantitative analysis on market microstructure and in surveillance to help in advance automation, knowledge discovery, decision-making and insightsCollaborate on implementing and prototyping new algorithms and write code for novel ML/AI/NLP solutionsKnowledge and ExperienceAdvanced or basic degree (PhD with few years’ experience, or MS / BS with many years’ experience) in a quantitative field such as CS, EE, Information sciences, Statistics, Mathematics, Economics, Operations Research, or related, with focus on data-driven statistical analysis & modeling.Strong, proven programming skills in MS SQL, Python and other well-known database applications.Strong knowledge of MS SQL, Spark SQL (PySpark), Iceberg tables, and Python libraries used for data science.Experience in Databricks.Experience working in Agile teams comprised of 5-8 cross-skilled engineers.Solid understanding of version control tools (ideally Git).Experience in applying data-driven statistical analysis & modeling solutions to quantitative analysis of financial market data.Experience with writing complex programs and implementing custom algorithms in above mentioned and other environments.Experience beyond using open-source tools as-is, and writing custom code on top of, or in addition to, existing open-source frameworks.Excellent communication skills (oral and written) to explain complex algorithms, solutions to stakeholders across multiple disciplines, and ability to work in a diverse team.Experience in an applied R&D environment, working in an agile and innovation-lab culture.Preferred SkillsExperience in data visualization and business analytics tools like Alteryx, Tableau and Microsoft PowerBI.Experience in scripting languages like PowerShell and AWK/bash shell scripting is beneficial.Experience in data management, data analytics middleware, platforms and infrastructure, cloud and fog computing.Additional experience with GPU programming for training deep learning models, and cloud environments such as AWS.#LI-MA1 Intercontinental Exchange, Inc. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to legally protected characteristics.