Data Engineer

Data Engineer

The Department of City Planning (DCP) plans for the future of New York City, working to create thriving and dynamic neighborhoods with access to housing and jobs, resilient infrastructure, and a vibrant public realm. The Department engages communities to develop inclusive plans, expands housing access and economic opportunity, and plans for long-term sustainability. The Department of City Planning offers staff meaningful opportunities to help shape the future of New York City and address some of its most pressing challenges. The Department values public service, collaboration, and creativity, and supports professional development.

Information Technology Division's (ITD) vision is to become a dynamic and vibrant technology delivery organization of the New York City government that leads business transformations and technology innovations to enable citizen-centric smart urban planning, while promoting a culture of collaboration, talent development, and equity.

The Data Engineering team is seeking a Data Engineer. The team's mission is to create and publish high quality public datasets, build transparent and automated data pipelines using open-source technologies, develop a data ecosystem by offering comprehensive documentation and analytics resources, and bring people together across teams and agencies to share data and learn from each other. The team is continually evolving, and this new hire will have the ability and support to improve existing infrastructure and processes and drive forward new projects and initiatives. The ideal candidate will be excited about advancing the adoption of new technologies in DCP and City government and passionate about data production, quality, and accessibility.

Under limited supervision, the Data Engineer's primary areas of responsibility include, but are not limited to:

• Designing, deploying, and scaling of current infrastructure to ensure that Data Engineering maintains and improves existing processes, and expands its abilities and offerings.
• Understanding the technologies used by the team and recommending new technologies. Ideating, designing, and implementing improvements to products and processes to ensure that the team is using the best tools for the job.
• Building Data Engineering data products, which involves:
• Operating data ingestion pipelines and managing data storage,
• Running scripts that transform input data by standardizing, geocoding, merging, aggregating, and performing basic spatial manipulations,
• Reviewing outputs by writing custom scripts and/or using established QAQC tools to identify errors, inconsistencies, and edge cases in datasets.
• Designing new data products from concept to completion, working on specifications with product owners, creating technical designs with team members, helping to plan sprints, contributing to documentation, and delivering a finished product.
• Collaborating with other data engineers in sprint planning, design, code review, and pair programming.
• Learning and wrangling complex legacy data systems and bringing the infrastructure into a modern development stack.
• Sharing your accomplishments and lessons learned with NYC Planning, other city agencies, and the world via talks, presentations, blogs, or other mediums.

The City of New York is an inclusive equal opportunity employer committed to recruiting and retaining a diverse workforce and providing a work environment that is free from discrimination and harassment based upon any legally protected status or protected characteristic, including but not limited to an individual's sex, race, color, ethnicity, national origin, age, religion, disability, sexual orientation, veteran status, gender identity, or pregnancy.