Research and Evaluations Lead (1053) - Department of Technology

Job Description

Job Description:\n\nCompany Description\n\nDepartment: TechnologyJob class: 1053 IS Business Analyst - SeniorSalary range: $134,082 - $168,688 annuallyRole type: Permanent Exempt What does this mean?Hours: Full-time\nSpecific information regarding this recruitment process is listed below: \nIMPORTANT: A COVER LETTER MUST BE SUBMITTED AS PART OF THE APPLICATION IN ORDER TO BE CONSIDERED. FAILURE TO SUBMIT ONE MAY RESULT IN DISQUALIFICATION FROM CONSIDERATION.\nAppointment Type: Permanent Exempt - This position is a full-time position excluded by the Charter from the competitive civil service examination process and shall serve at the discretion of the appointment officer. The anticipated duration of this project position is up to thirty-six (36) months.\n\n Application Opening: Thursday, September 17, 2026\n Application Deadline: Apply immediately. This recruitment is considered open until filled, but will close no sooner than October 1, 2026 at 11:59pm.\n\nWork Location: Incumbent will conduct the majority of work at the Department of Technology, (1 S Van Ness, Ave San Francisco, CA 94103). However, there may be situations where the incumbent will be required to work at other sites throughout the City and County of San Francisco as necessary.\nNature of Work: This is a full-time position. The department supports a hybrid work schedule in compliance with City & County of San Francisco policies on remote work which currently requires onsite work of a minimum of two (2) days per week.\nAbout the Department of Technology\nWhy Work for the Department of Technology (DT)? DT is the centralized technology services provider in the City and County of San Francisco (CCSF). We deliver technology infrastructure and services to approximately 33,000 employees! With an annual operating budget of over $140M and approximately 260 employees, DT provides a host of services that range from Public Safety radio and wiring and Network services to Enterprise Support and the Cloud.\nBenefits of Working for CCSF: In addition to challenging and rewarding work, the City provides a generous suite of benefits to its employees.\n\n Job security, pension, and robust retirement options\n Competitive pay with consistent bi-yearly or yearly increases\n Hybrid Work with a minimum of 20% of time spent in our office in San Francisco, California for all IT related roles\n Generous paid time off, family leave, and more!\n Diverse work environment in a diverse city\n Union protections and representation\n Career development and growth — move between departments, learn on the job, or take subsidized/reimbursed classes!\n\nPEOPLE-CENTERED SOLUTIONS — have a powerful, meaningful effect on the community each day with people at the heart of every solution!\nCLOSING THE DIGITAL DIVIDE — bring the benefits of the internet to low-income and marginalized residents!\nSHINE A LIGHT ON WHAT MATTERS — join an award-winning production team at SFGovTV to help residents watch legislators or learn more about what makes this City great!\nDRIVE INNOVATION — deliver new, cutting-edge technology to residents and city partners to help San Francisco serve its residents!Job Description\n\nAre you passionate about shaping the future of technology while ensuring its responsible and ethical use? The City and County of San Francisco (CCSF) is seeking a forward-thinking and driven Research and Evaluation Lead to join our Emerging Technologies team at the Department of Technology. This is your chance to help shape how artificial intelligence (AI) and other emerging technologies are enabled across CCSF, applying user-centered, data-driven approaches to understand staff needs and support effective and responsible adoption.\nIn this role, you’ll collaborate with City departments and other partners to identify needs, evaluate emerging technology tools, and turn research findings into practical recommendations. Your work will help guide decisions about what to build, procure, pilot, and scale, as well as how to support staff through training and other enablement efforts.\nThe Research and Evaluation Lead serves as a key member of the Department of Technology's (DT) Emerging Technologies team. This role advances two primary objectives: staff enablement and program evaluation.\n1. Staff Enablement\n The Lead is responsible for identifying the needs of City staff related to artificial intelligence and other emerging technology tools. These insights guide the prioritization of tools to build, procure, or pilot, and inform the development of training required for staff to use these tools effectively.\n2. Program Evaluation\n The Lead evaluates whether emerging technology tools can be deployed effectively and responsibly at scale. This includes advising departments on appropriate evaluation methodologies and collaborating with academic and non-profit partners to design rigorous and practical evaluations.\nThe position requires strong qualitative user research skills combined with quantitative and mixed‑methods evaluation expertise. The Lead works closely with the Director of Emerging Technologies, Lead AI Engineer, and Principal AI Policy Analyst, while partnering with City departments implementing emerging technology pilots.\nThe essential functions of this position include:\n\n Conduct qualitative and quantitative research on how City staff use, adopt, or discontinue use of AI tools, to identify friction points, unmet needs, and adoption barriers.\n Translate research findings into prioritized input for the Emerging Technologies team, helping inform which tools, features, product builds, pilots, or procurements to invest in.\n Assess staff training needs and recommend training or upskilling approaches to help staff use AI tools effectively and responsibly.\n Present findings and recommendations to the Emerging Technologies team, department partners, and City leadership.\n Design and run evaluations of AI and other emerging technology tools the City procures, builds, pilots, and rolls out — including experiments and quasi-experimental designs where feasible — to determine whether they work, for whom, and with what safeguards they're worth scaling.\n Support the City's emerging technology pilot evaluations by building the City's capacity to design and carry out rigorous, practical evaluations through academic and non-profit partnerships.\n Pair usage-log data with survey responses and departmental operational metrics to measure actual outcomes, and document methodology and findings to inform future purchasing and rollout decisions.\nQualifications\n\nDESIRABLE QUALIFICATIONS:\n\n Experience with qualitative methods (in-depth interviews, contextual inquiry, focus groups, usability testing) and quantitative methods (survey design, statistics, usage-log or behavioral data analysis).\n Experience translating research findings into product or feature prioritization, ideally in a fast-moving or pilot-stage environment.\n Experience designing and running program or pilot evaluations, including experimental or quasi-experimental designs (e.g., A/B tests, pre/post comparisons, matched comparison groups).\n Experience assessing training or skill-development needs and translating them into training recommendations.\n Experience with survey tooling (e.g., Fillout, Qualtrics) and data analysis (Excel/Sheets required; SQL, R, or Python a plus).\n Experience maintaining and leveraging academic and non-profit research partnerships.\n Ability to manage research and evaluation projects in fast-moving, ambiguous environments, balancing rigor with practical constraints, shifting priorities, and tight timelines.\n Strong written and verbal communication skills, with the ability to communicate findings across technical and non-technical audiences.\n Experience in public service or civic technology, and working with staff across different job classes, tenure levels, and language backgrounds.\n\nMINIMUM QUALIFICATIONS:\nThese minimum qualifications establish the education, training, experience, special skills and/or license(s) which are required for employment in the classification. Please note, additional qualifications (i.e., special conditions) may apply to a particular position and will be stated on the job announcement.\nExperience:\nThree (3) years in the information systems field, including system analysis, business process design, development and implementation of business application solutions or IT project management.\nEducation:\nAn associate degree in business administration, public administration, information systems, economics, finance, computer science, UX/UI design or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units [i.e., at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in one of the fields above or a closely-related field].\nSubstitution:\nOne year of additional experience as described above may be substituted for the required degree.\nCompletion of the 1010 Information Systems Trainee Program may be substituted for the required degree.\n\nVerification:\n Please make sure it is clear in your application exactly how you meet the minimum qualifications. Every application is reviewed to ensure that you meet the minimum qualifications as listed in the job ad. Please review our articles on Employment Application and Minimum Qualifications and Verification of Experience and/or Education for considerations taken when reviewing applications. Applicants may be required to submit verification of qualifying education and experience at any point in the application and/or departmental selection process. \nNote: Applications completed improperly may be cause for ineligibility, disqualification or may lead to lower scores. Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco. Resumes will not be accepted in lieu of a completed City and County of San Francisco application. \nSelection Procedure:\n The selection process will include evaluation of applications in relation to minimum requirements and assessment of candidates’ job-related knowledge, skills and abilities as well as a review of the required cover letter. Depending on the number of applicants, the Department may establish and implement additional screening mechanisms to evaluate candidate qualifications. This typically includes an oral interview and/or a written or performance exercise.\nIf this becomes necessary, only those applicants whose qualifications most closely meet the Department needs will be invited to continue in the selection process. Applicants meeting the minimum requirements are not guaranteed advancement in the selection process.\n\nAdditional Information\n\n Information About The Hiring Process \n Conviction History \n Employee Benefits Overview \n Equal Employment Opportunity \n Disaster Service Worker \n ADA Accommodation \n Veterans Preference \n Right to Work\n Copies of Application Documents \n Diversity Statement\n\nHow to Apply\n Applicants are encouraged to apply immediately as this recruitment may close at any time, but not before October 1, 2026 at 11:59pm.\n\n Your application MUST include a resume AND a cover letter. To upload, please attach using the "additional attachments" function.\n You may contact Jason Wong via email at View email address on ziprecruiter.com with questions regarding this opportunity.\n Late or incomplete submissions will not be considered. Mailed, hand delivered or faxed documents/applications will not be accepted.\n\nApplicants will receive a confirmation email from View email address on ziprecruiter.com that their online application has been received in response to every announcement for which they file. Applicants should retain this confirmation email for their records. Failure to receive this email means that the online application was not submitted or received. \nApplicants may be contacted by email about this recruitment and, therefore, it is their responsibility to ensure that their registered email address is accurate and kept up-to-date. Also, applicants must ensure that email from CCSF is not blocked on their computer by a spam filter. To prevent blocking, applicants should set up their email to accept CCSF mail from the following addresses (@sfgov.org, @sfdpw.org, @sfport.com, @flysfo.com, @sfwater.org, @sfdph.org, @asianart.org, @sfmta.com, @sfpl.org, @dcyf.org, @first5sf.org, @famsf.org, @ccsf.edu, @smartalerts.info, and @smartrecruiters.com). \nThe Office of the City Administrator is committed to ensuring that the City's services are inclusive, efficient, equitable, and culturally competent for San Franciscans of all races, ethnic backgrounds, religions, and sexual orientations. This commitment requires comprehensive review and thorough analysis of existing practices and policies to remove barriers to real inclusion. \nWe are also committed to ensuring that we have a safe, equitable, and inclusive workplace for individuals of all races. This includes creating opportunities for hiring, promotion, training, and development, for all employees, including but not limited to Black, Indigenous, and people of color (BIPOC).\n\nThe City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of their sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition (associated with cancer, a history of cancer, or genetic characteristics), HIV/AIDS status, genetic information, marital status, sexual orientation, gender, gender identity, gender expression, military and veteran status, or other protected category under the law.