Senior Data Engineer (1043) - DataSF - Office of City Administrator

Senior Data Engineer – DataSF, City and County of San Francisco Essential Duties Platform Administration : Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization. Administer our platform with a focus on data democratization and accessibility, while protecting privacy and security. Pipeline Development : Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources. Design infrastructure to support both streaming and batch processes, and both structured and unstructured data sources. Infrastructure as Code (IaC) : Use Terraform to define, deploy, and manage data infrastructure, ensuring our pipelines are reproducible, version‑controlled, and production‑ready. Best Practices & Innovation : Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing. Think creatively to find new ways to improve our data platform's capabilities and efficiency. Provide guidance to department partners on data engineering best practices. Collaboration : Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders in diverse domains to understand data requirements and build solutions that meet their needs. Monitoring & Support : Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability. Minimum Qualifications Education : Associate degree in computer science, computer engineering, information systems, or a closely related field from an accredited college or university OR equivalent in terms of total course credits/units (at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in one of the fields above). Experience : Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network. Substitution : Additional experience may be substituted for the required degree on a year‑for‑year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/forty‑five (45) quarter units with a minimum of ten (10) semester or fifteen (15) quarter units in one of the fields above or a closely related field. Desirable Qualifications Technical Knowledge Hands‑on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks. Expertise in writing advanced, performant SQL and using tools like dbt for SQL‑based data transformation and modeling. Strong programming skills in Python (with libraries like pandas, PySpark) for data processing and automation. Proficiency with an Infrastructure as Code tool, preference for Terraform. Experience building and deploying data pipelines using orchestration tools such as Azure Data Factory, Airflow, Dagster, or similar technologies. Deep understanding of data warehousing concepts, data modeling, and modern ELT principles. Understanding of data governance, data security, and data privacy principles. Experience with real‑time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe). Experience deploying and managing data pipelines for machine learning models. Collaboration and Communication Skills Strong problem‑solving skills with ability to design practical and effective data solutions. Excellent verbal and written communication skills, including ability to explain technical concepts to non‑technical stakeholders. A collaborative mindset with enthusiasm to work across diverse, cross‑functional teams. Mission Alignment Commitment to equity, transparency, and ethical data use. Passion for public service and using data to improve government services. Empathy for San Francisco’s diverse communities and a drive to make data and services more accessible for SF residents. Interest or experience in public sector data or social impact work. Equal Employment Opportunity Statement The City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition, HIV/AIDS status, genetic information, marital status, sexual orientation, gender identity, gender expression, military and veteran status, or other protected classification under the law. All information will be kept confidential according to EEO guidelines. #J-18808-Ljbffr City and County of San Francisco