Sr DevOps Engineer

Job Description

Job Description:\n\nPosition Summary\nThe Senior DevOps Engineer supports Credit Union of Texas's vision to be the trusted financial partner for our members and our community by building and owning the end-to-end software development lifecycle (SDLC) on Azure DevOps. The role modernizes and standardizes the SDLC for existing applications, implements CI/CD and multi-environment (Dev, UAT, Prod) strategies, and makes the Node.js application code changes needed to support them. The role establishes secure secrets management, artifact management, static and dynamic application security testing (SAST with JFrog and DAST with Invicti), code quality and test coverage gates (SonarQube), Infrastructure as Code, automated testing, and observability practices (Datadog APM and Logs) that make releases repeatable, traceable, and audit-ready. Once the foundation is in place, the Senior DevOps Engineer contributes to Node.js development and onboards new applications onto the standardized SDLC. The role uses CUTX-approved AI tools under defined governance, with mandatory human review of any AI-assisted code or configuration after it reaches production.Key ResponsibilitiesCI/CD, Source Control & Release Management\n\n Design and implement CI/CD pipelines (YAML) in Azure DevOps for existing and new applications.\n Manage Azure Repos, including branching strategy, branch policies, pull request workflows, and code review gates.\n Design and implement multi-environment strategies (Dev, UAT, Prod) with automated promotion, including the infrastructure and application changes required to support them.\n Set up release strategies such as approvals, environment promotion, and rollback.\n Implement feature flags using Azure App Configuration to reduce release risk.\nSecurity, Secrets & Code Quality\n\n Implement and standardize secrets management using Azure Key Vault and Managed Identity across applications, including the Node.js code changes needed to load secrets and configuration securely at runtime.\n Establish secure secret lifecycle practices, including rotation and access controls.\n Implement static application security testing (SAST) using JFrog Advanced Security, with critical and high findings blocking merges and releases.\n Integrate SonarQube for code quality and unit test coverage analysis (coverage thresholds, bugs, code smells, duplication, and maintainability), with quality gates enforced on every pull request and pipeline run.\n Integrate Invicti for dynamic application security testing (DAST) against deployed applications in Dev and UAT, with critical and high findings blocking promotion to Production.\n Coordinate triage and remediation of SAST, DAST, dependency, and secret scanning findings with development teams and Information Security, and track them to closure in Azure Boards.\n Implement dependency, vulnerability, and secret scanning (such as JFrog Xray, Snyk, GitHub Advanced Security for Azure DevOps, or Gitleaks) with results enforced in pipelines.\n Enforce code consistency standards with ESLint and Prettier in pull request checks.\nInfrastructure, Artifacts & Containers\n\n Implement Infrastructure as Code (Terraform, Bicep, or ARM) for repeatable environment provisioning.\n Set up and manage JFrog Artifactory for package and artifact management, including npm and Docker registries.\n Containerize applications with Docker where appropriate, and support a future move to Azure Kubernetes Service (AKS).\nTesting, Monitoring & Observability\n\n Build automated testing into pipelines, including unit tests (Jest) and end-to-end tests (Playwright or Cypress).\n Set up monitoring and observability with Datadog APM and Datadog Log Management across all environments, alongside Azure Monitor and Application Insights for Azure platform metrics.\n Instrument Node.js applications with the Datadog APM tracer and structured logging, correlate traces with logs, and build dashboards, monitors, and alerts for service health, performance, and release impact.\nNode.js Development & Application Onboarding\n\n Make code changes in Node.js applications to support DevOps practices such as configuration management, health checks, logging, and automated tests.\n Contribute to Node.js feature development and bug fixes as DevOps priorities allow.\n Onboard new applications onto the standardized SDLC process.\nGovernance, Documentation & Collaboration\n\n Set up Azure Boards for work tracking, with work items linked to commits, pull requests, and releases for end-to-end traceability and audit readiness.\n Document pipelines, environments, runbooks, and processes in the Azure DevOps Wiki, and train team members on them.\n Coordinate with Application Development, IT Operations, Information Security, and Compliance to align SDLC standards with enterprise priorities and control requirements.\nPerformance Outcomes & KPIs\nOutcome\nPrimary KPI\nReporting Cadence\nTarget / Direction\nApplications run on a standardized, automated SDLC.\nSDLC Adoption Rate — percent of in-scope applications deployed through standardized Azure DevOps YAML pipelines with automated environment promotion.\nQuarterly\n▲ 100% of in-scope applications [Timeline — confirm with Hiring Manager]\nReleases are reliable and low-risk.\nChange Failure Rate — percent of production deployments that require rollback, hotfix, or incident remediation.\nMonthly\n▼ ≤ 15%\nIssues are recovered from quickly.\nMean Time to Restore (MTTR) — average time to restore service after a failed production change.\nMonthly\n▼ [Target — confirm with Hiring Manager]\nSecrets and credentials are managed securely.\nSecrets Management Compliance — percent of in-scope applications loading secrets from Azure Key Vault via Managed Identity, with zero secrets in source control.\nQuarterly\n▲ 100%\nCode meets quality and security standards after release.\nSecurity Gate Enforcement — percent of production releases that passed JFrog SAST and Xray dependency scans, Invicti DAST scans, SonarQube code quality and test coverage gates, and secret scans with no unresolved critical or high findings.\nMonthly\n▲ 100%\nCode changes are covered by automated tests.\nTest Coverage — percent of in-scope applications meeting the SonarQube unit test coverage threshold on new code.\nMonthly\n▲ ≥ 80% coverage on new code [confirm threshold with Hiring Manager]\nProduction applications are fully observable.\nObservability Coverage — percent of production applications instrumented with Datadog APM and centralized Datadog logs, with active monitors and alerts.\nQuarterly\n▲ 100%\nChanges are traceable and audit-ready.\nChange Traceability Rate — percent of production releases linked to Azure Boards work items, pull requests, and approvals.\nQuarterly\n▲ 100%\nAI-assisted code and configuration are reviewed after use.\nHuman Review Rate on AI-Assisted Changes — percent of AI-assisted code, pipeline, or IaC changes that went through documented pull request review after merge.\nMonthly\n▲ 100%QualificationsEducation\n\n Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field, or equivalent practical experience. [Confirm with Hiring Manager — not specified in source posting]\nExperience\n\n Five (5) or more years of experience in DevOps, with a strong background in software development.\n Strong hands-on experience with Azure DevOps (Repos, Pipelines, YAML, Releases, Environments).\n Solid Node.js development experience, enough to independently read, modify, and ship production code.\n Hands-on experience refactoring application code to use a secrets manager (Azure Key Vault preferred) with Managed Identity.\n Experience with the JFrog Platform (Artifactory, Xray, and Advanced Security SAST) in CI/CD workflows.\n Experience with SonarQube for code quality and test coverage quality gates in CI/CD workflows.\n Experience with dynamic application security testing (DAST) integrated into CI/CD pipelines; Invicti preferred.\n Experience building automated testing into CI/CD pipelines, including unit testing (Jest) and end-to-end testing (Playwright or Cypress).\n Experience with application security scanning tools for dependencies, vulnerabilities, and secrets (such as JFrog Xray, Snyk, GitHub Advanced Security for Azure DevOps, or Gitleaks).\n Experience with Datadog APM and Log Management, including tracer instrumentation, log pipelines, dashboards, and monitors; experience with Azure Monitor and Application Insights.\n Experience with feature flags (Azure App Configuration or similar).\n Experience with Docker and containerized application deployment.\n Experience with Azure Boards or similar work tracking tools, including linking work items to code and releases.\n Proven experience introducing CI/CD and environment standardization to existing applications.\n Kubernetes (AKS) experience preferred.\n Experience with container image scanning and broader DevSecOps practices preferred.\n Prior experience in financial services or another regulated industry preferred.\nLicenses, Registrations, and Certifications\n\n No specific license or registration required for this role.\n Microsoft Azure certifications (AZ-400 DevOps Engineer Expert, AZ-104 Azure Administrator) preferred.\nKnowledge & Skills\n\n Working knowledge of Azure services such as App Service, Functions, AKS, virtual machines, and networking.\n Experience with Infrastructure as Code (Terraform or Bicep preferred).\n Git expertise, including branching strategies such as GitFlow or trunk-based development.\n Scripting skills in PowerShell and/or Bash.\n Experience enforcing code quality standards with ESLint and Prettier.\n Strong documentation habits, including runbooks and process documentation.\n Ability to explain technical standards clearly and train team members on new processes.\n Drive to learn and adopt new technologies, techniques, and CUTX-approved AI tools.\nCore Competencies\nCompetency\nProficiency Level\nWhy This Matters in This Role\nAI Literacy\nIntermediate\nThe role uses AI-assisted code and configuration tools (Tier 2) and must recognize when AI output is wrong or insecure, apply required controls, and ensure human review after changes are merged.\nTechnical Excellence\nAdvanced\nThe role owns the design of pipelines, environments, and release processes that every CUTX application team will depend on.\nRisk Awareness\nAdvanced\nPipeline, secrets, or access-control weaknesses can expose member data or disrupt services; the role must identify, remediate, and escalate security and change risk.\nOperational Discipline\nAdvanced\nPipelines, infrastructure, and documentation must be repeatable, version-controlled, and audit-ready to meet change management and examination expectations.\nCommunication\nIntermediate\nThe role must document standards clearly, train teams, and explain trade-offs to technical and non-technical partners.\nCollaboration\nIntermediate\nStandardizing the SDLC requires working across development, operations, security, and compliance teams to drive adoption.\nCompliance Orientation\nIntermediate\nSDLC controls must support GLBA data protection, NCUA information security requirements, and TRAIGA-aligned AI governance.AI & Technology ExpectationsAI-Augmented Workflows\nThe following workflows are AI-augmented in this role. The Senior DevOps Engineer is expected to work fluently within these workflows, exercise sound judgment over AI outputs, and follow all applicable controls.\n\n AI-assisted code generation and refactoring for Node.js applications.\n AI-assisted authoring of pipeline YAML, Infrastructure as Code templates, and scripts.\n AI-assisted triage and remediation of SAST (JFrog), DAST (Invicti), dependency (JFrog Xray), code quality (SonarQube), and secret scan findings.\n AI-assisted log, trace, alert, and incident analysis using Datadog APM, Datadog Logs, and Azure Monitor data.\n AI-assisted drafting of runbooks, wiki documentation, and training materials.\nAI Tier and Human-in-the-Loop Responsibility\nThis role operates in AI Tier 2 for its principal AI-augmented workflows (see Appendix A). The Senior DevOps Engineer retains accountability for any decision, communication, or member/employee-impacting action influenced by AI output, consistent with the CUTX Generative AI Usage Policy §3.4.\nThe Senior DevOps Engineer is required to:\n\n Review, test, and validate all AI-generated code, pipeline definitions, IaC templates, and scripts after they are merged or run against any environment.\n Route every AI-assisted change through the standard pull request, code review, and pipeline quality gates; AI output never bypasses these controls.\n Verify AI-suggested remediations for security findings against authoritative sources after applying them.\n Escalate to the Hiring Manager, IT Security, and the AI Council any use case that would let AI make changes to production systems without human approval, which is treated as Tier 3 and requires additional controls.\n Stop reliance on AI output and escalate immediately if the output appears inaccurate, insecure, non-compliant, or outside the role's documented scope (Generative AI Usage Policy §3.5).\n Refrain from entering secrets, credentials, connection strings, member non-public personal information (NPI), or confidential CUTX source code into any AI tool not explicitly approved for that data classification.\n Complete all required AI training within thirty (30) days of hire and maintain annual currency.\nApproved AI Tools\nThe role is approved to use the following AI tools in performing essential functions (subject to the Generative AI Usage Policy and any tool-specific guidance issued by the AI Council):\n\n CUTX-approved internal AI assistants (e.g., Sam) for general productivity and approved knowledge tasks.\n Microsoft Copilot for office productivity (drafting, summarization, spreadsheet support).\n CUTX-approved code-assistance tools used within sanctioned development environments and CUTX repositories.\n AI features built into CUTX-approved DevOps, security scanning, and monitoring platforms (e.g., Azure DevOps, JFrog, SonarQube, Invicti, Datadog, and Azure Monitor).\n\nUse of AI tools outside this list requires prior approval from the role's department leader and the AI Council, per the Generative AI Usage Policy §4.Prohibited AI Use\nIn addition to the prohibited uses defined in the Generative AI Usage Policy §3.6, the following are specifically prohibited in this role:\n\n Merging or deploying AI-generated code, configuration, or infrastructure changes without documented human review and passing pipeline quality gates.\n Entering secrets, credentials, keys, connection strings, member NPI, or confidential CUTX source code into any AI tool not explicitly approved for that data classification.\n Granting AI agents or tools autonomous write access to production environments, pipelines, or secrets stores without governance review and required approvals.\n Using unsanctioned third-party AI services for CUTX code generation, infrastructure changes, or log and data analysis.\nCompliance & Regulatory ResponsibilitiesEn