Embedded OS Engineer

Description

Position Summary

Teal Drones is looking for a hands-on Embedded OS Engineer to own the Ubuntu-based Linux firmware stack on our MK2 drone platform (Qualcomm QRB5165). You will be the person who builds the OS, keeps it secure, makes it reliable, and ships it through automated CI/CD pipelines.

The questions you’ll be asked every day are ones like: Why did that service fail on boot? How do we strip this unnecessary network daemon out of the image without breaking the build graph? What’s the fastest way to test a rootfs post-process change without waiting three hours for a full Yocto rebuild?

This is not a role for someone who has only worked at the application layer. We need someone comfortable navigating low-level OS internals, Yocto’s BitBake machinery, systemd unit files, and the realities of headless embedded hardware with no display and limited recovery options.

Essential Duties and Responsibilities

Yocto / BitBake Build System

• Own and maintain the teal-mk2-build repository, including layer configuration, local.conf tuning, and image recipes that produce the qti-ubuntu-robotics-image.

• Write and maintain .bb recipes and .bbappend overlays; create .patch files (git diff–based) against the pinned Qualcomm QRB5165 BSP (LU.UM.3.3.1) so that upstream sources are never modified directly.

• Debug BitBake task failures—understanding the do_fetch → do_compile → do_rootfs → do_image pipeline, sstate-cache behavior, and when to use -c cleansstate vs. -c rootfs -f.

• Optimize build performance: reduce full-build times by understanding QEMU emulation overhead vs. native ARM64 compilation, tuning BB_NUMBER_THREADS and PARALLEL_MAKE, and leveraging shared sstate-cache across Jenkins agents.

• Manage package inclusion, exclusion (IMAGE_INSTALL:remove, PACKAGE_EXCLUDE, BAD_RECOMMENDATIONS), and inter-recipe dependency graphs.

Rootfs Post-Processing & OS Hardening

• Implement ROOTFS_POSTPROCESS_COMMAND functions to perform post-build OS customization: package upgrades, apt security patching (Ubuntu ESM), service installation, and file system fixups.

• Remove unnecessary services from production images—including servers (lig TFTP daemons, and other attack-surface-expanding services identified in Blue List / Nessus security assessments.

• Write bash scripts and systemd service units that execute reliably in a headless, rootfs post-install context (no interactive terminal, limited /proc and /dev availability).

• Manage the dpkg / apt ecosystem within the embedded rootfs: pinning packages, handling held packages, validating package state, and ensuring apt lock files are clean across incremental builds.

Jenkins CI/CD Pipelines

• Maintain and improve Jenkins pipelines that build Yocto firmware images, including handling concurrent build isolation, artifact staging, and race conditions between parallel jobs.

• Diagnose and resolve Jenkins agent performance issues: Java heap tuning, Docker overlay2 disk pressure, container lifecycle management, and build environment reproducibility.

• Implement proper artifact copy patterns (stageDir isolation) to prevent race conditions when multiple builds run concurrently against shared directories.

• Manage GitLab repository structure and branch protection for the build repo; implement .patch-based change workflows that preserve the integrity of pinned upstream BSP branches.

Embedded Linux System Configuration

• Configure systemd-networkd for runtime network mode switching (DHCP ↔ link-local) on headless devices using udev rules, button-press event handlers, and LED feedback mechanisms.

• Write and maintain systemd service units and timers: understanding WantedBy, After, Requires, and ExecStart semantics for embedded boot sequences.

• Implement reliable USB logging and file transfer services (usb-gadget, udevadm) for field diagnostics on devices with no screen.

• Develop and maintain Prometheus node_exporter integrations and process-exporter configurations for drone fleet health monitoring via Grafana dashboards.

Security & Compliance

• Remediate Nessus / Blue List security findings affecting the embedded OS: SSH hardening (key-only auth, sshd_config tuning), open port reduction, and service inventory documentation.

• Maintain /etc/shadow hygiene, locked account policies, and PAM configurations appropriate for production embedded devices.

• Generate and maintain security compliance artifacts (port inventories, service lists, patch status reports) for internal security assessors.

Reliability & Performance Tuning

• Profile and diagnose slow boot sequences, runaway processes, and memory pressure on ARM64 embedded hardware.

• Use screen, tmux, and remote shell tooling to manage long-running build and deployment sessions on headless servers and devices.

• Implement OS-level monitoring: log capture services, boot-time diagnostics, and watchdog patterns for unattended field deployment.

• Advise on cloud build infrastructure choices (AWS Graviton / Azure ARM64) to eliminate QEMU emulation overhead and achieve 3–5× build time reductions.

Required Qualifications

• Bachelor's or master's degree in Computer Science, Computer Engineering, or a related field.

• 5+ years of embedded Linux engineering experience, including direct ownership of Yocto/BitBake build systems (or similar) in a production environment.

• Strong proficiency with BitBake concepts: recipes (.bb), appends (.bbappend), image recipes, ROOTFS_POSTPROCESS_COMMAND, sstate-cache, and task dependency graphs.

• Hands-on experience writing and applying .patch files against BSP/upstream sources to manage downstream customizations without modifying pinned branches.

• Deep knowledge of systemd: service units, network configuration (systemd-networkd, .network files), udev rules, and boot dependency ordering.

• Proficiency in bash scripting for OS-level automation: apt/dpkg management, file system operations, LED/GPIO control, and headless device configuration.

• Experience with Ubuntu on ARM64 platforms, including ESM security patching, package management, and kernel/BSP integration.

• Familiarity with Jenkins or equivalent CI/CD platforms: pipeline scripting, agent management, Docker-in-Docker build environments, and artifact handling.

• Experienced with SSH-based workflows, git patch workflows, and working entirely in headless terminal environments.

Additional Desired Qualifications

• Experience with Qualcomm QRB5165 or similar robotics/drone SoC platforms and their associated BSP layers.

• Experience configuring Prometheus, node exporter, process-exporter, and Grafana for embedded device fleet monitoring.

• Familiarity with Python Flask for internal tooling (device registration, fleet management dashboards).

• Knowledge of cross-compilation tools and methods.

• Experience with ADB (Android Debug Bridge) workflows for Qualcomm-based embedded targets.

• Exposure to radio licensing and MAC-based device authentication workflows (e.g., Doodle Labs radios).

• Hands-on experience in AWS Cloud management and infrastructure provisioning.

Physical Requirements and Working Conditions

• Must be able to walk, stand, and navigate large indoor and outdoor facilities for extended periods of time.

• Ability to lift, carry, and move materials and equipment weighing up to 25 lbs on a regular basis.

• Use of personal protective equipment (PPE) may be required in designated areas or when performing specific tasks, in accordance with safety protocols and company policy.

• May be required to climb ladders, stoop, kneel, or crouch during inspections, maintenance walk-throughs, or emergency response situations.

• Regular exposure to facility operations including noise, dust, temperature fluctuations, and industrial equipment.

• Occasional off-hours or weekend work required for emergency facility responses or projects as needed

• Requires frequent use of a computer and other standard office equipment for documentation, communication, and coordination tasks.

Background Check

This position will require successfully completing a post-offer background check. Qualified candidates with a criminal history will be considered and are not automatically disqualified, consistent with federal and state law.

EEO and ITAR/EAR Work Authorization Disclosure

Red Cat Holdings provides equal employment opportunities (EEO) to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. This position requires direct or indirect access to hardware, software, technology or technical data controlled under the International Traffic in Arms Regulations (ITAR) and the Export Administration Regulations (EAR). Successful candidates for positions subject to ITAR/EAR restrictions must provide proof of U.S. Citizenship or Permanent Residence and must not require sponsorship for export-restricted work authorization.

E-Verify

The company participates E-Verify ( ensure eligibility for employment and compliance with Right to Work ( rules.

Compensation: Salary plus generous annual equity package and potential bonuses.