Senior Data Engineer

Software Engineer**Active TS/SCI w/Polygraph REQUIRED**. Please do not apply if you do not currently possess this level of clearance.Basic Requirements:7 years of experience as a software engineerBachelor's degree in a technical discipline 2 additional years of experience as a software engineer may be substituted for a degreeJob Description:The Software Engineer shall possess extensive expertise in dataflow design, data transport mechanisms, and Apache Spark based distributed processing.In this role, the Software Engineer shall be responsible for designing, implementing, and optimizing data ingress/egress pathways to ensure efficient, scalable, and reliable processing of the organization's analytics workloads.Required Experience:Experience using the Linux CLI and Linux toolsExperience developing Bash scripts to automate manual processesRecent software development experience using Python and JavaExperience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflowsExperience with Distributed Big Data processing engines including Apache SparkFamiliar with SQL technologies such as MySQL, MariaDB, and PostgreSQL for querying, joining, and aggregating large datasetsExperience using Jupyter NotebookExperience with data wrangling and preprocessing using tools such as pandas, NumPyExperience working with structured, semi-structured, and unstructured data such as Parquet, JSON, CSV, XMLFamiliarity with data quality concepts, data validation, and anomaly detectionExperience with Git Source Control SystemDesired Experience:Familiar with HPC Job Scheduling tools including SlurmExperience using the Atlassian Tool Suite (JIRA, Confluence)Salary: $230,000 - $240,000 annually. Excellent benefits package including 25 days PTO, 11 paid holidays, 100% employer-paid healthcare for employees and dependents – available day 1, 8% 401(k) employer match – immediate vesting. Disclaimer: The salary range provided is an estimate based on current market conditions and may be adjusted based on factors such as experience, skills, and qualifications. The final salary offer will be determined after a thorough review of the candidate's background and alignment with the role. Please note that this range is subject to change and should be considered as a guideline rather than a definitive figure. Shield Consulting Solutions is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law.