Data Engineer

Position Summary Amerit Fleet Solutions seeks an experienced Data Engineer to design, build, and maintain a robust data lake infrastructure that will serve as the foundation for enterprise analytics, reporting, and AI/ML initiatives. This role is critical to establishing data governance, quality, and accessibility across the organization. The successful candidate will work cross-functionally with IT, Analytics, Operations, and AI teams to ensure data integrity, compliance, and optimal performance of our data infrastructure. Compensation & Benefits Salary: $150k - $170k per annum Comprehensive health insurance 401(k) matching professional development budget flexible work arrangements Essential Duties & Responsibilities Data Lake Architecture & Design: Design and implement a scalable, cloud-based data lake architecture (AWS/Azure/GCP) that ingests, stores, and manages petabyte-scale data from fleet management systems, maintenance records, vendor systems, and operational databases. Establish data zones (raw, curated, analytics) with appropriate access controls and retention policies. Data Integration & ETL/ELT Pipelines: Build and maintain automated data pipelines that extract, transform, and load data from multiple sources (work order systems, telematics platforms, financial systems, CRM) into the data lake. Ensure real-time and batch processing capabilities with minimal latency. Document all transformations and business logic. Data Quality & Integrity Management: Establish and implement comprehensive data quality frameworks including validation rules, anomaly detection, and reconciliation processes. Monitor data accuracy, completeness, and consistency. Create data quality dashboards and alerts to identify and remediate data issues before they impact downstream analytics. Maintain detailed audit trails for all data changes. Data Governance & Compliance: Develop and enforce data governance policies including data cataloging, metadata management, lineage tracking, and PII/sensitive data protection. Ensure compliance with data privacy regulations (GDPR, CCPA, etc.). Establish data access controls, role-based permissions, and audit logging. Maintain data dictionary and documentation standards. Performance Optimization & Monitoring: Monitor data lake performance, query execution times, and storage utilization. Optimize data structures, indexing, and partitioning strategies to ensure sub-second query response times. Implement automated scaling policies and cost optimization initiatives. Provide recommendations for infrastructure improvements. Data Security & Disaster Recovery: Implement encryption, secure data access protocols, and disaster recovery/business continuity plans. Establish backup, replication, and recovery procedures with defined RPO/RTO targets. Conduct security audits and vulnerability assessments. Maintain compliance documentation for SOC 2 and other security standards. Documentation & Knowledge Transfer: Create comprehensive technical documentation for data lake architecture, data flows, transformation logic, and operational procedures. Develop runbooks for common operations and troubleshooting. Provide training to analysts, data scientists, and other teams on data access, usage best practices, and available datasets. Cross-Functional Collaboration: Partner with business units to understand data requirements and use cases. Collaborate with AI/ML teams on model training data pipelines. Work with analytics teams to optimize queries and reporting. Support data strategy discussions and roadmap planning. Key Competencies and Minimum Education Required Skills & Qualifications Technical Skills Advanced SQL and relational database design (PostgreSQL, MySQL, or SQL Server) Cloud data platforms (AWS S3, Glue, Redshift, Azure Data Lake Gen2, Synapse, Databricks, Microsoft Fabric) ETL/ELT tools (Apache Airflow, dbt, Talend, or cloud-native alternatives) Data warehousing concepts and dimensional modeling (star schema, slowly changing dimensions) Programming languages: Python or Scala for data pipeline development Data quality frameworks and tools (Great Expectations, Talend, or similar) Version control (Git) and CI/CD practices Experience 5+ years of professional experience as a Data Engineer, with 2+ years building data lakes or enterprise data platforms Demonstrated experience with large-scale data infrastructure projects (100GB+ datasets) Experience implementing data governance, metadata management, and data catalogs Track record of designing systems with high availability (99.9%+ uptime) Knowledge & Certifications Strong understanding of data architecture patterns, normalization, and schema design Knowledge of data security, encryption, and compliance frameworks AWS Certified Data Analytics OR Azure Data Engineer or equivalent certification (preferred) Understanding of database performance tuning and query optimization Soft Skills Excellent written and verbal communication skills Ability to explain complex technical concepts to non-technical stakeholders Strong problem-solving and debugging skills Proactive approach to identifying and resolving data issues Ability to work independently and in cross-functional teams Preferred Qualifications Experience in transportation, logistics, or fleet management industries Familiarity with telematics and IoT data processing Experience with data lake solutions (Azure Data Lake Gen2, Microsoft Fabric) Knowledge of streaming data technologies (Kafka, Kinesis, Pub/Sub) Experience with Tableau, Power BI, or other analytics platforms Master's degree in Computer Science, Data Science, or related field Open-source data project contributions or personal data projects Experience with Claude Code or similar AI tools. Success Metrics (First Year) Data lake MVP deployed with ingestion from 5+ data sources Data quality framework implemented with 95%+ data accuracy threshold Automated monitoring and alerting system for data pipeline failures Documented data governance policies and implemented access controls Team training completed on data lake access and best practices Audit trails and compliance reporting automated 5%+ data pipeline uptime achieved Query performance optimized to #J-18808-Ljbffr