Lead Data Engineer

TECHNOGEN, Inc. is a Proven Leader in providing full IT Services, Software Development and Solutions for 15 years. TECHNOGEN is a Small & Woman Owned Minority Business with GSA Advantage Certification. We have offices in VA; MD & Offshore development centers in India. We have successfully executed 100+ projects for clients ranging from small business and non-profits to Fortune 50 companies and federal, state and local agencies. Hi, Greetings of the day! We are looking to Hire a Talented Professional for the below Job opportunity with one of our clients, If you're interested, please share your updated resume at your earliest convenience, and I'll be happy to provide more details about the role. Position: Senior Technology Consultant - Data / AI Native Engineer Location: Arlington, VA || New York, NY || St. Louis, MO (Hybrid work model | Local candidates preferred) Duration: Long Term Contract Job Description: We are seeking a hands-on Senior Data / AI Native Engineer with strong expertise in Databricks, AWS, PySpark, and modern Lakehouse architecture. The ideal candidate combines deep Data Engineering expertise with strong Software Engineering practices and has experience leading complex enterprise data initiatives. The role requires daily hands-on use of GitHub Copilot and/or Claude Code CLI to accelerate software and data pipeline development. The candidate should be comfortable rapidly building POCs and MVPs and applying AI directly within data engineering workflows-not just using AI for application development. Day to Day Job Duties Design, develop, and optimize high-volume enterprise data pipelines using PySpark, Spark, and Databricks. Build scalable Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze/Silver/Gold). Implement data governance, access controls, and cataloging using Databricks Unity Catalog. Design and develop cloud-native data solutions using AWS S3, EMR, Glue, Lambda, and Redshift. Lead complex Data Engineering initiatives and provide technical guidance to engineering teams. Develop batch and real-time data processing pipelines for large-scale enterprise datasets. Use GitHub Copilot and/or Claude Code CLI daily to accelerate coding, pipeline development, testing, troubleshooting, and documentation. Rapidly develop POCs and MVPs using AI-assisted engineering practices. Apply AI within data pipelines for use cases such as schema inference, automated data-quality rule generation, anomaly detection, and PySpark transformation generation. Build AI-ready data platforms supporting RAG, embeddings, vector stores, and AI/ML applications. Develop streaming pipelines using Structured Streaming, Kafka, Kinesis, or Databricks Auto Loader. Implement modern data engineering patterns including CDC, SCD Type 2, schema evolution, idempotent processing, and data contracts. Implement data quality, validation, monitoring, lineage, and observability across data pipelines. Conduct code/design reviews and establish reusable Data Engineering patterns and standards. Collaborate with Data Architects, AI/ML Engineers, Software Engineers, and business stakeholders to deliver enterprise data products. Basic Qualifications Must Have: 7+ years of experience in Data Engineering and Software Engineering, building production-grade enterprise data solutions. 4+ years of hands-on experience with Databricks, Delta Lake, Medallion Architecture, and PySpark/Spark. 4+ years of experience with the AWS data ecosystem, including S3, EMR, Glue, Lambda, and/or Redshift. Proven experience leading complex Data Engineering initiatives or providing technical leadership to Data Engineering teams. Strong hands-on experience processing high-volume datasets using PySpark, rather than Scala-only Spark development. Demonstrated daily use of GitHub Copilot and/or Claude Code CLI, with the ability to explain specific examples of how these tools improve Data Engineering productivity. Proven experience rapidly developing POCs and MVPs using AI-assisted development practices. Technical Skills Data Platform: Databricks, Delta Lake, Unity Catalog Data Processing: PySpark, Apache Spark Cloud: AWS S3, EMR, Glue, Lambda, Redshift Architecture: Lakehouse, Medallion Architecture, Data Lakes Streaming: Spark Structured Streaming, Kafka, Kinesis, Auto Loader

For applications and inquiries, contact:View email address on us.fitly.work