Data Engineer

Job Title: Data Engineer

Location: Charlotte, NC (Hybrid)

Type: Contract

Job Description:

• This role involves working with cross-functional teams to develop, manage, and optimize cloud-based data solutions for cybersecurity analytics.
• The ideal candidate will have experience in AWS Glue, Apache Kafka, Terraform, Data Warehousing, Data Lakes, Snowflake, and API development to ensure secure, scalable, and efficient data processing and storage.

Responsibilities:

• Develop and maintain AWS Glue ETL pipelines to process and transform cybersecurity-related data.
• Design, implement, and optimize Data Warehouses and Data Lakes to support security analytics and reporting.
• Manage Terraform-based Infrastructure-as-Code (IaC) deployments.
• Work with Snowflake to develop and optimize secure data storage, querying, and performance tuning for large datasets.
• Create and manage API calls to ingest, transform, and exchange data between front-end applications and backend data systems.
• Assist in data governance, security controls, and compliance within cloud-based data platforms.
• Monitor production data jobs, troubleshoot failures, and execute batch processing efficiently.
• Participate in code reviews, security audits, and architecture discussions to improve data workflows.

Job Requirements:

• 5+ years of experience in cloud-based data engineering and big data processing.
• Hands-on experience with AWS Glue for ETL development and job orchestration.
• Expertise in Terraform for cloud infrastructure automation and deployment.
• Strong background in Data Warehousing and Data Lake architectures (AWS S3, Snowflake
• , etc.).
• Proficiency in Snowflake for data storage, modelling, and query performance optimization.
• Experience in creating and managing API calls for data ingestion from front-end applications and external systems.
• Experience in creating SSIS packages/ Debugging the SSIS Packages.
• Knowledge of RESTful APIs, JSON, and authentication mechanisms (OAuth, API keys, etc.).
• Strong skills in SQL, Python, and Spark for data transformations and analysis.
• Hands on experience using Apache Kafka in ETL process
• Experience with data security best practices, encryption, and access control in cloud environments