Spark Python AWS Developer

Spark Python AWS DeveloperPerforming ETL jobs in Batch Modes. Performing ETL using Real-Time Spark streaming. PythonScala programming intermediate level. Hands on experience in Spark version 1.6 and 2. Working with different file formats Hive, Parquet, CSV, JSON, ORC, Avro etc. Compression techniques. Integrating PySpark with different data sources, example oracle, postgres, mysql, MS sqlserver etc. SparkSQL, DataFrames Datasets. Performance Tuning techniques.Good to have PySpark Basic Client techniques in spark. optional for Data Engineering Working with Hive, No Sql Databases like Hbase, Cassandra etcWorking in importing and exporting data files from AWS clustersS3-simple storage systems. Mounting clusters. Managing cluster configurations. Amazon EMR for BigData Processing using Hadoop, spark, hbase etc Real Time analytics using amazon kinesis.Basic navigational flow, working with Notebooks, scheduling, integrating with AWS for cloud storage.Parallel processing scenarios. Branching. Sub Dags. Trigger Rules.Working with different file formats and compression techniques. Managed and External tables. Partitioning and Bucketing concepts. Integration with Sqoop and Hbase or No Sql Dbs. Performance tuning techniques. Different parameters and their usages. Incremental Bulk Snapshot loading scenarios.