I want a PoC to be done for the below requirement.
Source data files(in CSV format and SQL extract format) and a schema files (in JSON format) will be placed in GCS. Based on the schema definition rules given in the json file, data in CSV file should be loaded to Kafka and using Kafka streaming, data needs to be transformed to 3NF form and loaded to GCS . The main objective is,When schema changes , dynamically the code should absorb the changes in kafka without modification in the code.
Skillsets: Big Data-Kafka,Spark-Scala,GCS
5 freelancers are bidding on average £198 for this job
Hello, I am working in Bigdata/Hadoop technologies for years and have experiences working in latest Spark,Kafka, Cassandra, Hive/HBase, ELK stacks using ava, Scala, python. Can we talk? Thank you!