Closed

Loading data to GCS using Kafka

I want a PoC to be done for the below requirement.

Source data files(in CSV format and SQL extract format) and a schema files (in JSON format) will be placed in GCS. Based on the schema definition rules given in the json file, data in CSV file should be loaded to Kafka and using Kafka streaming, data needs to be transformed to 3NF form and loaded to GCS . The main objective is,When schema changes , dynamically the code should absorb the changes in kafka without modification in the code.

Skillsets: Big Data-Kafka,Spark-Scala,GCS

Skills: Data Analytics, Google Cloud Storage, Scala, Spark

See more: kafka connect mysql, kafka connect, apache kafka, kafka to s3, bigquery to kafka, kafka vendors, kafka data flow, kafka connector, insert data xml using vbnet, data entry using spss, data mining using aspnet, data extraction using regex, serial wireless data transmission using at89c51 chip, example using data mysql using javascript php, data migration using php, data excel using cnet, fetch data site using curl php, extracting data html using, scrap data website using aspnet, data mysql using ajax

About the Employer:
( 0 reviews ) United Kingdom

Project ID: #17599939

5 freelancers are bidding on average £198 for this job

deytps86

Hello, I am working in Bigdata/Hadoop technologies for years and have experiences working in latest Spark,Kafka, Cassandra, Hive/HBase, ELK stacks using ava, Scala, python. Can we talk? Thank you!

£220 GBP in 3 days
(5 Reviews)
3.1
amitkumar0327

Hi, I am Amit. I have experience in Spark, Scala, Kafka and GCP. I will be able to do the POC based on the requirement. I will be interested to know more about the Schema files that will be placed in GCS. The Schema fi More

£150 GBP in 3 days
(2 Reviews)
2.7
gunjanchoudhary9

Key Skill set and expertise in several Big Data projects : Spark , Kafka, Spark SQL, GCP, RDBMS, Scala Proposed Solution:Using Spark and Scala for importing the schema json file for creating dynamic schema . Check th More

£177 GBP in 5 days
(0 Reviews)
0.0
£277 GBP in 5 days
(0 Reviews)
0.0
£166 GBP in 4 days
(0 Reviews)
0.0