Cancelled

Web-scraping a big dataset

The task is simple. You have to go to this website [1] and download the publicly available data. Manually it would take forever. That’s why you will have to study the JavaScript that creates the requests. You are expected to write a script which makes POST requests and batch downloads the data.

Details:

Currently you can get 24H data for all stations. Using a script create requests for all days one by one (without overwhelming the server). Requests are processed and then the resulting data is dumped on an ftp server [2] with the respective request number. 24H data for all stations should be about 1.5 GB of data. Download them and put them on a server where we can access them in bulk.

Download procedure to be automated:

1) Enter an e-mail address

2) Select data format: Mini-Seed

3) Select time interval: Loop over A single day from 1/1/2009 to 16/10/2016

4)Select stations: Select all stations and all channels

5) Submit request

6) Wait and collect the completed request from [2]

[1] [url removed, login to view]

[2] ftp://[url removed, login to view]

Skills: AJAX, Big Data, Data Processing, Javascript, Web Scraping

See more: big woman web, scrolling big image web

About the Employer:
( 0 reviews ) Zurich, Switzerland

Project ID: #11804239

21 freelancers are bidding on average €168 for this job

e3d

Hi, I can download all of those. where you want me to put downloaded data and in which format ?

€199 EUR in 3 days
(143 Reviews)
8.1
seaanddream

Hi, my name is Sevinc. I read your "Web-scraping a big dataset" project descriptions carefully before bidding. I checked the urls, and your requirements list as well... I got what you need and ready to go ahead as so More

€250 EUR in 5 days
(199 Reviews)
7.2
mantislin

Dear sir, I am scraping expert, I have did too many scraping projects, please check my reviews then you will know. Can you tell me more details? then I will provide example data/script for you. Thanks, More

€250 EUR in 5 days
(173 Reviews)
6.9
mirniyazuddin92

Dear Sir/Ma'am, I am a Web research, Data Entry & Webs Scrapping expert. I checked and understood your requirements. I can handle this job very well to your appreciation. I can find and extract the information More

€200 EUR in 5 days
(87 Reviews)
6.1
mananraja

Hi there, I have read the project description.. I can write a web scraping script to get this data from the website. Let me know & we can discuss details.. Thanks..

€100 EUR in 1 day
(77 Reviews)
5.8
techwelf

yes, let's clarify the requirement in details also if you want us to share our skill and previous work let us know.. hope to hear you soon.. Thanks & Regards RIMSHA

€257 EUR in 15 days
(15 Reviews)
4.6
bpestilence

Hello, I'm professional scraper with 10 years experience in PHP & Ruby. I will be glad to apply my skills :)

€277 EUR in 3 days
(3 Reviews)
4.3
lillysoft

Sir i am really interested in your project . sir i am an expert of software and web development. i have already developed many web and windows applications and some are similar to your project . you can check my port More

€184 EUR in 3 days
(3 Reviews)
3.8
redpassion

Hello, How are you. I'm a experter for web scraping, web bot, web crawler, spider with python, php, C#. I've just read your job posting and I'm very interested. I completed ADIDAS web crawler project that get link UR More

€155 EUR in 3 days
(8 Reviews)
3.6
pthreeinfotech

Hi...... I am Pritesh and I represent Pthreeinfotech. I can complete your project on time and within your budget . I have worked on similar projects to what you are looking for, and I am confident I can exceed your e More

€100 EUR in 3 days
(5 Reviews)
3.1
pradeep1205

Hello sir/madam, good day, i read your job description fully... I have worked on similar projects to what you are looking for, and I am confident that I can exceed your expectations. I can complete your project on time More

€30 EUR in 3 days
(3 Reviews)
3.0
€155 EUR in 3 days
(7 Reviews)
2.9
evrimdemir0

Hi, I can complete this task by using Selenium Web Automation in C# language. It is not going to take more than 2 days since I have been doing web scrapping in my job on daily basis. Please let me know if you are i More

€73 EUR in 2 days
(1 Review)
1.5
ahmedjaved94

Hi there! My full-time job is webscraping and I'm available for work. I understand that you want data to be fetched from the websites and dumped in to a server. My question is how do we get a request number? Is More

€166 EUR in 3 days
(1 Review)
0.7
sounbumsong

Price Negotiable! python 3 years+ developer have experience with web scrapping/ web crawling for collecting malware and information and downloading files have experience with web scrapping/ web crawling for co More

€111 EUR in 3 days
(0 Reviews)
0.0
tha192004

I believe/trust that you have come across many others who are equipped for the job in terms of capabilities and abilities; however, going through my past experience and work performance you will find out that I am one More

€155 EUR in 3 days
(0 Reviews)
0.0
€111 EUR in 5 days
(0 Reviews)
0.0
ellyf

A proposal has not yet been provided

€111 EUR in 3 days
(0 Reviews)
0.0
degee147

Hi, I'm Ken Waribo. Let me introduce me as a hard working and passionate Developer who loves to achieve beautiful looking end product of targets set forth. I read through your project description and can see you need More

€277 EUR in 10 days
(0 Reviews)
0.0
woonhow

I'm web programmer writing java scripts or tools download content from web site is kind of easy to me.

€277 EUR in 5 days
(0 Reviews)
0.0