Find Jobs
Hire Freelancers

Data Scraping Project -- Several Million Records

$100-325 USD

Cancelled
Posted almost 12 years ago

$100-325 USD

Paid on delivery
I have a mid/large size data scraping project. It involves scraping roughly 3 million web pages from a large public site. About 400,000 of these pages will require following a link on the page and scraping a second page that is related to the first page. The site lists the pages in sequential order (ie [login to view URL] then [login to view URL] etc.) so it is very easy to get the pages, there is no need to enter a search term. This should be a pretty simple project. You will be required to use your own server and deliver the data, not the scraping code. If you have experience scraping, this should be easy, if not please do not reply to this post. After receiving your intial bid and looking at your background I will share more information. The data will require some sorting and will be delivered in excel and mysql as well as potentially transferred into a server. Also, to make sure the scraping does not violate the sites TOS, you need to understand how to slow down scraping so it does not use excessive bandwidth.
Project ID: 2732063

About the project

Remote project
Active 12 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

About the client

Flag of UNITED STATES
United States
0.0
0
Payment method verified
Member since Apr 15, 2012

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.