Find Jobs
Hire Freelancers

Develop website scraping code - edited & reposted

$200-1000 USD

Completed
Posted over 10 years ago

$200-1000 USD

Paid on delivery
I need to develop code that can be placed on a server to scrape information from a website. The website is a public site which posts energy prices. Specifically, it is www.newyorkpowertochoose.com. I need all utility provider results and other fields to be scrapped from the website in regular intervals (e.g. every 6 hours, every day) and placed into a Microsoft SQL Server database. The website requests specific zip codes to provide information on pricing. I will provide approximately 16 target zipcodes. Each scrub would take all information for all specified zip codes at the interval specified. The data sought form the site includes: - Service Type (Electricity or Gas)* - Zip Code* - Utility Name* - Supplier Name - Rate Type (Fixed or Variable) - Rate (Numeric Only) - Green Offer (Yes/No) - Minimum Term (Months Only) - Cancellation Fee (Numeric Only) - Cancellation Fee Text (used only when qualifying text is present) - Comments (Text) - Date/Time Stamp of Scrub Lastly, I would also like to have an image of the page scrubbed saved in an image file. I will provide the Microsoft SQL Server database IP address and login credentials, a directory to save the page images, and the zip codes to be used for each scrub (you can use 10003 as a test zip code). I need a way to adjust the scrub increment (e.g. days / every 6 hours) and the zip codes in the future if necessary. This adjusting mechanism doesn’t need to be sophisticated it only needs to be able to be done quickly and efficiently. To be specific, I am reasonably tech savvy and could go into the code and adjust it if it’s only located in one or two places though a database driven interval and zip code table might work best. To help ensure we remain aligned throughout the development process, I propose the following gates, but I am open to your input: Milestone 1: Developer Review Target Site & Verify Feasibility of Scrubs (each service type, each zip code, at specified intervals) Milestone 2: Agree on Target Database Structure Milestone 3: Test Single Scrub (1 zip code) with input sent to target Microsoft SQL Server database & page image sent to specified directory Milestone 4: Full Scrub Code Finalized The budget is moderately flexible. I am willing to pay a reasonable price to get it done right, in a timely manner, and by a reliable coder with whom I could do business with in the future but without overpaying. I am estimating this project to be in the "Small" category ($200-$1000 USD) but please indicate if you feel reasonable compensation should be higher or lower. I will consider all reasonable bids.
Project ID: 4940594

About the project

33 proposals
Remote project
Active 11 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Ready to start the job
$200 USD in 3 days
5.0 (24 reviews)
4.9
4.9
33 freelancers are bidding on average $624 USD for this job
User Avatar
Hi, I am scraping expert, More details in PM.
$738 USD in 6 days
5.0 (249 reviews)
7.3
7.3
User Avatar
hi, i am expert in web scraping and code making, let me do this project and i assure you that code will be 100% perfect, accurate and according to your requirement, thanks
$631 USD in 3 days
4.9 (218 reviews)
6.9
6.9
User Avatar
Hi.. Expert web scraper/Data Minor here. Interested in your project. I assure you 100% accurate and good quality work. Ready to start. Regards
$631 USD in 3 days
4.9 (147 reviews)
6.8
6.8
User Avatar
Hi, Please check message
$333 USD in 3 days
5.0 (8 reviews)
5.8
5.8
User Avatar
Hi, Ready to start your work. Eagerly awaiting for your positive reply. Please check your inbox for further details. Thanks, Shaik.
$200 USD in 5 days
5.0 (44 reviews)
5.3
5.3
User Avatar
Check your PMB for details!!!
$809 USD in 22 days
4.9 (3 reviews)
5.6
5.6
User Avatar
Hello, I can deliver the scraper you need within two weeks after you accept my bid.
$936 USD in 12 days
5.0 (4 reviews)
5.3
5.3
User Avatar
Hi, I already finished few such projects.
$300 USD in 3 days
4.7 (10 reviews)
4.3
4.3
User Avatar
Please see PMB for detail and our expertise.
$333 USD in 3 days
4.9 (10 reviews)
4.3
4.3
User Avatar
Hi, I've built a number of similar web scraping applications before, built either as Windows scheduled tasks or fully-autonomous Window services, so they can run in the background on a regular schedule without any user input. I've checked the target website and the example zip code you've provided, and confirmed its suitability for web scraping. All my code is fully commented and compliant with Microsofts Stylecop design and development guidelines.
$722 USD in 7 days
5.0 (4 reviews)
4.2
4.2
User Avatar
Hello. I can make it for you in shortest time. I've made professional tool for Web Crawling and parsing in Python (executable version also - 32 and 64 bits) 1. Multi Threading. (parallel execution) 2. Very Simple Crawl rules language. (Based on XPath and CSS. I'll send you examples in private message). 3. Automatically store crawl results into any database system or other data format to filesystem (html,csv,xml). (Oracle, MySQL, MSSQL, MSACCESS, DB2, PostGres, FireBird, etc) There is some custom modules (from opensource projects) and support for python sqlalchemy. Also it will create database structure with all tables if they do not exists. 4. Proxies. SSL. (with custom user-agent header, cookies, auto-redirect etc). 5. Any encodings support (with auto detecting encoding of website and content). 6. Cache support (for web content with auto refresh by expiration time) 7. Auto-login on web-site (automatically find and fill the login form). + Captcha detection + Allow to enter captcha : With user dialog or Web site. Send captcha to website (post image via url post) and waiting for answer (so anyone with access rights can enter it). All settings stored in parameters file. Finished projects (using PyWebCrawler) : booking.com. All Countries and Hotels information+images. market.yandex.ru. All categories, goods, their descriptions and pictures. and many others. My profile : http://www.freelancer.com/u/kamosion.html.
$666 USD in 3 days
5.0 (3 reviews)
3.7
3.7
User Avatar
Hi, We're ready to take up your project. Kindly check PMB for more details. Awaiting your kind response.
$618 USD in 15 days
5.0 (5 reviews)
3.3
3.3
User Avatar
Please check my PM
$700 USD in 7 days
5.0 (3 reviews)
3.2
3.2
User Avatar
Let me start the project ,Please do consider us.
$1,000 USD in 15 days
5.0 (1 review)
3.0
3.0
User Avatar
Hello i have done many scrapings before. Please see my PM. Thank you.
$666 USD in 3 days
5.0 (1 review)
2.8
2.8
User Avatar
I can do that professional
$333 USD in 3 days
5.0 (3 reviews)
2.7
2.7
User Avatar
ready to start work. please check PM.
$444 USD in 10 days
5.0 (2 reviews)
2.8
2.8
User Avatar
hi sir, i'm ready let's start
$277 USD in 3 days
5.0 (2 reviews)
2.8
2.8
User Avatar
Please check PMB for reference..
$473 USD in 10 days
5.0 (1 review)
2.5
2.5
User Avatar
Ready to start now and deliver within week
$666 USD in 7 days
5.0 (1 review)
1.4
1.4

About the client

Flag of UNITED STATES
New York, United States
5.0
1
Payment method verified
Member since Sep 18, 2013

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.