Closed

Scraper code - possibly OutWit.

Website Scraper Code – preferably using OutWit-Hub (since I have a copy – but more suitable package would probably be OK).

The target site is a real estate website which provides DB of agents across the US which can be searched by location (town, suburb). I am interested in the selling/listing agents (the majority). Doing it manually, the agent list for the target location needs each agent name to be clicked one at a time. This then loads a second page for the particular agent which contains wanted fields for agent name and agency/brokerage name -2 of the 3 target fields. Also in the side panel box containing the agency name is a “Website” link (for most records probably 70-80% of them. This needs to be clicked/opened in order to find the 3rd required field for the particular record: an email address for the agent. The “Website” pages need to be scanned for the email address, which typically may be in the header-banner, or anywhere in the body of the page. Or often in other pages of this 3rd-level site, such as “About Us” page or other. The “Contact Us” page usually opens up a contact form and does not contain an email address, although some do.

Most agent records use a form for visitor communication with the agent, no doubt because the site operator gets some sort of fee or commission for introducing a prospective property seller to the agent. Also because most agents are lazy or stupid and trying to avoid automatically generated spam – and they also “avoid” prospective clients who hate filling out forms. But around half have the email address as well, and these are my targets.

The top level of page of the entire agents listing for a specific district in the directory has multiple pages, each containing maybe 10 agent record summaries and there is typically 100 to 500 agent records for any given location. So the first level search needs to automatically progress though the entire first level page-by-page.

There is no expressed restriction on manual roaming and extracting fields for multiple records. I do not use the data in any way which is competitive with the DB operator.

I will provide the DB id and sample (manual) extraction sheet when confidential dialogue with prospective Freelancer is established. DB is slow and clunky and quite a few website links do not work plus DB is fairly volatile due to turnover of agent member/ subscribers and presumable failure of member agents to update the data.

Good idea also to capture both agent and agency names fields from second level pages even for those agent records without a website link (can be saved to same spreadsheet and separated out later by myself).

If not possible to access and scan the pages in the third level (agent/agency website) pages, then in would still be helpful for me to avoid wasting time manually searching all the records which don’t have a website link on the second level page side panel (and save as per preceding para), and, for those with a website link, simply give me a the website hyperlink address instead of the email address. I have never seen an email address on either of the first 2 levels (because it would allow the visitor to bypass the DB operator).

Skills: Javascript, PHP, Python, Software Architecture, Web Scraping

See more: scrape data from multiple urls, outwit hub manual, outwit hub macro, outwit hub tutorial, scraper outwit, online web scraper, outwit hub review, outwit hub scrape multiple pages, need 100 500 cpa, 100 500 words, find words 100 500, story english 100 500 words, image resize 100 500, 100 500 pva daily basi, numbers words 100 500

About the Employer:
( 20 reviews ) Edgecliff, Australia

Project ID: #16408433

16 freelancers are bidding on average $698 for this job

seaanddream

Hi, my name is Sevinc. I will write a exclusive java-based script for you. I read your "Scraper code - possibly OutWit" project descriptions carefully before bidding. I checked the attached document as well... I got w More

$600 AUD in 5 days
(230 Reviews)
7.5
raindropsin

Hello ! If we talk about the scraper then happy to inform you that, Recently i had finished one of the large project in python with DJango. Hope you will be surprised when you will look into the project, it's perfor More

$1882 AUD in 23 days
(85 Reviews)
7.4
phpXpertbd

Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database More

$400 AUD in 10 days
(77 Reviews)
7.2
bestit4u

Hi.. I am an expert in Betting project. So, I am very interested in your project, and I have built many betting scraper and auto betting bot for bet365, bwin, pinnacle, betfair, marathon, 10bet and etc.. And I have w More

$555 AUD in 10 days
(90 Reviews)
6.7
shawnwilliams85

Hi I would love to discuss your needs for this scraper further. I am a full stack developer with 10+years experience and extensive experience building optimized web applications for small to large scale businesses. More

$1764 AUD in 15 days
(39 Reviews)
6.3
Harun1986

Hello, Sir, madam, I am not bid to make a script. But I have recently scrap all USA realtor data with full information. There is total near 800K near 1 million latest leads the list for the realtor. Please let me know More

$588 AUD in 10 days
(57 Reviews)
5.7
ganesh2218

Hello Sir, I can provide you the most recent 1.2 million + US Premium Realtor agents database for complete USA. My data has Agent name, Address,phone,mobile,email, License no,. Issue/expiry date etc. My data has 90 More

$611 AUD in 1 day
(6 Reviews)
5.0
whitehorsetechn

Greeting, I have understood your Scraper code task and can do it with your 100% satisfaction. Please ping me for more discussion. I have more than 5 years of experience in Javascript, PHP, Python, Software Archi More

$555 AUD in 10 days
(14 Reviews)
4.9
sgatich

Hi. I'm a programmer from Russia with strong skills in Python. I can scrape data about agents and agencies automatically. I'm good at web-scraping, browser automation and data extraction. I use Grab, Requests, Selen More

$555 AUD in 3 days
(9 Reviews)
5.1
ProfSoftStudio

Dear Sir! I make Scraper code ++ This is my job. I'm ready to begin now. Pm for me. Please check my reviews and profile https://www.freelancer.com/u/ProfSoftStudio.html

$750 AUD in 10 days
(38 Reviews)
5.0
manikhossain1111

Dear sir, I am a expert web scraper and I will try to provide you clean and fresh data as you want. Thank you, Manik

$529 AUD in 4 days
(14 Reviews)
4.5
AutomationAdept

Hello! I'm interested in working on this with you. I've completed many similar directory-scraping projects in the past, so this should be no different. As for your concerns about the email addresses - I can have the More

$500 AUD in 10 days
(18 Reviews)
4.4
Nada100200

Hello client. Hope you are doing well Over 9 +years experience writing almost exclusively web scraping code. I've done it all. I can scrape all LinkedIn profile My languages in order of experience and use is Python, Ja More

$250 AUD in 3 days
(11 Reviews)
4.6
ChanakyaNaag

Hey there! Please provide more info on the project, website details and the search queries (locations). Ill use python for this task. What would you prefer as database? excel or csv? My skills & experience: -- More

$455 AUD in 8 days
(10 Reviews)
3.6
maunik1996

A proposal has not yet been provided

$616 AUD in 5 days
(0 Reviews)
0.0
$555 AUD in 1 day
(0 Reviews)
0.0