Scrape websites with scraper and parse data to elasticsearch / kibana / logstash


- Show my products, and show competitor (29 sites) prices of that same product.

(!!! there needs to be a solution to link productname of theirs to mine productnames. I have 'Brand', and 'Modeltype' to make a link easyer!)

- Visualize price progression over time with kibana per product / store. (Schedule a scraper every night!)

- Show if a price is dropping or going higher per product.

Use the opensource python scraper 'Scrapy'.

Feature Goals:

Adding sales of my shop, adding positions of product in categories of competitors and mine.

The scraper will run on a DigitalOcean VPS. You install the server + scraper etc.

The Kibana Logstash etc. server will run on other server.

Skills: Elasticsearch, PHP, Python, Software Architecture, Web Scraping

See more: scrape data public websites pdf, scrape data multiple websites, programs parse data external websites, certified data entry websites, data entry websites home, data extraction websites, use php extract parse websites, capturing data 2008 websites, weather data parse, parse websites, email data mining websites, parse html screen scrape, data mining websites excell, data processing websites, data collection websites

About the Employer:
( 8 reviews ) Nijmegen, Netherlands

Project ID: #12186308

38 freelancers are bidding on average $634 for this job


Hi, Nice project you have there, let us help you with it! Our team is Russian-Canadian. We code in PHP. Check our recent projects here [login to view URL] Here’s a little video about our team too! ht More

$750 USD in 7 days
(106 Reviews)

Hello, my name is Olya. I represent Ukrainian IT Company «Webbook». We provide website design and web and app development services for organizations, public and government institutions, company or private web-pages. More

$600 USD in 20 days
(141 Reviews)

I have 297 reviews check I am Eng. Tarek have a Master degree of Information Technology I am a web programmer Professional in programming Mobile Phone using (ionic), PHP More

$722 USD in 7 days
(114 Reviews)

Dear Client , We gone through project description We are Really intrested to do work with you , We are Web & Software development company having more than 8 year experience in Software development we totally commit More

$555 USD in 10 days
(191 Reviews)

Hello, I want to show you all relevant Demo and Designs which is similar to your project completed previously. To make sure about the requirement set and customization, I want to discuss this project with you furthe More

$515 USD in 10 days
(27 Reviews)
$607 USD in 365 days
(113 Reviews)

Dear Client, Greetings from Flowgica technologies, I have experience with these skills. We do have similar experience doing something similar to yours therefore I am looking forward to discuss and move ahead. please ch More

$390 USD in 12 days
(39 Reviews)

Dear Sir/Ma'am, I am a Web research, Data Entry & Webs Scrapping expert. I checked and understood your requirements. I can handle this job very well to your appreciation. I can find and extract the information More

$526 USD in 10 days
(127 Reviews)

Hi, this is Anshuman. I have 6 yrs of experience in scraping, crawling, processing and mining data. My previous projects include- [1.] Automated Crawling of Google for SEO Keywords [2.] E-Commerce Crawling. Crawli More

$750 USD in 10 days
(53 Reviews)

Hello, I understood the initial scope of this project. Although i want to discuss further this job in order to prepare the final concept for this project. After Complete discussion over the call or in chat, i wi More

$773 USD in 10 days
(54 Reviews)

Hi, I am interested to work with you on this project, I reviewed the requests you listed and I would like to discuss further details with you. Since 2006 I am managing a Web Design Agency located in Romania, which cur More

$889 USD in 30 days
(46 Reviews)

Hello, We are happy to discuss our proposal with you. KEY POINTS THAT MAKE US DIFFER FROM OTHERS: ~ We have 5+ years of experiences in Unity 3D, Xcode, CoCoa 2d, Phonegap and major iOS and Android developement. ~ More

$1000 USD in 10 days
(56 Reviews)

Hello Web search ,Web scraping and data mining expert is here .I have done many similar scraping projects similar as website which you mentioned on description.I am able to complete your project at lower rate and qual More

$250 USD in 10 days
(69 Reviews)

I am eCommerce developer that have many experience to develop the online shop sites. In general, all products are from out-source, so it is efficient to scrap product details from outside, so I have developed many pro More

$1111 USD in 20 days
(61 Reviews)

Changing budget according to the new info and discussion in the chat for spider for 29 sites and other things from the desc

$5000 USD in 30 days
(64 Reviews)

Hi, Thanks for the opportunity. I have gone through the details of your project and I find it well within our capabilities. Please spare a moment to discuss this project. Waiting for your message Thank More

$555 USD in 10 days
(65 Reviews)

I am a skilled freelancer who has a working experience of more then 4 year I have working experience on- Web Research, lead generation, data entry, data mining, lead collection, All types of scrapping tools like - da More

$250 USD in 10 days
(32 Reviews)

Hello, Its a pleasure to let you know that I've Completed and Delivered similar project before. All I need to work upon customization part, if we can proceed towards more discussion. I have gone through your project More

$666 USD in 15 days
(12 Reviews)

Hello there, We are interested in offering our web scrapping services for collecting products related data. We have completed various projects related to web scrapping making use of both automated and manual tec More

$250 USD in 5 days
(53 Reviews)

Hello Dear Sir how are you hope you are well Dear Sir I am An Expert Data Scraper I Can Scrap Data Manually or by using tool (code) as per your requirement if you Really want 100% Error Free and Quality Work Done More

$250 USD in 10 days
(15 Reviews)