Need a ryanair price scrapper. So it could scrape these prices [login to view URL] and put them in database. I mean prices on the right like taxes, eu levy fee etc.
Also it needs to scrape other things, for example if there is a sign `2 seats left` it should get that too and some other basic stuff.
The idea is script goes through all ryanair search, for example starting with london-riga, london-frankfurt, london-kaunas .... after frankfurt-tenerife, frankfurt-riga, frankfurt-somewhere.... and do that for all routes.
If the routes are not being added automaticamlly it should be easy to add them.
After save eveything to a preferably mySQL database, but this can be changed if there is a good reason.
Also as the script will be updating prices almost all the time, there should be something so it does not get banned and it should know how to deal with errors ryanair sometimes have. Also the script has to be fast, so it would not take an hour to srape all routes, maybe it can be made on like 3 servers, to be faster, but thats up to you.
Preferably script would be written in PHP, but other languages like PERL etc are welcome too.
Few things I would need to know:
what language will it be written in, what databse will it use and how much time will it take for it to scrape all the routes.
Sounds like an interesting project. I have experience doing something similar for concert ticket sites. I would develop in PHP and use mySQL, depending on how quickly they ban IP's may also use TOR or something like that to hide the source of the requests from them. I wouldn't expect it to take more than say 20-30 seconds per route most of that would likely be network time, and then possibly introducing some random delays to make the requests look more like they are coming from a human user and reduce the chances of getting blocked. Requests could be done in parallel to speed things up.