(100$) Scraping work based on Cloud computer. -- 4
$30-250 USD
Closed
Posted almost 5 years ago
$30-250 USD
Paid on delivery
Hello. thank you for having interest in this topic.
*Please read description first thoroughly, if you're copy-pasting meaningless words I will instantly dump the proposal. Thank you *
If you have experience in scraping/cralwing work, You know there's a capacity which one scraper one thread can be running.
And in order to compensate this problem , we often adopt multi-threaded system inside of the script, But even when adopting this, We could be still having the same problem.
This is why most of crawlers end up with using Cloud computer/servers , setting multiple virtual computer(server,cloud), setting multi-threaded script inside.
Not necessarily to explain long about this, If you have experience, You'd definitely understand what I'm saying.
I'm here to find decent future co-worker who has related skills to setup such environment and have experience to setup and build such models.
Project price 100$ is related with the right next project we're going to jump on, that I will inform you through DM. (Direct message) or sending official offer.
but beforehand, let me ask you questions.
1) Do you have experience multiple server/cloud/VM based scraping work?
2) If you have answers yes to 1), Could you show me a simple demonstration video about how it gets to be run? (No need to show script or inside-tricks, just on surface, simply the scenes that should be enough to prove it's working)
3) Did you also have experience with proxy-rotation adoptation?
Please fill in the form with sending proposal. If our communication is not in exact form, We can not proceed any further. and I will kindly reject it all once it's not in the question form. Thank you.
Hi there
I read the job description with interest.
Now I am using godaddy and aws hosting .
I have skills in HTML5 / CSS3, PHP, javaScpript/ JQuery, PhotoShop and other programming languages and software.
I'll be happy to work on your project
I have a good experience in this job.
if you award me, I will fix your website.
I am available at full time and can start immediately.
If you award me, I can complete quickly.
Best Regard.
Thanks.
Hi.
I did read the project description and have a few questions.
1. Do you need the script as well or data only?
2. What is the format of the output data? CSV is OK? We can do other formats as well.
3. Which fields do you want to extract from the website?
4. What is the website?
5. How many results/urls are there?
Thx, waiting for these details and hope to collaborate.
1) Do you have experience multiple server/cloud/VM based scraping work?
Yes, I have the strong experience where we are using more 400VM's. Multi-Threaded Processor Intel Xeon E5-2680 V3
2) If you have answers yes to 1), Could you show me a simple demonstration video about how it gets to be run? (No need to show script or inside-tricks, just on surface, simply the scenes that should be enough to prove it's working)
Yes, I can share the Screen Shot and personal message or anydesk if you want .
3) Did you also have experience with proxy-rotation adoptation?
Yes, Actually Instead of proxy-rotation we have setup our VPN Server and each VPN Server have more then 100IP's and each machine is actually connected to its own dedicated IP.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I have been working on such project from last 2-months with one of our client.
We are using around 400 Vm's machine.
Our client has some plugin which is enabled the browser like (Chrome, Mozilla, Opera)
We have setup the VM's with almost 400 Vm's with Multi-threaded Processor Like Intel Xeon E5-2680-V3
You can add me on skype @datanethostingsolutions
hi this is Vikram
aws expect ..
presently working as cloud engineering intern in Piruby
I have helped the company to move to cloud..
I can send you the screenshots of the servers I have set up and installed things in them .also all the servers scale automatically based on cloud watch metrics and load balancer balance requests across the instances
I have got my own vpn server running which is just like a proxy server ,I can even set that up
I can do all kinds of work in the cloud whatever you mention ..
running scripts on startup of instance.. also taking up the image of instance for further use instead of seeting the servers again and again
launch template for future use and running the servers
auto scaling group for increasing the number of servers
many more things ...
proof will be shown later
thanks