Simple Scraper/Parser for 10M Queries (High Performance)

$30-250 USD

Completed

Posted

almost 4 years ago

$30-250 USD

Paid on delivery

Hi, I need a simple parser that is able to make millions of queries in a reasonable time (couple days) to a public API. It needs to support proxies, as the API is very volume-restricted on an IP basis. The parameter passed to the API is a keyword (a company name) and the return object is a small (ca 1kb) XML. Important: You need to parallelize requests / support concurrent queries. This is the main challenge. Save all results to a text file. I have some working code (in Java), however, its performance is not very good. I can give you the following: - sample list of 10.000 keywords/companies - the API endpoint - my existing code (if you like) You can choose any language you like, does not have to be Java. Roughly aiming for 3-5 days turnaround, fast payment. Thanks

Project ID: 25774911

About the project

14 proposals

Remote project

Active 4 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

Awarded to:

@elbruninh

⭐⭐⭐Hi, i'm interested, could you give me more details please? do you have a proxy service? can you send me the info? keywords list, API endpoint and your code i can do it with Java (multi-threads) let me know if you are interested [login to view URL]

$160 USD in 3 days

4.9

(76 reviews)

6.1

14 freelancers are bidding on average $208 USD for this job

@ramzitra

Hi, I am interested in your project related to build a highly performant scraper to do 10 M queries. Please send me a message so that we can discuss all the details. Thanks, Ramzi

$300 USD in 10 days

5.0

(534 reviews)

8.6

@mhmhz

Hi I used to create tool that support multi-threading, proxies. Waiting for discussing to estimate time/feee Thanks

$400 USD in 3 days

5.0

(130 reviews)

7.8

@fatimatuzzohura

Hi, I am a professional software developer with 5 years experiences of Python . I have all required skills and knowledge to successfully complete assigned tasks and provide an optimal solution. I am skilled in both back-end and front-end development as well as setting up multi-server environments. Thanks

$250 USD in 5 days

4.9

(136 reviews)

6.8

@nisharapath

Hello, how are you? Your description was gone with me well and it is clear for me. My python skill will be applied for this job. Please feel free to contact me. Regards.

$150 USD in 1 day

5.0

(81 reviews)

6.6

@benliao

Hello, sir. How are you? I have just read your project description and I can help you as my rich experiences. I have rich experiences in Web Scraping and python I am a new freelancer, but I can give you the perfect result you are satisfied with. Thanks and Regards

$140 USD in 3 days

4.9

(11 reviews)

5.1

@AIPDTech

Hi there, We reviewed your requirements and our Fullstack Developers Team with SKILLS - PHP, JAVASCRIPT, Python. We would like to work on web scrapping as per needs and willing to discuss scrapping requirements in detail. We are a reputed IT company with experienced professionals and have been web scrapping for websites from 7+ years. Warm Regards, Naveen

$175 USD in 7 days

4.9

(8 reviews)

5.4

@ozo42

hi. is it a public or private api? also how many proxies are available and how many queries can be dobe until they ban the ip?

$247 USD in 2 days

5.0

(9 reviews)

4.7

@dduraipandian

I have very good experience in web scraping. My recent scraping job was to scrape where i have to scrape product makers, engine family and product. Difficulties are, 1. search page is a form with javascript event changes. It is not a direct form submission. 2. Once you get to main pages, you will have pagination. Without pagination there are 467 urls to scrape. With pagination, 670 urls to scrape. 3. It is not easy to scrape at once as i get blocked. I scraped with all data with timeouts (without proxies) in one hour (development time is not included). Above can be optimized further with async calls and proxy implementation. Please let me know if you are interested to discuss.

$250 USD in 7 days

5.0

(11 reviews)

3.8

@panhavoan

Hello there, I could help you with that project. I've worked with nodejs for years. Puppeteer, x-ray and other supportive are the library that I normally use to scrape data. I have rich experience with this technology. I have completed many scraping projects. I'm a fast worker. I'll deliver the project on time. So please consider hiring me. You can check my recent work with scrapping project. Please contact me I'll send some of my previous project and we could discuss more about this. Thanks

$150 USD in 3 days

5.0

(8 reviews)

3.2

@SiniyaBoroda

***Delivery within a few days*** Hello. How are you? I have read your project descriptions in details. I am so interesting in your project as a professional expert in python scraping and building the rpa tools for getting datas by using selenium & bs4. I can start work right now if you want. And I'm sure that I can implement your project perfectly in time as I have already so many project whichever I have done. Looking forward to be your a good venture along with your project. Best Regards.

$200 USD in 3 days

4.7

(6 reviews)

3.1

@karmakarsantosh2

Greetings, Hope you are feeling very well. I think here you are looking for someone to build smart file in excel. I understand your project description. But I want to know more about your project. So, can we discuss more? I am really confident that, I can do this task very effectively. I have some good samples and references readily available for a similar kind of project. I'm ready to start now! So, if you hire me then I can be trying to complete this task within your timeline and 100% accurate. Please trust me, I’ll provide you the best quality work. So, please take your time to discuss more on your project with me.

$140 USD in 7 days

0.0

(0 reviews)

0.0

@amosso

Hi, I like your project. I have experience in doing parallel processing in Python handling large amounts of data. I can do this in 5 working days. Just share with me your current Java project and some input data for testing and I will deliver an awesome fast API. One question, Which database are you using? Regards. Alex Mosso

$190 USD in 5 days