Find Jobs
Hire Freelancers

Improve webpage scrapping solution

$30-250 USD

Closed
Posted over 3 years ago

$30-250 USD

Paid on delivery
I have a Java program to scrap information from a website. The architecture of the solution involves: 1) using Java Selenium to send requests to the webpage via Chrome Webdriver to trigger authentication and authenticated requests; 2) routing the requests from Chrome (headless) to Java BrowserMobProxy to capture three HTTP headers (Authorization, X-CSRF-TOKEN, and Cookie) and one query string; and 3) use these 4 elements in HTTPs requests from Java directly to the webpage (i.e. without Selenium, Chrome, and BrowserMobProxy involved) to retrieve the desired information. This program does the basic functionality of extracting the information but has a few problems: It depends on an external non-Java component: Chrome WebDriver It depends on Java Selenium and Java BrowserMobProxy, two dependencies that I would like to remove It is not optimized (too much refresh and too long sleep periods) relatively to the limit upon which the Webpage (Cloudfare) starts responding 429 errors. Thus, the retrieval of the information is taking much more time than needed. Deliverables You will get the current program Java code and you will need to solve the problems above. To do so, you will need to: A. Find out how to authenticate and refresh the 3 headers and the query string without depending on Selenium, Chrome Webdriver, and BrowserMobProxy. As most of this data is likely generated in JavaScript, you will need knowledge about JavaScript and how to execute JavaScript from within Java or convert the JavaScript code to Java (preferable solution). B. You will need to identify the limit upon which the Webpage (behind Cloudfare) starts responding 429 errors. You will need to tune the refresh frequency of the headers and sleep periods to the limit identified. You will need to demonstrate the benefits of your changes by extracting the information currently extracted by the program and measuring how long it takes. Note: you will need to create your own login/password in the webpage. No additional requirements exist to register.
Project ID: 26781086

About the project

11 proposals
Remote project
Active 4 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
11 freelancers are bidding on average $171 USD for this job
User Avatar
Hi, Greetings! ✅checked your project details: Improve webpage scrapping solution ✅Completed Time: In project deadline We have worked on 600 + Projects. I have 6 + years of the experience in same kind of projects. If you are looking for a true Freelancer, I am the Right person for you. I am available almost 24-7 and am very responsive. I feel proud that I am a trusted Freelancer who pleases almost every single client. You can rest assure, your work will be delivered well in advance of others, with passion and accuracy. I guarantee you instant communication & responses when you need me. Why choose me? I think every client is the reason for my success. I only take projects which I am sure I can do quickly. My Portfolio Items: https://www.freelancer.com/u/schoudhary1553 I would really like to work with you on this project. If interested, Kindly contact me via chat for further details and discussion.. Thank you Sandeep Digital screencast
$220 USD in 4 days
5.0 (144 reviews)
7.4
7.4
User Avatar
NOTE : I HAVE EXPERTISE IN JAVA AND JAVASCRIPT. I CAN COMPLETE THIS TASK IN QUICK TIME. With respect to this project I would like to present myself as a candidate for your consideration. I have more than 12 years of IT experience. I have successfully completed projects which involved Programming, scripting. Key features of my work are as follows: 1) Always within schedule 2) Always within scope 3) Best tools and techniques used for the completion of tasks Looking forward to hear from you. Regards
$140 USD in 4 days
4.9 (42 reviews)
5.7
5.7
User Avatar
Hello man. How are you today? I am very interested in your project, so I have prepared a bid. I know you need significant help. I sincerely want to help you. Of course, I know that to help you, you need to have experience with concepts and practices. As a developer for a long time, I have been practicing my skills and tech. So I'm sure I can help you as perfectly When you contact, our conversation will be more interesting. I will wait for your kind reply.
$100 USD in 1 day
5.0 (17 reviews)
5.2
5.2
User Avatar
Dear Employer, I have read the project details and confident to work on improving web page scraping. I have extensive knowledge on Java, javascript,python, software architecture etc . Kindly message me so that we can discuss more about the work. Regards Lucky
$222 USD in 3 days
5.0 (36 reviews)
5.2
5.2
User Avatar
Greetings Sir, I am Muhammad Faisal and i am a professional Java developer having almost 4 years of experience and we provide you quality work within your time and budget so, lets get started ? Thanks
$140 USD in 7 days
4.9 (23 reviews)
3.7
3.7
User Avatar
Hi, I am an expert developer who has done extensive web scrapping work only with Java. I can remove all dependencies on Chrome and remove any sort of sleeps. The way i'd do it is to only wait until the DOM is rendered or the specific elements are ready. Can get this done while keeping the code clean and easy to read.
$250 USD in 7 days
4.8 (2 reviews)
3.3
3.3
User Avatar
Hello,sir.. How are you?... I am very interested in your project. I have rich experiences in scraping website and processing data within selenium package. Have experiences in scrapers used chrome extension and proxy server. It is needed when scraper login in website and get enough data. After get all data, to act what you want is not problem for me. I have top skills of automated scraping and processing data what you want. I can create the scraper with PYTHON, PHP or C#. Will be provided the professional solution. If you hire me, I will work for you until you are satisfied and complete the project perfectly. Will wait for your reply in full time. Thank you.. Best regard..
$140 USD in 7 days
5.0 (4 reviews)
2.5
2.5
User Avatar
i have optimized a lot of distributed crawling projects and Selenium based crawlers ... please connect me over chat and share me your code. i will first identify all the pin points and alternatives for your authentication and authorization libraries .. believe me I will make it much simple and regarding delay and timeouts i need to look code.. please share me your source code.
$100 USD in 1 day
4.4 (1 review)
1.7
1.7

About the client

Flag of ROMANIA
Băilești, Romania
5.0
1
Member since Mar 8, 2020

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.