Find Jobs
Hire Freelancers

Modify Python Code to get around cloudflare 1020 error when scraping www.oddschecker.com

£20-250 GBP

Completed
Posted over 2 years ago

£20-250 GBP

Paid on delivery
Dear Freelancers, I have some code that I have been using since July 2020 which has now stopped working. The code will be supplied to the successful freelancer and the code will need to be modifed to fix two issues that have occured in the last two weeks. 1. The code will no longer run headless, code will run with the scraper showing in the interface. This issue started on 07 October 2021 2. The code stops running once it has scraped the first link - issue started 18 October 2021. The issue looks like once scraping loop gets to the second link, the cloudflare protection is stepped in Skills required: Python BeautifulSoup Selinium using Chromedriver I am looking for the following: -existing code to be fixed, not to be completely re-worked from scratch. -as much of the original source code to be kept as orignal as possible. -explaination of the the work done. The solution works in the following way: -A CSV file is used as a input for the scraper, containing URI's of oddschecker to scrape (Must not change) -The scraper reads in the URI's and scapes (Can be changed to get around Cloudflare check) -The output is saved to a CSV. (Must not change) I have done some googling and believe the issue may around the user-agent being used/sent by the webdriver. I do not know how to implement this, therefore my request for help! The successful bidder will have a long-standing relationship with me and I will come back to you for future work/enhancements. I will provide more detail/instructions to the successful bidder. The code may be very simple to fix, I don't care if it is a single line! - Might be a very quick job for somebody! VPN's must not be used to get ariound the issue as the code is run on a remote server. Thanks and Happy Bidding!
Project ID: 31857728

About the project

12 proposals
Remote project
Active 3 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
If the number of requests per second aren't too high then it is possible to work with cloudflare using a single IP. Send me the details in chat and I'll take a look and will update you within few hours.
£20 GBP in 1 day
4.8 (21 reviews)
4.3
4.3
12 freelancers are bidding on average £146 GBP for this job
User Avatar
+++++++++++++++++++++++++++++++++++ Hello Mathew Hope you are doing well I have checked about your requirements and my skills are a match. I have relevant 5+ years of expertise in Python, Data Analysis, object detection, computer vision, face detection, web scraping, chatbot, Django, machine learning, Artificial Intelligence. Let's meet in personal chat for a quick discussion in it. Thanks Virang Please note that this is a placeholder proposal, we can be more specific once we get all the requirements and information required to execute the project.
£200 GBP in 7 days
4.9 (35 reviews)
6.6
6.6
User Avatar
Hello, I hope this finds you well. I have just seen your project requiring; Python Web Scraping BeautifulSoup Selenium Webdriver I believe that my 10-year experience in this field is what you need right away. Avoid the headache of looking further. Let's save time and focus on the real task. My proposed timelines and budget are just placeholder and an open for negotiation to increase or decrease as per the full requirements at hand. Allow me to prove how better my review can be. Smile all the way! Click the message button, so initiate the conversation. Regards, Fridah
£200 GBP in 7 days
4.2 (14 reviews)
5.7
5.7
User Avatar
Hello, sir. I understand that you want me to fix the script so it will work headless again and Cloudflare stopping you from going to the second website. I can keep your original code as it is and do not change your output. The 1020 error code is the error code when the browser fulfills custom rule which's created by the website owner. It could be anything. As you said, it might be about the User-Agent. It might be about the Referer header too. I can't say something exact without seeing the website and its behavior.
£100 GBP in 2 days
5.0 (6 reviews)
4.1
4.1
User Avatar
Hi! This is interesting Project and I would like to do this for You! I have a experience with Web based Linux, Bash and Python scraping projects. - Python parser with transparent logging. - I use libraries libraries Selenium, Bs4, Requests. - Web develop In Django, Django REST API. - It is possible to use proxy lists for scraping and Multithreading. - Saving to CSV, Excel, SQL, JSON or a cloud. - Ability to track the appearance of new scraping objects during rescanning. - Send notifications by Mail,Telegram and etc I can do it QUICKLY and EFFICIENTLY! Best Regards! Viatcheslav!
£100 GBP in 2 days
5.0 (6 reviews)
3.8
3.8
User Avatar
Hi, I am a senior Python scraping expert, and I have rich experience in BeautifulSoup, Selenium. So I can check your code easily, and I will fix your issue in a short time. As you can see in my latest review, I have fixed python scraper issue. Therefore Please discuss with me via chat. Thanks.
£200 GBP in 1 day
5.0 (3 reviews)
3.3
3.3

About the client

Flag of UNITED KINGDOM
West Haddon, United Kingdom
5.0
3
Payment method verified
Member since Jun 30, 2020

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.