Web Scraping Using Python
How to use the Scrapy framework for Web scraping
We are looking for A Python Web Scraper to extract data from and transform it into JSON format for a one-time project. The candidat... and transform it into JSON format for a one-time project. The candidate will develop and maintain a Python script for web scraping, extract relevant data accurately, and transform it into JSON format based on our specific requirements. The ideal candidate should have experience in web scraping libraries such as BeautifulSoup, Scrapy or Selenium, knowledge of HTML, CSS, and XPath, and experience with data transformation and JSON formatting. Attention to detail and the ability to work collaboratively with the team are essential. New programmers are welcome, and we will provide necessary guidance and support.
I want a chrome extension that will run in the background of all pages will show a js alert with the xpath location or css locator of any element clicked
Hi, I need someone who is an expert with scraping and will help me set up a basic octoparse scraper. Page 1 take title > click URL and take other elements. ( some xpath scraping is needed)
Looking for someone who can help with setting up xpath for data extraction process.
Need to read data from XML/Json file using typescript. Need to get the value of child node values using its siblings key value pair and should build dynami xpath from obtained values
Hello, I would like to write a simple bot (i have already started writing this script) which would go to the specific page (service now) - one page - could be hardcoded and do operations like that: 1. double click on the specific TD in the same row when the specific text occurs (I think this element can be found only by xpath - there is no ID or CLASS) and input text should be loaded 2. enter text into input and unclick this TD 3. refresh the page 4. return to point 1... etc.. only one problematic thing can be I don't have testing environment and I cannot share the code of the environment so share screen session will be only one option to complete this task
...Product comparison 1.6-) Creation of filtering template by product categories 1.7-) Having a product vendor web panel 1.8-) XML or API integration with product vendor 1.9-) Adding and updating products via the admin page 1.10-) Matching the products of the product seller with the products on the site via the Admin page 1.11-) Roboto to check price change via admin page 1.12-) Price reading robot like xpath. 1.13-) Graphical creation of historical price archive Step 2 2.1-) Creation of the membership system 2.2-) Integration of the comment system 2.3-) Notify system when price drops 2.4-) Price search with barcode code 2.5-) Making a mobile application 2.6-) Product price comparison with QR and barcode on mobile application...
I would need an app that can pull product data from ANY site that I choose. This app must be dynamic and this task is only for advanced web developers. Please do not apply if you have no experience in programming Java Script or manipulating CSS at an advanced level. The app will eventually produce an excel file THAT IS CLEAN. Please no questions about the project as this post is self explanatory if you are experienced. Furthermore, the respective candidate will continue to work on this as time goes by so this task will be an introductory assessment of your skills. Please move forward for experienced candidates if you do not have high level web development experience.
i need help with php and xpath to get value between span it's not get text budget max 10$
I looking for one professional who can use Scraping some websites using my account on , and is a long term job I will need a lot of hours scraping a lot of websites. You need to create task on this software on Octoparce and then send to me to import into my octoparce. Only using this software , not scraping with other tools. Could be good if you did before Scrapy XPath APIs and another tools to scraping. Fell you free to chat if you have any questions. best regards
My company uses Octoparse to aggregate car rental prices from a popular booking website. The website changed their layout a few weeks ago so the previous Octoparse workflow is now defunct. I've managed to recreate the URL lists and workflow with the new website layout, but the scraper misses the first 10% or so of the data on the site. It appears the first 10% of listing have a different XPath which the scraper does not recognize. I've tried manually selecting these fields on different test runs to have the necessary data extracted, but the scraper just duplicates the first selection down the page. I've managed to solve every hiccup up to this point, but this one has me clueless as the scraper is not even able to identify the first few lines of data. This has caused a...
...followings: 1. scrape a web page ( all bundles of the site ) The input of the function is web page source file The outpot should be a python list , for ex: bundle_id, desc, price, GB_count, Min_count, sms_count,auto_renew, period, xpath, url , active_bundle = scrape_page(web_page_source) 2. a function to scrape a bundle details from bundle_id The input of the function is bundle_id The outpot should be a python list like previous one for ex : description, price, GB_count, Min_count, sms_count, auto_renew, period, xpath, url, active_bnudle = scrape_bundle(bundle_id) I included a file which contains a web_page_source called if you are able to extract the above data, then bid Note (1), This site is in Arabic language I prefere the bidder must...
Hello all, A GSM prepaid company sells minutes bundles and 3G bundles, need a python function to scrape the following data: input is mobile number output is the following: 1. bundle ID 2. description 3. Price 4. GB 5. Minutes 6. auto renew ( true, false ) 7. period ( daily, weekly,monthly,renew ) 8. xpath ( must be a seperate function need to be recalled in other places in the python code) 9. url for bundle picture ( to be downloaded ) 10. number of sms 11. active bundle ( if any ) we must input mobile number the function will reurns the above information must develop a seperate function to extract these details for a specific bundle ( input is bundle id ) and it will returns the same above details
I have access to webscraping software that allows the creation of custom scans. I need somebody to provide the xpaths to complete the attached information in the file. Initially it will be 5 sites. If these can't be done for whatever reason I will provide more. Strong possibility of future work.
hi i need help to get the h1 from some link this MUST be with curl + PHP !!!! not pyton if u can do this tell me i send u link and need u realese the h1
hi i have code all work most time but have some issue with some site i see element and not get the text on element it's most work curl and php if u can help, u can with anydesk connect to me to see and get the code if u need only 1 element i need
I have a custom php script which uses xpath to grab product prices from a target website. It worked fine till the target website activated cloudflare's protection and it does not allow my script to parse/grab prices anymore. I suppose cloudflare somehow realises that i am not a human, so target website denies access. Is there any php developer knowing how to overcome this problem? Please respond to this only in case you know how to solve it. Target website is this in case you want to test it:
Am having trouble selecting multiple row at the same time using xpath. Each of the elements as these ids //*[@id=":xa"] //*[@id=":sm"] //*[@id=":u6"] Am in need of a freelancer with a decent knowledege of xpath to help create a global xpath that will select all the items in the table
I am creating a site scraping script and want to get the url with xpath. HTML syntax you want to get onclick="('url') " I want to get this 'url' part. The driver uses selenium's chromedriver and also uses BeautifulSoup.
I am webscraping using Python, Selenium, XPATH etc. I have a basic understanding - however I would like someone to look at the XPATH in the attached file. It is scraping a list from a table - however the XPATH I am using returns a stray element in the list. I don't know where this element comes from or how to filter it out. I am after someone to provide the correct XPATH or solution using SELENIUM. I would like this solved ASAP. Thankyou.
If you access the site and click the check box to search, it will be judged as a checkbox click uses selenium's I want to do is click the checkbox and transition to the next page. T...selenium import webdriver import time ChromeOptions = () driver = (options=ChromeOptions) driver.implicitly_wait(20) url = '' (url) (1) driver.execute_script("(0, 1200);") # click1 driver.find_element("xpath", "//div[@class='searchItem']/ul[@class='horizonal_list w22']//li/label").click() # click2 driver.find_element("xpath", "//input[@id='form_submit_button']").click() # robot judgment -------------------------------------------------------
I am webscraping using Python, Selenium, XPATH etc. I have a basic understanding - however I'm constantly getting errors that I spend way to much time ironing out. For example - the attached code returns "" - however the XPATH does locate what I am after in the DOM. I am after someone to help/ tutor me with this. Thankyou.
Using Python, Selenium, XPATH and Webscraping skills, I would like to: For this link: Webscrape the prize money for each horse. So for this race the desired list would be: [$35,000, $83,210. $38,500, $40,200, $40,550, $47,700, $42,950, $38,475, $52,815, $81,730, $32,250] It is the XPATH and code of how to get this list that I am interested in. I do have Python, Selenium, XPATH and Webscraping skills - but at a basic level. I need this done ASAP. Thankyou.
You will write simple extensions that will concatenate selection with a given string. 1. User selects the string 2. User clicks the right mouse button to show a context menu 3. User then clicks CONCATENATE 4. Extensions then connect to 3rd party API (sending the value of the selected text, div Id, full XPath, URL, and other browser data like version, language, etc.). API has an authorization with JWT, so there has to be an option page where the user enters the token first and saves it (it has to work between resets and ideally use the same system like google cloud sync that is built in V3 manifest) 5. Retrieved value will be used to concatenate the selected text & value. 6. Finally, it will replace the text in the browser. It is designed to work in text inputs. It has to w...
Java Web Services and XML The parsing of XML must be performed, which is further presented using one of the technologies: SAX, DOM, StAX, Xpath and XSALT, JAXB, and at the output, to display detailed information about all books whose price is higher than 10 and which were published after 2005. The project must be sent as a complete Maven Java project made in IntelliJ Idea, SDK Java version 16. When sending project as archive do not forget to delete the target folder.
I am trying to webscrape a value from a website - but I can't figure out what I doing wrong - it returns nothing. Please look at the attached code and pinpoint where the error is. I will pay once the solution is found - I figure it will be easy for someone who is good with Python/ Selenium/ Webscraping. Expected output is: ['Sugar And Sweet', 'Beaudacity', 'Kadena', 'Koyuga Bound', 'My Artemis', 'No Baby No', 'Lily Allez', 'Frisson', 'What Happened', 'Bastyan', 'Wasabi', 'Eastern Pegasus'] Thankyou Angus
Only needed one day to consult on highly technical difficult java item Expert High Level Java Skills SQL skills Selenium Reg Expressions Xpath
Only needed one day to consult on highly technical difficult java item Expert High Level Java Skills SQL skills Selenium Reg Expressions Xpath
Job description Responsibilities: o Create new E2E tests o Analyze test-reports o Modify existing tests cases Skills Must have: o Good knowledge of BDD (Behavior Driven Development) o Experience with Selenium or selenium-based frameworks o Strong understanding of Xpath and CSS locate strategy o Understanding of REST architecture o English A2/B1 o Experience with E2E API testing o Knowledge of any programming language (priority Java/Groovy) Nice to have: o Experience with Spock framework o Conception and execution of performance and load tests o X-Ray know-how and skills o Experience with Git o Experience with Gitlab/Jenkins CI/CD to configure jobs for auto-tests o Experience with Docker
Hello, I started a small webpage where people can try and learn XPath. The functionality should be very straightforward. 1. a field or div should show some XML (pre tags?) 2. from an input field people can enter an XPath expression 3. XPath should be evaluated on keypress 4. if there is some match, then the selected block or line should be highlighted. This should work on a standalone page. You can use Jquery. I will share what I have so far.
...Service. Basically the addin will monitor the current url that is navigated to in the browser. Should it be of a particular website page type, then it will extract some values from the current page using XPath supplied, and then contact the ASP.Net Core WebAPI with these values. The server will then check a SQL Server Table for each URL passed, and will send back a response as to whether the url is in the database table or not. The response will then be updated on the addin window within Chrome to show the status, but also update the various elements they relate to based on the XPath that was originally used, to add some extra markup like opacity values to some html tags. There is an add button for each one, that will be a new html element added, and if clicked the ad...
...1 or more Party objects; each Party will contain one @Id attribute, and one Outcome/Choice attribute (OverallChoice/Choice/Outcome @Choice). It is important not to validate the files against the provided XSD - the XSD is provided for information/assistance, but the XML files may contain data that is not included in the XSD. However - the 3 data points we need will always be found at the same XPath locations: TransactionId: /BaseSchema/BaseSchemaResponse/Result/SchemaExtension/Response[@TransactionId] Party Id: /BaseSchema/BaseSchemaResponse/Result/SchemaExtension/Response/Parties/Party[@Id] Outcome/Choice: /BaseSchema/BaseSchemaResponse/Result/SchemaExtension/Response/Parties/Party/OverallChoice/Choice/Outcome[@Choice] The output file will be a plain text CSV file ("&q...
Need to develop a python script to automatically some value from a website by using XPath
...test automation for Web ● Solid test plan and test case writing skills ● Excellent understanding of SDLC and quality process ● Experience with Industry standard test automation, test and defect management tools ● Intermediate programming skill in Java (Must have) ● Experience with web automation testing by Selenium, Appium ● Experience of OOP and design patterns ● Experience with CSS selectors / Xpath selectors ● Experience with REST API testing ● Experience with using CI tools ● Experience with any form of integrated development environment. ● Fluent in English to liaise with international teams What do I get when working at Aspire: ● Work full-time remote (permanent) for a Singapore fintech product company ● Work with a Singapore fintech product ● MacBook Pro M1 2020 (256GB, ...
To develop a Campus Facility Booking system with 3 required modules which are: 1. Login/Registration (Admin and User panel) 2. Book Facility 3. Geting user feedback The PHP Web application does not require a fancy UI but needs to implement the following list: 1. Suitable Design Patterns (Eg: Ada...Campus Facility Booking system with 3 required modules which are: 1. Login/Registration (Admin and User panel) 2. Book Facility 3. Geting user feedback The PHP Web application does not require a fancy UI but needs to implement the following list: 1. Suitable Design Patterns (Eg: Adapter, Observer, Facade, etc | Preferably: laravel) 2. Secure Coding Practices (Eg: Hashing) 3. Usage of XML, XSLT & XPath (Eg: convert database file into XML and display XML data into PHP) 4. Web Servi...
I need to scrape some data from webpage and the problem is my knowledge on xpath is limited i need a query to tell me if the specific table rows is empty or no based on Header if the data exist get the data. if the developer need access to TeamViewer is ok
... Qualifications 5+ years of experience in IT support or software development, testing. Sound knowledge in Microsoft BizTalk, C #, and SQL Server. This must include the following. Creating Schemas (XML, Flat file, CSV, etc.) Configuration of WCF Custom adapters for Basic HTTP, SQL, Oracle, FILE, FTP, SFTP, etc. Creating WCF or REST API web service using C# code. Working knowledge of XML, XSLT, XPATH is a must. Strong focus on delivery and service-oriented mindset. Strong understanding of object-oriented programming Should have the strong analytical skills to understand the development requirements. Proficient understanding of code versioning tools like TFS (Team Foundation Service) Must understand concepts like B2B, EAI, and ESB. Working experience on JIRA tool and Agile methodol...
I need an Xpath generating expert to help me create xpaths. I will say things like "write an xpath for the first search result on this page" and need to have the xpath created.
Hi All, I would like someone to create a Chrome extension that ables me to find CSS selector...The chrome extension should able to find the following when right-clicking on the component . CSS selector . Xpath . rxpath . Id (if the element has ID) There are similar extensions available in Chrome such as RANOREX SELOCITY. However I want one for my own company. I need someone to be expertise in Front end development and Chrome extension development
...recording in complete xpath as below. This is needs to be converted to workable C# format as FindElementByXPath("*//").click "/Pane[@ClassName="#32769"][@Name="Desktop"]/Window[@ClassName="ePace!Admin30"][@Name="AFX Engagement Administrator (Main Office Server)"]/Pane[@ClassName="AfxMDIFrame140"]/Pane[@ClassName="AfxMDIFrame140"]/Pane[@ClassName="AfxFrameOrView140"]/Tree[@ClassName="SysTreeView32"]/TreeItem[@Name="My Firm"]/TreeItem[@Name="Staff Groups"]/TreeItem[@Name="StaffQATest (1)"]"; If you can send me the sample xpath code that works in findElementByXpath().Click(). for above sample test, I will commit a project $...
I need help for Test Automation using Window App Driver and Xpath
I looking for one professional who can use Scraping some websites using my account on , and is a long tern job I will need a lot of hours scraping a lot of websites. Could be good if you did before Scrapy XPath APIs and another tools to scraping. Fell you free to chat if you have any questions. best regards
Looking for an experienced coder to write scripts to extract data from which are in xbrl (xml) or json formats and feed it into mysql database and also provide a CSV / excel format. Xpath option proposed can be considered. Any recent effective solution is invited as well.
My application developed in c# desktop application to send mass messages (text and image) to selected numbers by using whatsapp. i got error in xpath as following code (("//*[@id='main']/footer/div/div/div/div/div/div/div/span")).Click(); and not sent the message got error in this breakpoint.
How to use the Scrapy framework for Web scraping
This article describes almost everything about what to expect from the new updated XML TextView.