Web scrapping using R: Auto updating forms



Currently, I’m working on a web scraping project which involves forms. I’m using “rvest” package in R.

Problem: The website has auto-update forms which loads more questions based on options we select (dynamic). Is there any way to deal with this?


Not sure about R but selenium would be the path using Python ?
Is that not possible?



It so happens you can use selenium in R to submit info into web forms. I found the following two links, the first introduces selenium and the other one gives a glimpse how to use it for submitting forms -

  1. https://www.r-bloggers.com/scraping-with-selenium/
  2. https://www.computerworld.com/article/2971265/application-development/how-to-drive-a-web-browser-with-r-and-rselenium.html

Also, check out AV’s own blog on web scraping in R using rvest - https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/

Hope this helps!
Sanad :slight_smile: