I gone through some small tutorials about scrapping using python. But I have a huge doubt about legality of this technique. How can i know from which website I can scrap data? I tried reading about robots.txt. Still confused. What if I try scraping and caught in some law breaking issue. Ha ha i m thinking too much, but it would be great if anyone help me out.
Web scraping is issue with some websites. Some websites are OK to scrape while some are not. it is usually mentioned in user agreement of every website. for Ex. for LinkedIn it is mentioned in their user agreement that they do not allow scraping (see attached pic).
However there are some websites that do allow scraping. Article that I am attaching gives you very good idea about rules of scraping.