Ask coding questions

← Back to all posts
1
lower number of users than expected (web scraping)
MaxVandervelden (0)

https://repl.it/@MaxVandervelden/Medium-Scraping

Hey, so I just started working with web scraping and I wanted to try to see if I could get the general links of people that follow me on Medium. The code works fine, except it always returns a much lower number of links than expected (20) when it should be 163. The links for the users are all under the exact same class, so I don't think that's the problem. I don't know exactly why this is happening, and my html knowledge is pretty garbage.

Thanks

Commentshotnewtop
1
a5rocks (502)

I don't really know if you are doing anything wrong, but maybe try using more high-level tools like here to begin with (at least that's what I recommend):

https://repl.it/talk/share/A-Webscraper-Written-in-Python/9773

However, the site lazily loads, which means it requests more content when you reach the bottom, so maybe on the initial load of the website, only 20 profiles are shown? That or your class selector is off xd