Share your repls and programming experiences

← Back to all posts
Web-Scraping-XKCD903
h
timmy_i_chen (1003)

Here's a fun little project that puts that old XKCD comic to test: Repeatedly navigating to the first non-parentheses non-italics link in a wikipedia article will eventually lead you to Philosophy. It starts at a random article (but you can totally choose its starting point by changing line 64.

When I hand-tested this myself many years ago when I should've been paying attention in school, it definitely worked. Sadly, it seems to no longer be the case - although now, it seems as though all articles eventually lead to Mathematics... which I suppose also works.

https://repl.it/@timmy_i_chen/Web-Scraping-XKCD903

Link to original comic (hover over the image to see the alt-text):
https://xkcd.com/903/

Commentshotnewtop
pyelias (907)

I'm coming for you! 42 votes and counting right now.

ebest (590)

I put in Mathematics but it didn't work. Output:

Starting at Feriz Beg
main.py:39: UserWarning: No parser was explicitly specified, so I'm using the best availableHTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 39 of the file main.py. To get rid of this warning, pass the additional argument 'features="html.parser"' to the BeautifulSoup constructor.

  return BeautifulSoup(''.join(result))
Navigating to /wiki/Ottoman_Empire
Navigating to /wiki/Anatolia
Navigating to /wiki/Asia
Navigating to /wiki/Earth
Navigating to /wiki/Planet
Navigating to /wiki/Astronomical_body
Navigating to /wiki/Physical_body
Navigating to /wiki/Physics
Navigating to /wiki/Natural_science
Navigating to /wiki/Branches_of_science
Navigating to /wiki/Science
Navigating to /wiki/Knowledge
Navigating to /wiki/Fact
Navigating to /wiki/Reality
Navigating to /wiki/Existence
Navigating to /wiki/Ontology
Navigating to /wiki/Philosophical
Navigating to /wiki/Problem_solving
Navigating to /wiki/Cognition
Navigating to /wiki/Intellectual_function
Navigating to /wiki/Template:TWCleanup2
Navigating to /wiki/Wikipedia:WINAD
Navigating to /wiki/Dictionary
Navigating to /wiki/Word
Navigating to /wiki/Linguistics
Navigating to /wiki/Science
Stuck in a loop (we've already been to /wiki/Science), exiting..
Looks like it didn't work.  Super sad. :(

That was errors included, but it "Looks like it didn't work. Super sad. :("

masonclayton (13)

seems like all roads lead to /wiki/Mathematics now

Spandan14 (10)

Nice, but what is the usage?

rileyball2 (22)

Change Philosophy to mathematics (like I did in a branch) and it works almost 100% of the time

CyanCoding (1415)

Awesome! I also made my own version of this (it scrapes regular wiki pages though): https://repl.it/@CyanCoding/Wiki-Web-Scraper

serdav74 (5)

Nice! Though there is a problem: it crashes when the first link leads to external website (like wiktionary)
https://puu.sh/zmUoa/3b90af9a9e.png

TimmyChen1 (111)

@serdav74: Ooooo, that shouldn't happen, it's in parentheses! Thanks for letting me know - I'll try to fix it :D

Gymnast456 (1)

@timmy-chen: Do you know how I can add an image to python with turtle??

SomeRandomGuy (0)

"Traceback (most recent call last):
File "python", line 99, in <module>"

Every time I run it. Please fix it
IndexError: list index out of range"

TimmyChen1 (111)

@somerandomguy: Added a quick conditional to break out of the loop if the index runs out of bounds. Thanks for letting me know :)

426729 (144)

It's all leading to Philosophy. This is a great program! Keep up the great work!

DarthVader1 (0)

all roads lead to science

NicholasChorett (1)

..but whats the point of this...? sry im dumb

SCollierCode (2)

@nicholaschorett: It has no real, meaningful purpose. However, it is a funny reference to XKCD, a popular web comic, and it tests a rather old joke/piece of trivia. So in the grand scheme of things, there is no point, but if you're a fan of XKCD or just computer-based jokes and "tech" culture in general it may be of some relevance and/or interest.