Dealing with "smart" quotes and other HTML annoyances when parsing HTML text using Python and BeautifulSoup
I've recently been writing some homegrown scripts to parse web pages and do something useful with the information using Python 2.7 and the BeautifulSoup library. This involves printing the output to a terminal (e.g. Gnome Terminal) where the output is hopefully human readable but in some cases was not. The most common issue I ran into were various forms of "smart quotes" and similar html annoyances. If this is not properly handled you'll get output like John X s Blog (where X is the following 7 character group without the intervening spaces - hard to display it verbatim in the blogger environment!) & # 8 2 1 7 ; or even John X s Blog (where X is some awkward block symbol that can't be adequately described or rendered here. This actually results from the MS quote being "decoded" as Latin-1 - so it's sort of a cascade of errors.) The first of these is known as an "HTML entity" and here is a list of them . ...