Even more slogs

So as I alluded to last week, my friend over here gave me idea. You guys are all writing slogs of your own, supposedly about things that might be interesting to me, but as he/she said, "what about when I ... want to find out what others have said [about a specific topic]?"

Well, the answer of course, is to write another script. I mean, I already know how to get python to read page-source. And I already have your slog addresses, all of them. So what am I waiting for? Break out the urllib!

slogdict = {}

slogfile = open("Slogcontent.txt", "w")

for slog in urls:

    print(slog)

    try:

        x = urllib.request.urlopen(slog)

        lines = x.readlines()

        lines = [str(i) for i in lines]

        slogdict[slog] = lines

    except:

        print("FAILED: {}".format(slog))

slogfile.write(str(slogdict))

slogfile.close()

Yeah. Well, now I have all of your slogs.

Well, almost. There are a few of you whose slogs don't seem to work. If so, I don't have yours (but you might not either, so I guess we're even).

I'm also missing some content, for those of you who have blogs for which not all content is visible from the first page, for example, this beautiful creation. Oh well.

Anyways, I've now created this dictionary, one that associates everyone's slog with its contents, and written it to a file. I can read this file, get the string, and then convert it back into dictionary! Right? Right? Well, look at this simple example:

>>> x

'{1:11,2:222,3:3333}'

>>> dict(x)

Traceback (most recent call last):

  File "<pyshell#8>", line 1, in <module>

    dict(x)

ValueError: dictionary update sequence element #0 has length 1; 2 is required

Oh.

So, google being my friend, I went to find the fastest solution I could.

Here is what I found:

>>> from ast import literal_eval as ale

>>> ale(x)

{1: 11, 2: 222, 3: 3333}

As usual, there's probably a better way, but hey, it works. And as far as I'm concerned, there are harder ways.

So now I've got the dictionaries. Now I clump them together, and make a dictionary of word frequencies, like we did in 108. Except harder, because I'm trying to remove things that aren't actually content, all the "WebFontConfig" and "<title/>" and stuff. And while I managed to parse through it, I don't think I did a particularly good job. Some of it had to be done at the end, simply brute-force removing words like "Powered", "By","Solid" and "Normal". Ah well. I won't be sharing it (unless of course you ask nicely, in which case I'd be happy).

One thing I've done is to implement a simple search engine. I type in a string, and a list of all of the blogs which contain that string are returned to me. Cool, eh? All this power! I wonder what I should do with it...

Well, I now present the result of automated frequency analysis, the answer to the question, "What is it that my classmates are talking about?" Here is the list of, from most- to least-used, the top fifty most-blogged words:

['THE', 'TO', 'A', 'OF', 'I', 'IN', 'IS', 'THAT', 'IT', 'THIS', 'WE', 'FOR', 'WAS', 'AS', 'WITH', 'MY', 'BE', 'ON', 'CAN', 'HAVE', 'BUT', 'ARE', 'RECURSION', 'YOU', 'AN', 'ABOUT', 'CLASS', 'MORE', 'SO', 'WHICH', 'HOW', 'PROGRAMMING', 'WHEN', 'WHAT', 'FROM', 'ONE', 'AT', 'WILL', 'CODE', 'ME', 'FIRST', 'OR', 'ALL', 'SOME', 'WEEK', 'WOULD', 'RECURSIVE', 'LIKE', 'OBJECT', 'SLOG']

Hm. I don't know about you guys, but I found that unenlightening. Maybe I need a better question...

Life's making me a little busy (this code that I talked about here? I wrote it back during reading week). I'm taking a week off from random slogs. Until next week, I really am looking for questions to answer with my search engine, so if you have some, just give me a shout.

--David