Why’d I go to the trouble of all this, instead of just using random WordPress plugins? A couple of reasons:

• WordPress plugins tend to do real-time pulls
• WordPress plugins are exceedingly limited when it comes to customizing how data is displayed
• WordPress plugins are, well, WordPress plugins

WordPress Plugins and Real-Time Pulls

The activity stream plugins I’ve found for WordPress generally had one thing in common. Well, they had a lot of things in common, but the first tour-de-farce is this: They tend to rely on real-time access to a third-party service. That is, when a browser hits a page with a Twitter stream, a request is sent to Twitter. Makes sense, no? It provides up-to-the-minute accuracy, right? What could be wrong with that?

So, what happens when, say, Twitter is down? The better plugins allowed for a timeout and a failure message. I actually hesitate to say better – better than what?

Here’s what’ll happen. A browser will hit your site; the page will begin loading, and the user will be treated to a spinning circle of death, or a rotating beach ball of death, or some similar irritant. For how long? That depends. What’s your timeout value? For the pull request? For PHP’s execution? Whatever it is, chances are, it’s too long. We live in a world of instant gratification, and the easiest way to lose visitors is by having slow or partially loading pages. I don’t partificularly care about browsers and page views and traffic on this site, but if you’re going to do something, you might as well do it right.

Even assuming your real-time pull fails gracefully, what then? Do you display your own fail whale? Someone with a clue will say, “Oh, Twitter’s down.” The vast majority of people out there don’t have a clue. A computer is a magic box; the Internets are dark sorceries understood only by the Pope and/or Stephen Hawking. Display an error message, and your browsers will believe it is your site that is broken.

Finally, there’s traffic. What happens when you’re suddenly slammed with over 9000 hits per second? Are your pulls asynchronous, or is Apache going to be left holding connections to individual browsers for no good reason, consuming increasingly massive amounts of RAM and spiking load until a small puddle of molten steel drips from your rack? And are you going to have your API account taken out for abuse?

Direct pulls are stupid.

Granted, your average website about cats isn’t likely to have any of these problems. But they’re so easily avoided in the first place.

WordPress Plugins and Data Customization

The single most important thing to me, and the reason pretty much every WordPress plugin failed, was customization of how data is presented. The average plugin presented data in a fixed manner; if you wanted to change how it was displayed, you’d have to modify the plugin itself. You can go that route, sure – but good luck keeping up with plugin updates after that. And I’d rather maintain my own code than someone else’s.

Of course, this is sometimes a limitation of the third-party service – my Github stream, for example, is fairly h4x, because the atom feed spews broken links and a bunch of crufty markup.

WordPress Plugins and WordPress

Finally, WordPress plugins are designed to work with WordPress. What a shocking revelation, I know. That’s fine for now, really. WordPress presently suits my needs. What if it doesn’t, in the future? The obvious answer is to switch to whatever activity stream capabilities any new platform might have. There’s no guarantee that any given platform has the capabilities I need, however – and I’d have to deal with the first two items on this list all over again.

This certainly wasn’t a determining factor – but since I wasn’t happy with real-time data pulling, or the lack of customized presentation, this issue was solved as a bonus.

Right, then…

So, taking these problems into consideration, and espousing the Unix Way(tm), I wrote a series of small scripts to handle the matter entirely. The scripts are invoked via cron, and can thus provide anything from occasional to near real-time data. The scripts parse data returned via API or feed (in the case of Github), and insert any records not previously seen into a MySQL database table. That – the involvement of SQL – eliminates problems. Which problems? All of the problems.

Since a page load now references the local MySQL database, there’s no risk of users sitting around waiting for the page to load while Last.fm, Github and Twitter are contacted. Those services have already been contacted. There’s no issues of connection failure between the site and third-party services – if a data pull fails, users will merely see out of date information. There’s no risk of API abuse, no matter how much traffic is involved – because the traffic hitting my webserver is not resulting in additional API requests.

In terms of presentation of data, I can do whatever the hell I want. I can select precisely the data I need, and plug it into the markup I desire. There is an exception – as noted, Github and its Atom feed – but even that’s now easier to customize.

Finally, since the data’s pulled into MySQL, it’s available to anything. In my case, I am using my WordPress database for storage, but I can easily have other software reference the database and tables. Or export and import the data into a different database. The data pull will function regardless of what CMS (or lack thereof) I use for the site; the historic data can be transferred to any other software. Hell, with a quick bit of scripting, I could have the entirety of my activity stream scroll by on a USB LED message board…

…You know, I might actually do that. ;)

At any rate, data customization really was my primary drive here – but the other benefits are icing on the cake. Suet in the pudding. Rum in the coke. Bacon in the omelette.

So I’ve finally gotten around to finishing off Gunslinger Girl: Il Teatrino. The first opening, as seen above, is purely epic. It has a nice grounding effect with regard to the ridiculously brutal extreme that GSG took the ‘girls with guns’ theme to – the real-world images merging with the series’ theme is simply delicious. Il Teatrino mainly revolves around Triela, rather than Henrietta. Plot’s fairly decent; the only real hole is how cybernetically enhanced killing machines are capable of having issues dealing with a human assassin. Like the first season, Il Teatrino is filled with plenty of pure WTF moments – this time mostly revolving around the origins of the girls.

All in all a worthy successor to the first season. Then again, I’m a sucker for anything that takes an overly cutesy theme and twists it into something grim and dark. :p

In other news, I’ve gotten around to finishing the conversion from Drupal to WordPress. That was easy(tm). WordPress, frankly, owns the living crap out of Drupal in terms of running a blog. Granted, Drupal can do so much more, but, well – I’m running a blog here. WordPress > Drupal for this use case. The right tool for the job, and all that.

Still have some stuff to work out, mostly revolving around comments and image positioning in posts.