Test

test

Blogged with the Flock Browser

A remedy for GooTube. Also cures gout, improves temperament, and de-angrifies the blood.

There’s been much ado of late about Google‘s acquisition of YouTube. Many are referring to YouTube now as GooTube, and it’s clear from some videos on YouTube that some folk are pretty nervous about the acquisition, fearing that the video service will become a pay-to-play service or that Google will find some other way to ruin YouTube. My own attention lately has been directed largely to the preparation of my home for sale, and as part of those preparations, I’ve purchased a product, pictured here, that may be of interest to those worried by this new business partnership. I present to you “Goo Gone,” which in my experience is better at killing ants than it is at anything else. Apply to GooTube and rub vigorously. Results may vary.

Caching, Grouping, and Additive Filtering

I’ve wiled away the last few months of my work life doing statistics. For example, I try as best as I can to figure out how many people have clicked to our download page, how many initiated a download, how many downloads successfully finished, and how many people are actually using the browser. It’s a lot trickier to figure this stuff out than you’d think, though I won’t bore you with the details.

I plan to bore you with different details. As you begin to collect more and more data, providing live statistics becomes challenging. Say you’ve got a database table with a few million rows of data in it, and somebody wants to pull a query from that database for a given date range and get a report that requires a lot of calculations. Suddenly, you not only have a query that has to comb through a lot of data, but once you’ve finally got all the data, you have to iterate over it to perform calculations. In our case, a couple of months ago, we found that we had enough data being churned that an overview report was taking 20 seconds or so to run. I spent a lot of time tweaking indices and optimizing queries, and I managed to save a lot of time, but I knew that as our data grew, the reports would bog down again. The solution, I figured, lay in caching data for speedy retrieval.

We typically care (at least for now) about daily data. This means that I can occasionally run the big heavy queries, group the data by date, and store data in such a way that I can simply add up counts for a date range to get very responsive reports. So for example, a query like this from a huge table of log entries:

SELECT date, count(*) FROM data WHERE date BETWEEN '2006-09-01 00:00:00' AND '2006-09-03 23:59:59' GROUP BY date

might yield a (partial) result like this:

2006-09-01       23
2006-09-02       28
2006-09-03       27

I stuff those values into a caching table. Then, when somebody asks for a total count of some piece of data for the date range 09/01 – 09/03, I can do a lightning-fast query from the caching table and add up the values in the count column to return the result (quite possibly using mysql’s SUM function to eliminate the need for any post-processing of the data). This gets a little dicey when you start having to calculate averages (for just one example) or when you start selecting based on more criteria than just date. This last selection type is something else I’ve handled in what I think/hope is a pretty nifty way that I call, for lack of any more fitting term I’m familiar with, additive filtering.

So, the scenario. Imagine that you want to allow users of your reports to filter on various criteria. In our case, we provide a standard version of Flock and a Photobucket version. We also provide versions for different platforms for both Flock and Photobucket. And for Flock releases, we provide builds for multiple platforms for multiple languages. The simple count-per-date caching I used as an example above doesn’t allow us to filter the data on these criteria efficiently, so we need another mechanism. Enter more sophisticated grouping in our caching tables. The query of the huge table we used before becomes something more like this:

SELECT date, vendor, os, locale, count(*) FROM data WHERE date BETWEEN '2006-09-01 00:00:00' AND '2006-09-03 23:59:59' GROUP BY date, vendor, os, locale

And our results more like this:

2006-09-01	flock	win	en_US	12
2006-09-01	flock	mac	en_US	 4
2006-09-01	flock	linux	en_US	 1
2006-09-01	flock	win	zn_CH	12
2006-09-01	flock	mac	zn_CH	 4
2006-09-01	flock	linux	zn_CH	 1
2006-09-01	pb	win	en_US	12
2006-09-01	pb	mac	en_US	 4

Note that I’m showing just one sample day here (and by the way, these aren’t real numbers, so don’t try to extrapolate and announce a download count for Flock). A few times a day, I put this data into a table with a unique key comprising all columns but the count, and each time I run the big query, I replace the values. For nearly-live reporting, I can then run queries summing the counts and adding constraints (ahem, “additive filtering”) on any of the columns desired. So if I want a total count, I just do “SELECT SUM(count_col) FROM cache_table” for the requested date range, and if I want a Photobucket count, I do “SELECT SUM(count_col) FROM cache_table WHERE vendor = ‘pb'” for the date range. For any further narrowing I wish to do, I just add another constraint to my WHERE clause. Again, it gets a little tricky if you’re doing any sort of averaging without constraints (you can’t just do a sum in that case but have to calculate a weighted average per would-be constraint set), but by and large, this has so far been a pretty good quick and dirty way of doing caching to generate speedy reports.

There are no doubt better ways to handle caching of large data sets, but for our purposes, this little trick has proven effective. I imagine about 1% of my already limited readership probably found this even remotely interesting. Still, I occasionally wonder if it might be appealing to some to have a glimpse into some of the less glamorous things that go on behind the scenes in a project like Flock. So, what do you think is better? Turkeys and neat little discoveries or things like this post?

Flocked Minis

While waiting in line at a craft store today, I saw the item pictured here in a display by the cash register. Next to it were some other “Flocked Minis” of Halloween figures (ghosts and pumpkins), but for my purchasing power, the birds were it. (There were also some little bears that I almost always think about getting for my daughter when I see them, but they didn’t come in the tangentially-Flock-relevant packaging, so I opted out of those once again.)

Lest you think this is an ill-advised product placement, let me explain that flocking is (among other things) a technique of texturing or patterning something with pulverized wool or felt (source). These turkey figurines are flocked. I wonder if turkeys actually flock in the wild.

I’m no good with version numbers, but I hereby propose that whatever version number we’ve reached when we’re down to the letter “T” in our count (our already having used Buzzard and Cardinal and heading into Danphe) that we name the browser Turkey.

My daughter enjoyed playing with the turkeys.

Flock in Print

I picked up Cal Henderson’s Building Scalable Web Sites the other night hoping to learn something about “The Flickr Way” of building scalable web sites. I’m about halfway through and have mixed feelings so far about its usefulness for my purposes, but I couldn’t help feeling a little thrill when I encountered a reference to Flock on page 142. This portion of the book gives a brief explanation of the http request/response life cycle, and Cal happened to be using Flock while generating his example request (which returns a 404, if you’re interested, though the example suggests otherwise 😉 ).

I suppose I shouldn’t have been terribly surprised to see a Flock reference in a book by a guy from Web 2.0 darling Flickr. That Lloyd and Cal are buddies (there are photos to prove it) and that Cal has been spotted at SuperHappyDevHouses (the creation and proliferation of which past Flockers Termie and Factoryjoe have been instrumental in) should have rendered it even less of a surprise. Still, when I saw the example, I flopped the book down and exclaimed to my wife, “Hey, Flock’s in this book! Woo hoo!”

Timed disabling of WordPress comments

I wrote the other night about my frustration with comments in WordPress. The anti-spam tools that come with the software are great for squashing spam, but I still occasionally go through periods during which I get a bunch of emails asking me to log in and moderate spam, and that irritates me. Most of these moderation requests are for spam on old posts. So I decided to write a WordPress plugin that would auto-disable comments and pings on old posts. This allows me to keep comments open for those who read my stuff and would like to comment within a reasonable timeframe but keeps me from having to go back and manually run mysql queries by hand (or, worse yet, manually edit old posts) to turn off comments on individual posts. (The WordPress options for commenting affect only future posts.)

The plugin is a simple one, providing an option under the “Manage” and “Manage Comments” administrative screens allowing you to set the number of days old a post must be to have its comments disabled. If you post infrequently, this plugin won’t do you much good, as its action is triggered by the saving of a post. If you post every few days as I tend to, though, it’ll do an ok job of keeping old posts from picking up comment spam. The default threshold is 14 days. So any time I add or save a post, WordPress runs a query that disables comments and pings on all posts created more than 14 days ago. That’s all there is to it.

Download the plugin here if you think you’d find it useful.

Late-night confessions

From a late-night IRC conversation with a Flock staffer and a Flock community member. It’s sort of a running joke that, a Southerner, I partake of all the bad habits and am characterized by all the provincialisms generally associated with the South.

daryl: (sorry, it’s late here and I’m working on a blog post entitled “My complex relationship with meat,” so it’s fitting that I should be in a weird mindframe 😉
yosh: mmmmmmmeat
daryl: yosh, I’m occasionally eating meat again
daryl: we ran out of vegetables in Tennessee
daryl: except for tobacco, that is, and it tastes really bad
yosh: well, TN really didn’t have that many
yosh: daryl: tobacco can be good with the right sauce
daryl: like a durian sauce?
yosh: durian-natto sauce
daryl: heh
daryl: actually, I’m mainly eating meat b/c my newly pregnant wife craves it and I’m tired of cooking two meals a night
yosh: heh
[redacted]: daryl: you actually eat tabacco?
daryl: [redacted], it’s a staple in Tennessee
daryl: that and buggering cows 😉
daryl: (no, I don’t eat tobacco)
daryl: (though I am married to my sister, who is also my grandmother and my third cousin six times removed; and my father)
yosh: daryl is his own grandpa
daryl: and grandma
daryl: I’m also my own sandwich
daryl: (my other grandpa having mated with a tobacco plant, that is)
[redacted]: daryl: good (you don’t eat tobacco) 😉
daryl: opium, now that’s a different story 😉

Knoxville Blogger’s Meetup Post Mortem

Tonight, I attended the blogger meetup that Mike organized. Counting Mike’s brother and girlfriend (both of whom were mostly absent but were warm bodies in occasional attendance, so I’ll count them provisionally), there were eight of us, all tied to Mike through past or current jobs or the aforementioned relationships. One guy was an apparently random acquaintance of Mike’s who since connecting with him has gotten a job at Mike’s place of work through no help from Mike (more or less at random, that is). Weird. Anyway, so we had a pretty decent crowd for a first meetup, though I hope that for future meetings, we can expand our network a bit and bring in some new folk. (Which let me say is a strange thing for me to hope because I’m generally pretty reclusive and not interested in adding more people to the list of those I feel obligated to remember or communicate with. Let’s keep that our little secret.)

We had discussion of three items on the agenda: photo manipulation/hosting tools, Flock as the blogger’s browser, and video blogging.

First, Mike gave an overview of Google’s Picassa, which includes a desktop client for photo manipulation and a (beta) web service for display of photos in albums. The client looks pretty nice, with basic and pretty easy-to-use tools for manipulating photos. Want to bring out highlights in a picture? Just hit the highlights button and adjust the levels (or something like that). It seems like a great lightweight tool for doing the sort of basic operations that those of us who’re intimidated by Photosho or The Gimp are likely to want to do. As for uploading and displaying, there are some weak points. You can’t upload one-off photos, for example — they all have to go into an album. For most users, this is probably fine, but it’s nice to be able to post a one-off screen shot as well. The web display itself seems pretty weak. There’s a concept of favorites or friends, and there are settings to be able to publicly list albums, but there seems to be no interface for searching for photos or friends. If I happen to know Mike’s gmail username and know the base url for the photo service, I can assemble a url that will show me his albums, but that’s not user-friendly to say the least. I imagine the service will be expanded to fix these problems. All of this is basically moot for me, as the client isn’t available for linux, but it was interesting nevertheless.

Next, Mike gave a demonstration of Flickr’s capabilities. I hadn’t expected to learn much here, but he showed me some things I hadn’t looked into before, most of the details of which I’ve forgotten by now but will delve back into as need arises. There’s more you can do with Flickr’s organizer tool than I had ever discovered; I had always thought of it as merely a way to organize sets, but you can batch add tags, set privacy, etc., and with some of the gaps in Flock’s ability to batch edit photos, these things are very useful. There are also some neat views of photos and tags that I hadn’t looked into. My impression of Flickr in recent months has been a better one than previously. Things seem more discoverable since some features were added to the site. Between that and Mike’s demo, I may find myself actually using the site more than I’ve been accustomed to doing in the past.

At this point in the evening, after a late start, we’re more than an hour into the evening and it’s my turn to present. Mid-presentation (baby’s bed-time), I get a call from my wife that I dismiss. The moral of the story is that we need to plan less stuff for these meetings and trust the power of gab to carry us through to a sufficiently lengthy time. I thought at this point about suggesting that we push either Perry’s vlogging segment or my segment off to a future session, but I wimped out, not wanting to hijack the meeting.

As I result, I rushed through my segment, for which I’m a little embarrassed to admit I wasn’t terribly well prepared. It was clear that Mike had spent some time thinking about what he was going to demo. I had run through doing a blog post in Flock’s editor but hadn’t really scripted anything out, and I think that between that and my being in a rush, I probably did a pretty poor job of showcasing Flock’s capabilities. We’ll be releasing a new version of the software before too long, and there’ll be big changes then, so perhaps I can get some more time then and do a better job. The 30-second version of my presentation is roughly as follows: Hey, there’s an html rich editor, so you don’t have to code html anymore. There’s also this little shelf thing at the bottom that you can drag pictures and text into and then back out of to construct rich blog posts. And there’s this photobar that shows your Flickr photos (and those of others) for easy dragging into blog posts. And you can easily drag/drop upload photos straight from within your browser and get notifications when your friends post their photos. (Not covered in my presentation but important is the fact that this uploader tool works in linux and thus has caused me to upgrade to a pro Flickr account and actually bother to snap photos.)

Now Perry stepped up to the plate to talk about screencasting. A screencast is basically a movie of somebody’s desktop as they use software and explain the process. He reviewed several tools that I was interested in seeing but that were sort of dead ends for me because they can’t be used on linux. (Side note: A week or two ago, I briefly evaluated something called xvidcap for linux; after hacking the config so that it would compile on my system, I wound up finding the software difficult to use, but probably about as good as it gets for this type of software on linux.) In one case, Perry used a piece of screencasting software to do a screencast of the software itself. (It was during this window that my future self came back to visit me and prevented my future untimely demise by suggesting that I take an alternate route home; it was strange.) I found myself thinking during Perry’s presentation that the perfect synthesis of our evening would be Perry’s doing a screencast of Picassa and posting it to his blog using Flock. (My future self had nothing to say about whether this would actually happen, though I did press for an answer.) (Ahem. It’s late and I’m tired.)

After Perry’s talk, we briefly discussed finding a mechanism for publishing events. Orkut (which several of us had signed up for) sucks for this sort of thing, and who wants to pay meetup.com for this? We discussed using Gmail’s calendar, which we should be able to syndicate for publication on a web site if we ever build one. We also discussed finding a plugin for WordPress and just having a blog site. I think we finally concluded to not worry too much about web infrastructure until the core group’s a little more established and we have any hope of attracting a broader audience.

And so concluded our meetup. I think Mike briefly proposed discussing next time (probably a month or so from now) some of the options for hosting your blog. All in all, it was a good meeting, and I’m frankly a little surprised to report (see note above about my being nearly pathologically anti-social) that I look forward to the next one.

Knoxville Blogger’s Meetup

A couple of weeks ago, I was in the driver’s seat for a Flock meetup here in Knoxville. My pal Mike has since begun organizing a Knoxville bloggers meetup to discuss blog tools more generally. Details (straight from Mike’s blog):

When: Wednesday, August 23rd 7:00pm
Where: Mike’s Place (directions when you RSVP)
What: Knoxville’s local bloggers get together to talk about tools and services that help bloggers.
Who: Anyone who has a blog, wants a blog, or wants to learn about blogs.
Why: We don’t need a why!

You can RSVP to me or go over to Mike’s site to RSVP. Mike’ll give a little presentation on some image hosting/manipulation tools, I’ll give a half-assed demonstration of some of Flock’s features, and Perry will talk a bit about video blogging tools. From there, we’ll talk amongst ourselves about the tools presented and figure out where to go from here with the meetup group.

If you’re already a blogger or are just curious about what all this blogging stuff is all about, this first meetup should provide a good introduction to some of the tools and services available to you.

Down (err, Up?) with Bottom Bars

Flock’s usability guru, Will Tschumy, posted the other day about topbars and how useful they are. Just last week, I was thinking about the usefulness of topbars and was going to do a blog post, but other priorities slid in front of it, and I put the post aside. Since one of Will’s proposals is to use bottombars instead of topbars and I’m wildly, passionately opposed to that, I thought I should go ahead and post my feedback.

First, let me take a moment to revel in the past. Once upon a time, we had a thing called the shelf that lived in a happy little sidebar. This was way before Flock was even Flock. (Disclosure: I wrote that version of the shelf and so may have an overzealous fondness for it, though I’ll say right now that many of the things in the current version of the shelf are hands-down better than in the first version; it’s just the user interaction that I find unpalatable.) We later moved the shelf into a popup window, and then into a topbar and ultimately into the bottombar that it currently occupies. Although I like the core functionality of the shelf (seriously, I gushed here about how it helped me to blog more and be more productive), I find it unusable because it’s a bottombar. There are two key issues that make its being a bottombar a big problem for me.

1. It doesn’t display horizontal content in a useful way. Only a few items can fit in the bottombar, and it’s difficult for me to differentiate among them if they’re mostly text. As my habit had been to use the shelf (web snippets, whatever) primarily as a scratchpad for blog post ideas, it became unwieldy for me to stick blurbs — even short ones — down in the bottombar with any hope of being able to see at a glance what they were about (I talk about this in more detail here). Earlier versions of the shelf allowed me to preview more of the text at a glance (two or three lines), which made for a more efficient workflow. A vertical orientation of items (as in a sidebar) with the potential for horizontal display of text makes more sense for anything into which text can be dragged.
2. I use a laptop, and bottombars are physically painful to use. My hand and wrist muscles are accustomed to the motions associated with darting upward on my trackpad to press application buttons and menus. I do this casually thousands of times a day with no pain. Tracking downward to a small area at the bottom of the screen seems to require different muscles, a greater rigidity of my fingers. It’s difficult for me, and it sort of hurts me to do it. It makes my wrist ache. So moving many of the things I like about Flock into a bottombar will cumulatively cause me enough pain to disincline me to use the browser. Which is a real shame, because I’ve decided I like our photo functionality enough that I’ve started taking more pictures and have signed up for a Flickr pro account. (Also, as an employee, I sort of have to use the browser, so Flock’ll have to invest in some wrist braces for me if we switch to a bottombar-exclusive UI.)

Add to these things the fact that moving topbars to the bottom doesn’t solve the problems Will enumerated with topbars:

1. Notifications of new photos
2.  Being able to see a group of photos (and not just public photos on Flickr or Photobucket)

and I’m dead-set against the move.

So, now we know that bottombars don’t appeal to me, and in the post I never finished the other day, I was going to propose that even topbars weren’t terribly useful. When Flock was starting out, one of our goals was to mix up the browsing experience a bit, and that meant experimenting with things like topbars. They seemed neat at first, but we found some cases (e.g. the shelf) for which they weren’t ideal (well, that’s my opinion, at least), and I’ve begun to wonder if the topbar (and the bottombar) isn’t something we’ve hung onto for the sake of its invention rather than for its actual usefulness in real life.

I’ll admit that for photo viewing, the topbar works pretty nicely because with photos, there tends to be a sense of graphical chronology. The topbar shows photos in what amounts to a timeline, and it’s really a good fit. Arguably, showing avatars of people in a topbar can make sense if you’re showing them either in order of recently-added content or of addition to your people list. And showing thumbnails of videos or saved locations for a mapping program might work fairly well in a horizontal view.

On the other hand, chronology display isn’t limited to the X-axis. Blogs display their most recent content at the top of the page, and nobody’s confused by that. So a sidebar listing your people in descending order of content produced or of addition to your queue stands to work as well as a topbar or bottombar listing. Further, with a sidebar, there’s room for a thumbnail and a narrow column of meta information or for meta information in a couple of lines underneath the thumbnail. Or, if your gripe is that a sidebar can’t show as many photo thumbnails as a horizontal one, there’s always the option of allowing a two-column view within the sidebar (an option we already allow to be toggled in the news reader). Further, sidebars are a user interaction paradigm that everybody understands. If you read email, you’re comfortable using sidebars.

In reading feedback to Will’s post, I see that opinions are pretty mixed. Some love the idea of moving topbars to the bottom. Others think it’d be awful. Still others want photos on the bottom and snippets on the side so that both are accessible at once. I’m leaning toward a sidebar view of everything with optional horizontal displays of some types of info.

Flock, we’ve often said, is about choice. We let you choose Flickr or Photobucket, Shadows or Delicious, whatever blog platform you want that uses one of several standards. It can be bad to offer too much choice (you know the old sketch comedy gag wherein the waiter unfolds tiered set of choices upon tiered set of choices for the diner who simply wants a glass of water or a cup of coffee). Providing too many ways to access your people-centric content stands to make it very hard to create discoverable features and to support them. Applications like Photoshop and The Gimp can be confusing with all
their palettes, and we should avoid that sort of experience for simple
web browsing and basic people management.

Still, I wish there were a way to let users define their experience to a degree, perhaps with sensible default placements of content views but with the option to move them around and dock/pin them as you see fit. I think I would keep photos anchored at the top (toggleable with an access key, of course) and snippets on the side (where I’d actually start using them again to jot down quick notes). A view of people I would probably also keep on the side because I’m accustomed to my vertical GAIM listing of people.

What kind of workspace would you set up? Will, is anything akin to what I’ve proposed even remotely possible if it seems to appeal to people?