Thursday, January 22, 2009

How To Properly Review an Operating System


I read too many OS reviews: Win7, OS X, Ubuntu, Mandriva, Fedora, PCBSD, etc. The list is practically endless, and since I like operating systems, I keep reading. Most of the reviews are really quite poor, however -- the reviews end up being almost useless because the method is so screwed up and the critiques so opinionated that the review is only valid for the reviewer. The comments on these reviews are generally flames disagreeing with the author's viewpoint.

I'm going to highlight some of the major mistakes made and recommend how to make your next review bulletproof.

1. Geek Cred vs. Open Mind

Probably the most common type of review right now has a title like "I dropped Windows and used Linux for a week." The reviewer is almost always a Windows power-user who writes for a semi-major tech blog. He has a lot of Windows knowledge and normally writes as an expert on his subject. He's used to knowing what he's talking about and rarely has a task he can't complete.

When we step into a new arena, we need to leave our pride behind. Hubris doesn't help us review something we are unfamiliar with. An expert car mechanic needs to take his self-confidence down a notch when he starts training on jet engines for airplanes, wouldn't you agree? Reviewers need to realize that they are experts no longer and not to take the frustration out on an operating system just because it doesn't work exactly the way the one they've been learning for years does.

This holds true whether the transition is Windows XP to Ubuntu 8.10, Solaris to Windows Vista, or Windows XP to Windows 7. There is, of course, some room for the "Differences from $MyOldOS" section, but the majority of the review should be impartial.

2. Clicky Clicky vs. Research

Perhaps because of this power-user mentality, reviewers also seem to think that they should be able to understand everything about the system, even deep architectural details, without doing a lick of research on the new OS. Clicking around to try to find a solution may work when you know what you're doing, but it took you years to get to that point on your old OS. Clicking in new surroundings is foolhardy. Nothing works that easily. Anything with any complexity needs a manual, even your MP3 player. Read that manual before you start your review, eh?

Go on-line and read an overview and "Get started" guide from the producer. Buy a book and spend two hours reading before you dive in. Do something. Don't, however, do the computer equivalent of walking into a foreign country without having learned any of the customs or language and expect everything to be laid at your feet. You're just being a bad OS tourist.

3. Hardware Compatibility

"I booted only to find that my wireless headset and my pro sound board didn't work!" Did you look at the hardware compatibility lists before you started?

Make sure your hardware is supported. Sure, you can comment in your review about how short the compatibility list is, but review the operating system with supported components so that it works the way it was intended to. Don't piece together a Hackintosh then complain that this OS X frenzy is all BS because it didn't work for you. Borrow hardware from a friend if you need to.

4. Surface vs. Depth

The installation is not the operating system. You do not need to spend five of your ten pages detailing the installation unless that is the process you are reviewing. You do not need to spend the other five on how you failed to get WoW to work.

Instead, have a list of common activities or tasks you want to evaluate. Go on Facebook. Watch a movie. Listen to some music. Use a popular (but supported) portable music player. Print to a (supported) printer. Try to re-frame your normal publishing work-flow into the new operating system. Install an application. The list could be endless. Make the review useful for more than just yourself by including common tasks for your readers. If your readers are all DJs, those tasks will probably be completely different than mine. Make the target audience clear so that Digg boys don't flame you in the comments.

5. Realistic Expectations

Take a look at how the producer describes the OS and what niche it's supposed to fill. Is the system a dedicated firewall, like IPCop or Smoothwall? If so, judge it according to its firewall ability and configurability, not its desktop appearance. Is it touted as an audio / video production system the way Ubuntu Studio is? Then expect it to perform those tasks well but be lacking in some other areas.

Unless the OS is claimed (by the maker, not fans) to be a drop-in replacement for your old OS, don't judge it on that criteria.




It's amazing how many of these idiots go in without doing any research whatsoever, click around stupidly, and then complain that it's badly designed.

The prospect they may even get paid to write this droll is saddening.


I'm bookmarking this. I'm bookmarking it and linking people to it when they say stupid things where applicable. Thank you for writing this so I don't have to; it cuts to the heart of the matter much faster than I would have (I'm ever poor at brevity. You might be able to tell... ;) ).


Thanks to both of you for your supportive comments.


I purchased a mailbox and a post to put the mailbox in. The post consists of a V-shaped piece of metal about three feet long and a piece of wood about two feet long, that you hammer into the ground, and it includes an installation manual! (It's printed on the box, but the same point applies.) The mailbox is a little more complicated because you have to bolt it to the post, but it also includes a manual. And if you've never done a mailbox install the instructions are important. Now if an almost no-brainer task like installing a mailbox post and a mailbox have instruction manuals - and they are necessary - how can we expect perfect understanding of a software package that is potentially a thousand times more complicated than pounding a stump?

WhyHelloThere said...

I like this article. I just ordered a free version. Does anyone know if Ubuntu is as resource consuming as Vista? I really hope it isn't. Vista has too many useless features I don't use that only take up resource.


My "pull a number out of my rectum and claim it's gospel" feature claims that Vista uses about ten times as much resources as Ubuntu. Seriously now, Linux systems will run the same capability as anything windows does as far as how the User Interface looks on equipment that is anywhere from 1/2 or less as powerful as what is required for Windows. People have gotten reasonable XWindows performance on 486 machines. If it will run at least Windows 98 it will probably run Ubuntu.

WhyHelloThere said...

Thanks, Paul. =]

Post a Comment

Other I' Been to Ubuntu Stories