I was in the audience at the SxSW panel, and when I saw the video of the testing session (Marybeth, told to test all three in a row with the instruction that she couldn’t eat lunch until she finished), I almost had a heart attack. It’s no wonder that Marybeth was confused. The video of their testing session showed that she was logged in to WordPress as a subscriber, rather than as an author, editor or admin. So, the dashboard was completely non-utilitarian in terms of creating content and almost all navigation was hidden, since subscribers can pretty much only edit their profile. I would very much like to see a comparison of the sites with all of them being tested with the same access levels. The first time I logged into a site as a Contributor, I freaked out, too, since almost the entire application is invisible.
The lines of code for WordPress included a custom installer, b/c the WordPress team thought the test subject would be installing the software on a shared host, not testing a pre-installed version. I think Matt said that accounted for about a thousand lines of code. The representative from Joomla also mentioned that there was code not included in that number for them b/c some code was pre-written. It sounded like how lines of code were counted was not standardized.
I thought the idea of the panel was interesting, but wondered about the impartiality of it all. The spec for the site everyone created a version of was created by someone affiliated with Drupal, and may have unintentionally favored “out of the box” Drupal functionality. I think they’re talking about doing another round of testing, addressing some of the issues with the process that was used for the panel test. I’ll be very interested to see results on a new round of testing.