I’m no great fan of Phoronix - sorry, I don’t want to needlessly bash anybody, but seriously, what is the point of their favourite trick: doing two default installs of different distros, then seeing which one runs faster? What possible useful information could you glean from that? (Far be it from me to judge the probably numerous people out there with completely default Linux installations. But are they the ones that are bothered about a 10% performance shift in some obscure compression algorithm?)
Anyway, they seem to finally be making a stab at isolating variables. This could actually produce some interesting data, given time…
Phoromatic Kernel Performance Tracker - Daily Benchmarks Of The Linux Kernel!
What would be more useful perhaps is if as well as the dates, they gave the changing version numbers of the system - but hey, it’s a new test system, so to be given some leeway.
By the sound of things people will be able to set up pages to look at various different combinations on different packages - as an example something like: fixed graphics driver, tracking kernel development; tracking graphics driver, fixed kernel; or tracking both developing - hopefully all on the same page, maybe even overlaying graphs.
Obviously, once you’ve got more than a few packages involved you start to tend towards staring at tea-leaves again. And you’ve also got to find out which people submitting data are actually doing it reliably, and which ones ‘just thought they’d look at a few things on youtube while it was running in the background’. Still, these are technical hurdles - and SETI @ home and such projects seem to have proved that massively distributed experiments on people’s idle computers aren’t completely unfeasible, if you’re clever about how you handle the data. Perhaps, with a few years’ development of things like this, we will usher in an era when computer programmers can change the implementation of a routine, then watch as systems around the world pour their experiences of performance (and stability) in the field back to some central server, which then works out the common factors between them.
But still, in the interim, credit where credit is due. This is, as best as I can tell, a genuine scientific experiment. Well done Phoronix. 