… but I do know that the analysis has some … er … flaws. Yeah. Flaws.
I’ll ignore their sample size issue for the moment (though it does go to the size of their error bars … I hope they appreciate the inverse functional relationship between these two).
Take two sets of data with error bars. Put them down on the same graph. The data from each set overlaps within the error bars of the other set.
Are these distinguishable?
No. They aren’t.
Can you draw any meaningful distinction between these data points?
No, you cannot.
Yet, somehow, this is what they did.
Yeah, I know. Lies, damn lies, and statistics. I’ve seen (and corrected/critiqued) my fair share of the same.
They invent a programming language called Randomo. In it, they literally use random characters. They then have novices program in 3 languages, their Quorum, Perl and this Randomo. At the end of the process, the results are scored, analyzed, etc. Charts generated. Error bars drawn.
And thats when they make some pretty classic errors. They take a set of data which is effectively within the error bars of another set of data (its symmetric in this manner actually), and they claim that they can distinguish. Really? Their data, and the analysis say otherwise.
And then, this is where I note the tedious aspect of this … they attack Perl as being, literally, indistinguishable from a line noise language. One they call Randomo. And they do so with a poor interpretation of their data that doesn’t actually support their conclusions.
I took the time to point this out. I was nice about it, I didn’t fisk it, or rip them as others did on this and the other site.
It is kind of sad to see this, as this study, and the concept behind it have potential merit. But they got lost somewhere along the way. Putting out a white paper on this as the beginning of an ongoing study, indicating that preliminary data was available for the first group … yeah, that would have been worth while. This paper … maybe not so much. It looks like a conference submission of some sort.
There are more than a few folks out there who still like to bash on Perl, still think thats cool. You know, its indistinguishable from a line noise language and this paper “proves” it. Thats why this is more dangerous than the usual fanboi tedious arguments. It has something akin to academic heft behind it.
The way I look at it, I am unaware of another language with anything close to Perl’s CPAN library scope/size/variety. R has a far smaller variant (CRAN), there have been passing attempts at this for Python and others. Octaveforge is cool. But the mere fact that there is such a huge resource
The Comprehensive Perl Archive Network (CPAN) currently has 100,828 Perl modules in 23,628 distributions, written by 9,299 authors
… yeah, this rather belies the fundamental point of anyone bashing the language as being indistinguishable from randomly chosen characters. If it were so bad, then why would so many people spend so much time and effort building such useful things from it? Wouldn’t they obviously switch to an easier system? The authors suggest Java (no really, stop laughing) as being easier.
Yeah, I wasn’t impressed with the analysis. They could have done better. And their potshots at Perl, apart from being unwarranted, weren’t supported by their own data. Won’t push them to retract. Did push them to get more data so they can reduce the size of their error bars. And include many more languages in their study.