"And all I got was a giant piano player..."

...or the perils of trusting an incompetent genie.

A few weeks ago my attention was brought to the Gender Genie: an online tool for determining whether a text of 500 or more words was written by a man or a woman. Plausible, I thought, if there really is some significant difference in the Zipf distributions of the words that men and women use, or in the order of parts of speech that they prefer. Then I read that the authors claimed 80% accuracy and that has been touted as a serious tool rather than a parlour trick. Hmm - I thought - are they claiming to be able to check whether I get David to write my journal articles?

The authors of the Gender Genie (Koppel and Argamon) used texts from the British National Corpus (listed here). From the [PDF] paper it seems like this is an honest attempt to determine what the differences between male and female writing might be.

However, it's certainly not working for me. When the Gender Genie first came to my attention I was in the middle of writing a conference paper, so I tried it on that. Bad news - on a randomly selected few paragraphs the results were as follows:

Words: 671
(NOTE: The genie works best on texts of more than 500 words.)

Female Score: 368
Male Score: 1058

The Gender Genie thinks the author of this passage is: male!

Ok, fair enough, perhaps a scientific article will throw the Genie off (though the makers don't suggest any such caveat - it is supposed to be a generally applicable tool). Here's what it thinks about my blog post on Jackie Cochrane. It couldn't be more "womany":

Words: 326
(NOTE: The genie works best on texts of more than 500 words.)

Female Score: 80
Male Score: 544

The Gender Genie thinks the author of this passage is: male!

Hmmm. Not according to the Gender Genie then.

What about that über-male, St Paul, and his letter to the Colossians? [1]

Words: 277
(NOTE: The genie works best on texts of more than 500 words.)

Female Score: 732
Male Score: 401

The Gender Genie thinks the author of this passage is: female!

Really? Wow. That's going to require a serious dose of hermeneutics.

I should be clear: I'm not being methodical here, and I'm not attempting a genuine critique of the Gender Genie. I could look more deeply into the methods and work out why both I and St Paul are apparently so hard to place. But just as it takes only one red fleem to disprove the axiom "all fleems are green", my experience calls into question the general applicability of the Gender Genie algorithm.


UPDATE: apparently I (and St Paul) are not the only ones finding that the Gender Genie performs poorly:
Alexander Chancellor in the Guardian reports that all but one of the Guardian's female writers were classified as male. Of 9 journalists, 8 were classified incorrectly. Now if Gender Genie was guessing randomly you'd expect a better hit rate than that.

UPDATE++: The Gender Genie site gives the stats since 2003. It's not impressive, and certainly not up around the 80% mark that the authors claimed in their article.


------------Notes-------------
[1]Colossians 3.12-3.25, NIV. The verses include "Wives, submit to your husbands, as is fitting in the Lord".