[Wiki Loves Monuments] Plotonides Jury tools

[Peter Ekman]

I'll suggest that we won't effect much of anything this year with this
conversation, so let's leave it for after the contest is over.  I'll
ask everybody to keep a journal on their jury procedures this year so
we can improve them next year.

A couple of issues with Plotonides comments first:
Nobody is gaming the system by self-nominating - that was built in as
a compromise in the beginning.  I'd take it out next year.

The "positive version" of Kaldari's tool would output the file names
of favorite pix to a special sandbox page for further processing.  The
only problems that I have with Nuno's tool is that I don't know how to
output data, or how to scale it up for general use.  That's because
I'm not a coder, not because of any fault in the software.

Anonymity in the screening process is probably necessary. Without some
anonymity things might get very tricky and emotional.  I'll readily
admit that I've made some mistakes in screening - if you go through
1,000s of pix fairly quickly how could there not be mistakes? and I
think most people will understand this. Except when somebody puts
forth his special "baby" to the public ...

I'll admit to 2 types of mistakes - Type I (the most important)
removing pix that shouldn't have been removed at this stage. I'll
admit to a 1% error rate here, and if you look at the numbers - the
implications are truly horrendous.  Type II errors - leaving in pix
that shouldn't have been left in - I'm not really very concerned about
. I'll admit to a 10% error rate here.

If somebody were to complain about his baby being removed when it
shouldn't have been, and then compares it to some Type II errors - if
I were publicly identified, all I could do is give the seemingly lame
excuse "everybody makes mistakes"  But of course everybody does.

I think with "crowd sourcing" or just crowd voting, we have the chance
to make this contest something very, very special.  Everything that's
put in front of the final jury may not be perfect, but at the very
least, it could be "crowd pleasing."
A voting system for masses of photos - I've come to believe - is a
very hard technical project.  Perhaps, if we can define exactly what
we want, we could get the Foundation to work on this for us before
next year.

I'll suggest a 3 stage general process.
1. Review for obvious technical problems (probably 2 strikes and
you're out, rather than 1)
2. Crowd voting to get down to a limited number of crowd pleasing photos, and
3. A final round of judging by highly qualified judges.  After all
photography is an art, and experts must play a role.

Hope this helps (for next year)
Pete


Message: 8
Date: Sat, 15 Sep 2012 00:25:14 +0200
From: Platonides <platonides at gmail.com>
To: Wiki Loves Monuments Photograph Competition
        <wikilovesmonuments at lists.wikimedia.org>
Subject: Re: [Wiki Loves Monuments] jury tools
Message-ID: <5053AECA.5090809 at gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On 13/09/12 05:05, Peter Ekman wrote:
> Kaldari came up with a tool for WLM-US which works quite well IMHO as
> a part of the jury process, but is not the whole answer.(...)
>
> The logic is that you can include a photo for further consideration by
> clicking "Nominate", or you can eliminate it from further
> consideration by clicking "Decline for nomination" and when you can't
> decide click "can't decide."  It works by manipulating categories on
> the photo's file "Category:WLM-US 2012 unreviewed"  to start off, then
> "reviewed" and/or "nominated"  We're aiming to eliminate 90% at this
> point - keeping up as the photos come in.  But in reality we're
> keeping 14%.
> I think one of the quirks is when folks manually change categories -
> it can just stay in the unreviewed category forever.  The fix here is
> just manually changing the categories to the correct state.
>
> It's a tough system (but quick) "one strike and you're out."  But it
> is needed if you have several thousand photos to go through and want
> to get down to a few hundred in a reasonable time so that you can
> properly consider the best ones.

One drawback of this is accountability. There's no trace of who voted a
picture through the app (categories are changed by the bot).

Another problem is that authors are adding themselves the categories
nominating the images they just uploaded:
https://commons.wikimedia.org/w/index.php?title=File:Comac_Church.jpg&diff=78180783&oldid=78180598
https://commons.wikimedia.org/w/index.php?title=File%3ADramatic_Obelisk.jpg&diff=78175836&oldid=78173883
https://commons.wikimedia.org/w/index.php?title=File:Protestant_Children%27s_Home_Sept_2012_05.jpg&action=edit&oldid=77720478
https://commons.wikimedia.org/w/index.php?title=File:Bellows_House_Missoula,_Montana.jpg&action=edit

(these are all different users tricking the system, random images
provided by the tool)


> I'd love it if there could be a positive version of this - say click
> keep and the bot places the file name in your sandbox - but the
> problem there would be you don't eliminate anything and the possible
> selection just keeps on growing.

You mean something like Nuno tool?