[Wiki Loves Monuments] acute problems with the WLX jury tool during WLE-2016

Thu Aug 4 04:12:11 UTC 2016

+ Yuvi

On Wed, Aug 3, 2016 at 3:31 AM, Alexander Tsirlin <altsirlin at gmail.com>
wrote:

> Dear Ilya, Leila, and others,
>
> Thank you for your response. Unfortunately, I am not programmer, and I
> can't help you with the development. I only know that Tools had severe
> problems in the past, and it may not be a very reliable solution because,
> if something happens to the server, you won't have direct control and won't
> be able to solve the problem quickly. I had enough trouble when map scripts
> for Wikivoyage were running there.
>

Yuvi, what kind of guarantee can we have for Tools' availability?

> I have spoken to our jury members, and their request is two-fold. First,
> the jury tool should be fully operational and stable before the contest and
> not 1.5 months after its end. Second, it is highly desirable to start the
> grading already in September, because processing 20,000 photos in 25 days
> would be difficult.
>

Our goal is to have a functioning tool before the start of the contest.
This being said, I only can commit to a fully functioning jury tool
availability by October 1, not before that (again, we will do our best to
have the tool ready earlier, I just can't guarantee it at this point). The
assumption is that countries do not start the jury process until all photos
are in (though I understand that some countries start earlier).

> When I am writing about alternatives, I also mean that the grading can be
> organized with a very simple system of galleries and grading sheets that
> lacks the nice graphical interface, but works reliably and directly on
> Commons. And it perfectly fits our needs mentioned above. After our recent
> WLE experience I seriously think that such a primitive system will save us
> a lot of time and nerves. However, let's wait until the end of August and
> see where we are.
>

Yes, I understand. I can comment more on such a system after I finish this
round of researching all the tools available.

> From my side, I can test the system in the end of August and write my
> comments about its usability. It will also help me to decide whether we use
> WLX jury tool for WLM this year.
>

Perfect. Thank you! :)

Leila

>
> Sincerely,
> Alexander
>
> PS. Thanks for submitting the Phabricator tickets!
>
>
>
> On 03.08.2016 11:38, Ilya Korniyko wrote:
>
> I'll add on
> * Creating next round - sorry for latency. This is already implemented. On July
> 16 as Alexander mentioned himself
> * Tool unavailability
> ** I installed monit service to reload jury tool and mysql when it goes
> down. Will have to wait and see if this helps. Before this I tried upstart
> and systemd configurations that also have respawn option. Unfortunately
> they worked on Ubuntu/Debian/Centos 7, but not on Centos 6 that is on
> WMUA's server, so I have to look for other tool and used monit for this.
> ** Tool is already deployed on Tool Labs -
> <https://tools.wmflabs.org/wlxjury/auth>
> https://tools.wmflabs.org/wlxjury/auth
> * [T141908 <https://phabricator.wikimedia.org/T141908>] Restriction that
> users have contest assigned to them and cannot be added to another - will
> be fixed by end of the week.
> * [T141909 <https://phabricator.wikimedia.org/T141909>] Complete export
> from the jury tool. It is actually available in the form of exporting each
> juror rating or selection separately and overall rating for rating rounds.
> What is suggested is to have overall number of jurors that selected an
> image in the selection round and (Leila's suggestion) one cross table
> with images as rows, jurors as columns and rating/selections as cell. This
> is a trivial change as everything is already loaded in statistics module,
> I'll just to have to output these specific fields. Also will be done by end
> of the week.
>
> Regarding alternate tool. Overall I think it's always good to have
> alternative and understand what we have. I know at least two tools which
> can to be good -
>
>    - Israel jury tool by User:Ynhockey
>    <https://commons.wikimedia.org/wiki/User:Ynhockey> (his own claim)
>    - Germany jury tool -  <http://tools.wmflabs.org/jury/>
>    http://tools.wmflabs.org/jury/ (Germany has lots of images and usually
>    select winners fast, so I hope their tool is good)
>
> Unfortunately I have not seen them, and their sources are not published.
>
> Regarding developing of some completely new tool. I don't think it is a
> serious option. It will be very effort consuming and most certainly will
> not provide a better tool than already existing ones. At least for this
> years WLM.
>
> Regards,
> Ilya
>
> On Wed, Aug 3, 2016 at 9:16 AM, Lily <lilyofthewest.wikimedia at gmail.com>
> wrote:
>
>> Hi Alexander and others, :)
>>
>> *What I did in the past 2+ days*
>> I reviewed your comments carefully and talked with a subset of the WLM
>> international team including Ilya about the issues you have raised. I
>> also reached out to few people outside of the team, to hear their thoughts
>> and experiences, and here I am proposing the next steps.
>>
>> *Next steps*
>> * Ilya will be looking to onboard one person to work with him on the
>> code-base. This person will also create the redundancy that you mentioned
>> in your notes earlier. Ideally, between Ilya and this person, the
>> response time will be "same-day" except for non-critical requests. (If you
>> know Scala or you know Java and you are passionate to get to the Scala
>> world, please contact Ilya, and cc me.)
>>
>> * Ilya will give it a try to move the code-base to Tools this week. This
>> will make sure that the service will have a standard up-time we can count
>> on. There are three tasks associated with this effort: T141910, T124902,
>> T141908. You can find them at https://phabricator.wikimedia.org/project/view/2005/.
>> Ilya and I will review the tasks at the end of this week and re-assess
>> the time they require.
>>
>> * Given the time crunch, Ilya and I discussed pursuing a plan B or an
>> alternative tool that the international team can work on while Ilya is
>> busy fixing bugs and improving WLX in the coming weeks. Based on this
>> conversation, I have started reaching out to the owners of the 5 other jury
>> tools. In the coming days, my attention will be focused on assessing where
>> these tools are at the moment, and figuring out if offering one of them as
>> an alternative makes sense. We also discussed the option of developing
>> another tool. Basically, all options are on the table with plan b at the
>> moment.
>>
>> * I will do my best to send updates to wikilovesmonuments public list
>> about the progress with regards to the jury tool. If you don't hear from me
>> and you want to hear more, please ask on the list, and I will get back to
>> you there. :)
>>
>> *Help needed*
>> We have a short time before the start of the contest, and I want us to
>> feel confident that we will have a solid tool for everyone when we start on
>> Sep. 1. We cannot do this alone though:
>> * If we are to adapt any of the current tools or create a new tool, we
>> are under a time crunch. If you can help with usability testing of the tool
>> we will work on under time constraints, please let me know off-list.
>> * We also may need help for further developing one of the current tools,
>> or creating a new one. If you have time to help with that under the current
>> time constraints, please email me off-list with information about the
>> languages you are comfortable with and maybe a GitHub repository of (a
>> sample of) your past work.
>>
>> And last but not least, I want to assure you that coming up with a
>> solution for the jury tool(s) that we can use more smoothly during this
>> year's WLM contest has a high priority for the international team. You have
>> our commitments here. :)
>>
>> Best,
>> Leila, on behalf of the WLM international team
>>
>> On Sun, Jul 31, 2016 at 7:22 AM, Lily <
>> <lilyofthewest.wikimedia at gmail.com>lilyofthewest.wikimedia at gmail.com>
>> wrote:
>>
>>> Hi Alexander,
>>>
>>> Thank you for sharing your experience. We will review your comments in
>>> the WLM international team and will get back to you by 2016-08-03. In
>>> terms of the timelines, there is a general understanding in the team
>>> that we'd like to have a fully functional jury tool before WLM kicks
>>> off on Sep. 1. Otherwise, we will have many sleepless nights and create
>>> many more of those for others. It's our high priority to avoid that. :)
>>>
>>> Best,
>>> Leila, on behalf of WLM international team
>>>
>>> On Sun, Jul 31, 2016 at 6:55 AM, Alexander Tsirlin <altsirlin at gmail.com>
>>> wrote:
>>>
>>>> We had very bad experience with using the WLX jury tool this year. I
>>>> described it at length here:
>>>> https://commons.wikimedia.org/wiki/Commons_talk:WLX_Jury_Tool
>>>>
>>>> Several possible solutions are also mentioned on that page. I would
>>>> like to hear your opinion about this situation. Could anyone help Ilya with
>>>> developing and maintaining the jury tool, and make sure that it works at it
>>>> should work?
>>>>
>>>> If we are planning to meet a very sharp deadline of October 25 for WLM
>>>> photos, the jury tool should work absolutely smoothly, and user support has
>>>> to be very prompt. Or it would be good to know already now that we can't
>>>> rely on the jury tool, and alternative solutions should be searched for.
>>>>
>>>> Sincerely,
>>>> Alexander
>>>>
>>>> _______________________________________________
>>>> Wiki Loves Monuments mailing list
>>>> WikiLovesMonuments at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
>>>> http://www.wikilovesmonuments.org
>>>
>>>
>>>
>>>
>>> --
>>> User: LilyOfTheWest
>>>
>>
>>
>>
>> --
>> User: LilyOfTheWest
>>
>> _______________________________________________
>> Wiki Loves Monuments mailing list
>> WikiLovesMonuments at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
>> http://www.wikilovesmonuments.org
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "WLX Russia organizers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to wlx-russia+unsubscribe at googlegroups.com.
> To post to this group, send email to wlx-russia at googlegroups.com.
> To view this discussion on the web visit
> <https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDys7geY1wG0dTgxFS9GYpw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> https://groups.google.com/d/msgid/wlx-russia/CAN8OKLZdymV7xTTJj-8X9BaLxticDys7geY1wG0dTgxFS9GYpw%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> _______________________________________________
> Wiki Loves Monuments mailing list
> WikiLovesMonuments at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
> http://www.wikilovesmonuments.org
>

-- 
User: LilyOfTheWest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/wikilovesmonuments/attachments/20160803/b74cd13a/attachment-0001.html>