Team moderation: let's learn together from the Friday 13 team
C
Claude Almansi
started a topic
over 12 years ago
Update, June 14 2013: the links in this post don't work anymore, but the lessons to be drawn about the risks of developers beta-testing innovations on their own, without involving users, still obtain.
Dear Amara developers
First of all, thank you so much for having lifted the moderation imposed in June against the settings of teams that don't want it and in some cases, can't have it because they work on videos that are not theirs. So whether that imposition was deliberate or the result of a mistake is now immaterial, fortunately: could you just remove its remaining traces, please? See my comment to Darren Bridenbeck's July 3rd Amara Update post, about still blocked subs.
Experimenting in the developers' mirror site
While this imposed moderation was going on, though, I kept thinking: why don't the developers try that in the https://dev.universalsubtitles.org mirror site that's meant for that kind of experimenting, and let us subtitle in peace in the main http://www.universalsubtitles.org site? I kind of remember I even wrote that, perhaps in stronger terms, both in this help forum and in e-mails to Dean Jensen and Darren Bridenbeck, even volunteered to participate as guinea pig in such experimentation.
I should have checked better: there are several teams in that mirror site, particularly teams named by dates starting from December 2011, where you apparently did experiment on moderation. But some can't be joined, some you have to apply to join. And anyway, even joining - as in the main site - does not enable one to know what the settings are and in particular, if they foresee moderation or not, as only owners and admins can see the settings.
Friday 13 team
However, yesterday, I found the Friday 13 team in the mirror site, which can be freely joined and whose description says: "workflows off", which suggests it was meant for studying the effects of imposed moderation when the settings don't foresee it. So I joined it, and even from the very limited viewpoint of a contributor, it is an extremely interesting team:
Its name seems due to the fact that it was created on Friday January 13, 2012: this suggests that developers started examining the effects of imposed moderation at least since that date;
Its other participants, before I joined, were: Maggie S (owner), P.Culture (contributor), sjl (admin), maggieAdmin (contributor), Manager Dec5Team (username: maggieManager, contributor), Firstname5 Lastname5 (username: testuserapi5, contributor), fernandotakai (contrinutor). I.e. 1 owner, 1 admin, and 5 contributor profiles, of which 3 might be managed by the same person called Maggie or a variant thereof: Margarita Shamraeva, the QA Analyst of the UniversalSubtitles team, who used to reply on the old help forum until the end of March 2012?
What's striking in this members' list is that none of these profiles seem to belong to a banal Amara user, i.e. someone concretely doing collaborative real subtitles with other real subtitlers (1).
Subtitles
None of the subtitle sets in this team are real subtitles. Some were created by uploading an existing subtitle file for another video, some consist only of fillers like "one" "two" "three" "four" "five". And some present a combination of both types: for the English subs of the Bad Communication Ep. 01 video, the various "subtitlers" first used the fillers solution. Then on June 25, for the last revision, Manager Dec5Team uploaded the subs of the "About Amara" video showing on the home page of the main site. The result is very funny - see attached amara_bad_communication.jpg.
Moderation
However, I'm not sure humor was the goal in uploading those subs to that video: it rather looks as if this was an attempt to eliminate the moderation task added to these subs. If so, it didn't fully work: while the subs now work in the player and the English subs page looks normal, with a clickable Edit Subtitles link, if you attempt to edit the subs, you get a message saying "These subtitles are moderated. See the Friday 13 team page for information on how to contribute.", followed by an OK button. And there is still a moderation task for these subs:
Other hypothesis: Manager Dec5Team intended to illustrate how full subs, if not marked as such, still triggered the moderation.
I don't know. The point is that a team with "no workflow" should not have moderation at all - that's why you lifted it in the last version of the software: thanks again - but Friday 13 has this and a full range of the moderation blocks that variously hit real "no workflow" teams for 4 weeks.
These moderation blocks actually started with the very first set of subs made in the team, on the day of its creation: the English subs for The Friday 13th Song! got a banner saying "Draft only Showing Revision 0, created 01/13/2012 by P.Culture." and a disabled Edit Subtitles link, but no visible task.
As to the task-creating blocks, besides the already mentioned one for the English subs of Bad Communication Ep. 01, there are 5 further ones.
9 moderation blocks on 17 videos seems a lot for a team with only - until yesterday - 6 participants' profiles, of which 3 seem to be IDs of a single person, and 1 did not participate at all in "subtitling" activities (see note 1). A team created, according to its "no workflow" description, to study the effects of imposed workflowed moderation on similar real "no workflow" teams.
Some questions
Again, the important thing is that you eventually lifted the mistakenly imposed moderation. However, lest this experience repeat itself, some questions should be examined:
When moderation was mistakenly imposed on these real "no workflow" teams and people started reporting the same blocks that the Friday 13 team had already been gathering evidence about since January:
Why were participants in these teams told - by admin -> members Amara messaging and by personal e-mails - to send descriptions of the blocks encountered to be forwarded to you, developers, when you already knew their technical aspects full well through that Friday 13 team?
Why the insistence that we keep these descriptions concise and technical, when what you actually sorely lacked, due to your way of using subtitling in that team, was even the vaguest idea of the concrete impact of these blocks on real collaboration in real subtitling by real people?
Why did you let us struggle for four weeks with these problems, when the evidence gathered in that team was more than enough to show that imposed moderation was a long way from being ready, and therefore should be lifted immediately, especially as its imposition was a mistake?
Or wasn't it? Are you developers perhaps still planning to impose moderation in all teams in future, once you've refined it?
Let's work together again, shall we?
An imposed hierarchical moderation that works might indeed make perfect sense for Amara's entreprise services. Many entrepreneurs are probably wary of the informal, Wikipedia-like, flexible collaboration for which many other users, like me, are attracted to Amara. Not all entrepreneurs are fans of Benkler. So if you can tell them: "Yes, but if you create a team with us, you'll have all the rigid hierarchy you want", that will reassure them.
However, if it is so, please let us non-entrepreneur users know. We are aware that entreprise services are a needed part of Amara, some of us would even volunteer as guinea pigs to help you refine them. But we need to know, because if moderation is to be imposed on all Amara teams, several will have to close as such and set up external collaboration structures instead, as they use videos that are not theirs. And then, in several small teams, where participants converged because they shared a purpose but are not otherwise connected, workflowed, hierarchical moderation would be impractical.
And letting us users know is also a good idea for other innovations. Take the short-line, 4-line format you also rolled back - thanks again - but after it frightened us because it amputated formerly working subs in the player. Had you announced that you wanted to introduce that, and needed volunteers to test it and add videos for this testing in a team on the developers' mirror site, the people who have been asking for multiline format on the old help forum might have told you immediately that software-imposed short lines was not the way to go. And if they hadn't come up, having heaps of videos in a test team where it was implemented would have shown that too.
Or take your presently in force idea of replacing copy-paste by .txt file uploading for plain transcripts. You possibly have a good developers' reason for that, but if you don't tell us about it, all we see is that the accented letters that are essential in several languages don't appear in the resulting unsync'd subs. And yet you surely tested this .txt upload in mirror videos between yourselves. But either you forgot the existence of languages other than English, or being geeks, you never thought people might encode .txt files in formats that Amara can't process properly. Had you announced your intention and asked for volunteers to test the new method on the mirror site, the accented letters issue would have come up there.
True, being developers, you know the software better than us. But being users, and many users at that, with collaboration experiences, we know what we do with the software better than you. With the possible exceptions of Margarita Shamraeva and Craig Zheng who used to directly reply to our questions on the old help forum. But that was when Amara was called Universal Subtitles.
The change to mediated interaction at the end of March makes sense too, considering the growth of the platform. And Marisa Jean Browne first, and now Darren Bridenbeck are good, dedicated mediators. However, to really work, mediated with a community of users requires appropriate communication concepts and means. And there seems to be room for improvement there. Again, let's work together on that.
Best,
Claude
(1) Not even fernadotakai, who the joined Friday 13 team "1 week, 5 days ago" as I'm writing this (July 8, 2012) seems to be a real banal subtitler No other activity for him is listed in the team's Activity page anyway. And in the other teams - both on this mirror and on the main site - where this profile participates, the activity lists are just as discreet as to what he exactly does in these teams. However, a Fernando Takai is listed as Software Developer in pculture.org/pcf/about.
Claude Almansi
Dear Amara developers
First of all, thank you so much for having lifted the moderation imposed in June against the settings of teams that don't want it and in some cases, can't have it because they work on videos that are not theirs.
So whether that imposition was deliberate or the result of a mistake is now immaterial, fortunately: could you just remove its remaining traces, please? See my comment to Darren Bridenbeck's July 3rd Amara Update post, about still blocked subs.
Experimenting in the developers' mirror site
While this imposed moderation was going on, though, I kept thinking: why don't the developers try that in the https://dev.universalsubtitles.org mirror site that's meant for that kind of experimenting, and let us subtitle in peace in the main http://www.universalsubtitles.org site? I kind of remember I even wrote that, perhaps in stronger terms, both in this help forum and in e-mails to Dean Jensen and Darren Bridenbeck, even volunteered to participate as guinea pig in such experimentation.
I should have checked better: there are several teams in that mirror site, particularly teams named by dates starting from December 2011, where you apparently did experiment on moderation. But some can't be joined, some you have to apply to join. And anyway, even joining - as in the main site - does not enable one to know what the settings are and in particular, if they foresee moderation or not, as only owners and admins can see the settings.
Friday 13 team
However, yesterday, I found the Friday 13 team in the mirror site, which can be freely joined and whose description says: "workflows off", which suggests it was meant for studying the effects of imposed moderation when the settings don't foresee it. So I joined it, and even from the very limited viewpoint of a contributor, it is an extremely interesting team:
I.e. 1 owner, 1 admin, and 5 contributor profiles, of which 3 might be managed by the same person called Maggie or a variant thereof: Margarita Shamraeva, the QA Analyst of the UniversalSubtitles team, who used to reply on the old help forum until the end of March 2012?
What's striking in this members' list is that none of these profiles seem to belong to a banal Amara user, i.e. someone concretely doing collaborative real subtitles with other real subtitlers (1).
Subtitles
None of the subtitle sets in this team are real subtitles. Some were created by uploading an existing subtitle file for another video, some consist only of fillers like "one" "two" "three" "four" "five".
And some present a combination of both types: for the English subs of the Bad Communication Ep. 01 video, the various "subtitlers" first used the fillers solution. Then on June 25, for the last revision, Manager Dec5Team uploaded the subs of the "About Amara" video showing on the home page of the main site.
The result is very funny - see attached amara_bad_communication.jpg.
Moderation
However, I'm not sure humor was the goal in uploading those subs to that video: it rather looks as if this was an attempt to eliminate the moderation task added to these subs. If so, it didn't fully work: while the subs now work in the player and the English subs page looks normal, with a clickable Edit Subtitles link, if you attempt to edit the subs, you get a message saying "These subtitles are moderated. See the Friday 13 team page for information on how to contribute.", followed by an OK button. And there is still a moderation task for these subs:
Other hypothesis: Manager Dec5Team intended to illustrate how full subs, if not marked as such, still triggered the moderation.
I don't know. The point is that a team with "no workflow" should not have moderation at all - that's why you lifted it in the last version of the software: thanks again - but Friday 13 has this and a full range of the moderation blocks that variously hit real "no workflow" teams for 4 weeks.
These moderation blocks actually started with the very first set of subs made in the team, on the day of its creation: the English subs for The Friday 13th Song! got a banner saying "Draft only Showing Revision 0, created 01/13/2012 by P.Culture ." and a disabled Edit Subtitles link, but no visible task.
Same for the Spanish subs of How to Practice Friday the 13th Superstitions (dated 02/06/2012 in the draft banner) and the Dutch subs of Firday May 13th, 2011 Weather Forecast ("Firday" sic; same date).
As to the task-creating blocks, besides the already mentioned one for the English subs of Bad Communication Ep. 01, there are 5 further ones.
9 moderation blocks on 17 videos seems a lot for a team with only - until yesterday - 6 participants' profiles, of which 3 seem to be IDs of a single person, and 1 did not participate at all in "subtitling" activities (see note 1). A team created, according to its "no workflow" description, to study the effects of imposed workflowed moderation on similar real "no workflow" teams.
Some questions
Again, the important thing is that you eventually lifted the mistakenly imposed moderation. However, lest this experience repeat itself, some questions should be examined:
When moderation was mistakenly imposed on these real "no workflow" teams and people started reporting the same blocks that the Friday 13 team had already been gathering evidence about since January:
- Why were participants in these teams told - by admin -> members Amara messaging and by personal e-mails - to send descriptions of the blocks encountered to be forwarded to you, developers, when you already knew their technical aspects full well through that Friday 13 team?
- Why the insistence that we keep these descriptions concise and technical, when what you actually sorely lacked, due to your way of using subtitling in that team, was even the vaguest idea of the concrete impact of these blocks on real collaboration in real subtitling by real people?
- Why did you let us struggle for four weeks with these problems, when the evidence gathered in that team was more than enough to show that imposed moderation was a long way from being ready, and therefore should be lifted immediately, especially as its imposition was a mistake?
Or wasn't it? Are you developers perhaps still planning to impose moderation in all teams in future, once you've refined it?Let's work together again, shall we?
An imposed hierarchical moderation that works might indeed make perfect sense for Amara's entreprise services. Many entrepreneurs are probably wary of the informal, Wikipedia-like, flexible collaboration for which many other users, like me, are attracted to Amara. Not all entrepreneurs are fans of Benkler. So if you can tell them: "Yes, but if you create a team with us, you'll have all the rigid hierarchy you want", that will reassure them.
However, if it is so, please let us non-entrepreneur users know. We are aware that entreprise services are a needed part of Amara, some of us would even volunteer as guinea pigs to help you refine them. But we need to know, because if moderation is to be imposed on all Amara teams, several will have to close as such and set up external collaboration structures instead, as they use videos that are not theirs. And then, in several small teams, where participants converged because they shared a purpose but are not otherwise connected, workflowed, hierarchical moderation would be impractical.
And letting us users know is also a good idea for other innovations. Take the short-line, 4-line format you also rolled back - thanks again - but after it frightened us because it amputated formerly working subs in the player. Had you announced that you wanted to introduce that, and needed volunteers to test it and add videos for this testing in a team on the developers' mirror site, the people who have been asking for multiline format on the old help forum might have told you immediately that software-imposed short lines was not the way to go. And if they hadn't come up, having heaps of videos in a test team where it was implemented would have shown that too.
Or take your presently in force idea of replacing copy-paste by .txt file uploading for plain transcripts. You possibly have a good developers' reason for that, but if you don't tell us about it, all we see is that the accented letters that are essential in several languages don't appear in the resulting unsync'd subs. And yet you surely tested this .txt upload in mirror videos between yourselves. But either you forgot the existence of languages other than English, or being geeks, you never thought people might encode .txt files in formats that Amara can't process properly.
Had you announced your intention and asked for volunteers to test the new method on the mirror site, the accented letters issue would have come up there.
True, being developers, you know the software better than us. But being users, and many users at that, with collaboration experiences, we know what we do with the software better than you. With the possible exceptions of Margarita Shamraeva and Craig Zheng who used to directly reply to our questions on the old help forum. But that was when Amara was called Universal Subtitles.
The change to mediated interaction at the end of March makes sense too, considering the growth of the platform. And Marisa Jean Browne first, and now Darren Bridenbeck are good, dedicated mediators. However, to really work, mediated with a community of users requires appropriate communication concepts and means. And there seems to be room for improvement there. Again, let's work together on that.
Best,
Claude
(1) Not even fernadotakai, who the joined Friday 13 team "1 week, 5 days ago" as I'm writing this (July 8, 2012) seems to be a real banal subtitler No other activity for him is listed in the team's Activity page anyway. And in the other teams - both on this mirror and on the main site - where this profile participates, the activity lists are just as discreet as to what he exactly does in these teams. However, a Fernando Takai is listed as Software Developer in pculture.org/pcf/about.