WG1 Workshop in Coimbra
by Ana Luís
The WG1 Workshop in Coimbra focussed on (corpus) examples, their potential for language material, various aspects of finding good examples for various teaching and learning purposes, and the potential role(s) of crowdsourcing in the entire process from example selection to exercise preparation. The workshop was attended by 23 participants and took place over 2 days in Coimbra, Portugal.
Both days had a similar structure: two presentations in the morning, followed by group work, then the presentation of results from the groups, and in the end the discussion.
Day 1 was dedicated to the investigation of the potential of explicit crowdsourcing with teachers. The morning part included two presentations: the first presentation was given by Ildiko Pilar on HitEx, a tool for automatic selection of good example candidates from corpora. The tool is targeted at language teachers and offers various customisable filtering options. After presenting the tool, Ildiko Pilar also considered potential avenues for using crowdsourcing with teachers (and to lesser extent with learners) on the results provided by the tool.
The second presentation given by Johannes Graën on how to harness the power of crowdsourcing when preparing exercises from parallel examples. The presentation also served as an introduction into the group work part of the day, as that included an exercise using parallel sentences. As part of the exercise, the participants had to first use parallel Swedish-English examples to deduct the meaning of 7 Swedish expressions. Then, the groups had to find three good examples per expression. Finally, each group had to devise an exercise based on the selected good examples. The groups proved to be very resourceful and some devised more than one exercise. Furthermore, searching for examples sparked various discussions among group members and later among groups about the potential of not only good but also bad examples for language teaching. Group work concluded with a discussion on how crowdsourcing could be introduced into this process – and while teacher motivation has often been identified as problematic, the participants have identified some possibilities for crowdsourcing, even in the tool that was used for the exercise.
Day 1 was concluded by the presentation of learner survey, given by Elżbieta Gajek, Lina Miloshevska and Çiler Hatipoğlu. In the first part, the presenters gave an overview of the results from the pilot study conducted with Turkish and Polish students. Furthermore, an update on the current status of the survey questionnaire and plans for the future were provided. In the second part, the presenters invited the workshop participants to provide comments and suggestions. The comments included recommendations to avoid having a long survey, and to carefully select its focus rather attempting to cover many different aims. There was also a discussion on the problem with reaching under-18-year-old learners due to GDPR.
Day 2, which focussed on language learners, started with a presentation by Kristina Koppel on her experiment in which she evaluated automatically extracted examples on a group of learners and a group of experts (lexicographers). The findings were that there were not many differences between learners and experts in identifying what a good or bad example was. To some extent, this highlights the fact that both experts and learners seem to be equally suitable for a crowdsourcing task of identifying good/bad examples, which teachers can then use for exercise preparation.
The second presentation of the day was given by Tanara Zingano Kuhn. In her presentation, Tanara presented the progress of a project that aims to filter corpora for development of language learning resources by using crowdsourcing. The presentation was divided in two parts, with a group exercise between them. The first part focussed on the aims and methodology of her project, while the second part, following the group work, presented the results of the experiment and the use of GDEX tool for identifying good examples.
For the exercise, the groups were given a sheet with 22 automatically extracted English examples. Each member had to first mark all examples containing offensive content (and thus unsuitable for language teaching material), then all group members compared the results and reached the final group decision for each example. Then the group sheets were swapped between the groups so that in the end all the groups saw all the examples and group decisions on them. In this way, the participants were given first-hand experience in various aspects of crowdsourcing work: as crowdsourcers voting, as referees agreeing on the final decision, and as researchers discussing possible advantages and shortcomings of the procedure. The group reports on their observations pointed to a high level of agreement among group members, and the fact that they encountered similar issues, e.g. what “offensive” means, the need to mark sentences problematic for other reasons, and occasional divergence in interpretations of example connotation.
In the second part of her presentation, Tanara reported the results of the experiment with crowds from Brazil/Portugal,Serbia,Slovene, and the Netherlands, and showed examples to illustrate some interesting findings. These results showed several similarities between the observations of the workshop participants and the experiment output. However, there were also several differences, which pointed to shortcomings of using GDEX, and these differences were then further discussed with the participants in more detail.
The workshop was concluded with a general discussion in which the participants were asked to reflect on the two days, consider existing practices of obtaining examples for exercises, and how crowdsourcing methods could be implemented with teachers and learners. Teacher participants reported that they currently search for examples online (newspaper pages, corpora etc.), also use social media such as Facebook (example for complaints), and that they give tasks of devising exercises to their (advanced) learners.