Objectives & Results
Main Objective
The main objective of the enetCollect Action is to address the major European challenge of fostering the language skills of all citizens regardless of their diversified social, educational, and linguistic backgrounds. To this end, the Action is concerned with the domain of Language Learning and specifically focuses on enhancing the production of learning material in order to cope with the increasing demand for language learning material and the striking diversification of learner profiles. EnetCollect will address this challenge by kick-starting a Research and Innovation (R&I) trend that combines the well-established domain of Language Learning with recent and successful crowdsourcing approaches in order to trigger an innovation breakthrough for the production of language learning material.
To address its main objective, enetCollect will also involve a variety of R&I players who are working on language-related topics and have laborious and/or complex tasks that can be approached by crowdsourcing. Indeed, even though Language Learning might not be of direct interest to them, being able to make use of a vast and continuous workforce generated by crowdsourcing on learners’ and teachers’ actions constitutes a ground-breaking opportunity for experts e.g. in linguistics, computational linguistics or software engineering/ergonomics. By involving language-related R&I players, enetCollect aims at favoring its flexible and sustainable success in numerous, concurrent and mutually beneficial ways.
The main objective of enetCollect will be addressed by pursuing seven specific COST objectives. COST Actions distinguish objectives related to the creation of knowledge, termed Research-Coordination Objectives, and objectives related to creating and empowering a community, termed Capacity-Building Objectives. We detail below these objectives and the achievements performed so far. More specific information about the enetCollect’s objectives and Workplan can also be found in the Memorandum of Understanding.
Results. EnetCollect has successfully brought together an interdisciplinary community of stakeholders to explore the high-potential subject of the combination of language learning and crowdsourcing. EnetCollect members came from related and relevant domains, such as Language Learning, Natural Language Processing, Computer Science, E-Lexicography, Learner Corpora and Ethics. They successfully tackled exploratory tasks and implemented working prototypes, allowing to better understand the extent of the high-potential behind this unusual combination mixing a linguistically-oriented field with a technically-oriented one.
Among other things, the efforts of the enetCollect members allowed to better understand how familiar language teachers and learners were with the concept of crowdsourcing, how and what type of language learning material could be crowdsourced from them, what criteria would foster their participation, and what technical, ethical and legal aspects should be taken into account by any solution for crowdsourcing learning content. The efforts also allowed to better understand how interesting the subject could also be to create language-related datasets.
Overall, the efforts resulted in hundredths of participations and presentations to meetings, several dozens of scientific stays and intensive collaborations, several dozens of scientific publications by authors that had never collaborated before and in several new projects funded. All of this despite the major hindrance induced by the Covid-19 health crisis.
In completing such achievements, it has been gradually confirmed that the combination of Language Learning and Crowdsourcing is a subject for which it is worth establishing a dedicated community. In that perspective, enetCollect created a solid groundwork for this brand-new community to thrive beyond the completion of the COST Action itself.
Evaluation
In July 2019, for its mid-way report, enetCollect achievements were all evaluated with grades ranging from “very good” to “excellent” (see evaluation report)
In December 2021, the final report was submitted. The different deliverables are available on the pages of the WGs as well as a transversal document, the handbook, providing an overview of enetCollect’s outputs. [link]
Research-Coordination Objectives
Objective 1: Creating a theoretical framework for achieving a shared understanding of the topic, ranging from very concrete aspects such as defining a common terminology, to higher-level aspects such as evaluating the potential and limits of related approaches. This objective is pursued by the Working Groups and focuses on transforming the tacit knowledge of members into a consensual and codified form, and on the creation of blueprints (design specifications) at any relevant level (technical, usability, ethical, etc.).
Results. EnetCollect successfully approached the creation of a theoretical framework for the combination of language learning with crowdsourcing techniques, by addressing different relevant aspects of the theoretical framework within its five working groups. The work on the theoretical framework proceeded well to the extent of gathering relevant data and reaching a consensual understanding on several key aspects of the topic as explained hereafter. The main part missing yet is a proper summing up and closure of the subject which could not be fully achieved due to the pandemic which made it impossible to hold any intensive meetings in presence. These would have been needed to deepen further and bring together all aspects, and to integrate them into a rounded up framework description.
Approaches of explicit and implicit crowdsourcing and their application to language learning and language-data creation have been investigated both through extensive literature gathering and review[1], exchanges through relevant talks at the enetCollect meetings[2], which were partly video recorded[3], and various shared efforts led by groups of members (see WG-related deliverables, Dlv. 1-14).
With regards to the language learning side of things, large-scale surveys have been carried out with teachers[4], in relation to language learning platforms[5], and are still ongoing for learners[6]. The achievements and outcomes of all these efforts are described in more detail in deliverables 1.3 (Dlv. 2), 2.3 (Dlv. 5) and 3.2 (Dlv. 8).
Technical, legal, ethical and commercial aspects have been addressed and could be completed to varying extents. While considerable work has been completed on ethical aspects, the network could not create the grounds to substantially assess the commercial, legal and technical implications related to the subject. The main reasons for that were the lack of stakeholders with the expertise and interest to explore these subjects or a lack of practical use cases to direct the efforts. For further details see explanations on deliverables 4.1 (Dlv. 10), 4.2 (Dlv. 11), 5.2 (Dlv. 13) and 5.3 (Dlv. 14). The achievements and outcomes regarding the ethical aspects are described in detail in Deliverable 5.1 (Dlv. 12).
The overall efforts achieved by the members allowed to create a noticeable amount of consensual knowledge on the subject. The shared understanding of the subject at its current state is summarized in the handbook report (Deliverable 7.13, Dlv. 23) and detailed in more than 50 scientific publications authored by the members[7].
[1] https://www.zotero.org/groups/2174884/_enetcollect_bibliography_ and https://www.zotero.org/groups/2352007/_enetcollect_bibliography_-_wg2_review_group_
[2] https://enetcollect.eurac.edu/cost-tools-events/meetings/
[3] http://videolectures.net/enetCollect
[4] https://hrcak.srce.hr/file/353213
[5] https://enetcollect.eurac.edu/mapping-the-language-learning-platforms-worldwide-in-order-to-foster-the-crowdsourcing-of-language-related-datasets-and-the-automatic-generation-of-exercises
[6] https://www.nalans.com/index.php/nalans/article/view/391
[7] https://enetcollect.eurac.edu/results/r_i_dissemination
————————-
Objective 2: Gathering evaluation data for complementing the theoretical framework with factual information obtained from prototypical experiments and wide-ranging surveys in order to extend and root the theoretical framework and evaluate which aspects should be approached first.
Results. Prototypical experiments and hands-on activities have been carried out for all working groups, even though to different extents. They comprise the development of prototypical language learning applications and crowdsourcing experiments (see details in deliverable 1.4 (Dlv. 3) and deliverable 2.4 (Dlv. 6) of WG 1 and WG 2), reviews of current regulations, ethical principles and conditions (see details in deliverable 5.1 (Dlv. 12) of WG5), and critical evaluations of existing language learning environments and usability conditions (see details in deliverable 3.1 (Dlv. 7) and deliverable 3.2 (Dlv. 8) of WG3). In particular, the organization of two Hackathon-like meetings (Crowdfests) (more were planned but impeded by the pandemic) strongly boosted the practical development work within enetCollect and the formation of sustainable task forces, of which many continued working and creating outputs until way after the Crowdfest. During these events, varied lists of topics[8] have been addressed.
[8] https://www.enetcollect.net/ilias/goto.php?target=wiki_wpage_36_546&client_id=enetcollect and https://enetcollect.net/ilias/goto.php?target=file_976_download
Objective 3: Disseminating the knowledge created and the achievements obtained to any relevant audience.
Results. EnetCollect was very successful in disseminating the created knowledge, which is demonstrated by 89 presentations and publications to conferences, and journals of the NLP, language learning, and ethics stakeholder groups (including LREC, NLP4Call, EuroCALL, PL-Call, elex, LDK, conferences, etc.)[9] of which 9 were supported by COST funded conference participations (5 dissemination meetings and 4 ITC conference grants)[10].
Furthermore, members with various backgrounds from research and (to a lesser extent) private businesses carried out overall more than 600 travels to participate in 11 enetCollect meetings. And six webinars with a total of twelve presentations[11] attracted more than 100 participants from the enetCollect network and outside.
In addition, newsletters and informal communications on the enetCollect-all mailing list[12] frequently communicated relevant outputs of the Action throughout its lifetime and beyond. As well, results and ongoing activities got communicated via enetCollect’s facebook and Twitter accounts. Research articles and recordings of presentations were promoted through enetCollect’s resarchgate and videolectures pages (see more information in MoU5).
Occasionally, also transfer-oriented dissemination activities were carried out on the local scope of enetCollect’s member institutions (see deliverable 7.7, Dl. 19).
[9] https://enetcollect.eurac.edu/results/r_i_dissemination
[10] https://enetcollect.eurac.edu/results/conference-grants/
[11] https://enetcollect.net/ilias/goto.php?target=file_1081_download and
https://enetcollect.net/ilias/goto.php?target=file_1102_download
[12] https://groups.google.com/g/enetcollect-all
Capacity-Building Objectives
Objective 4: Creating a balanced interdisciplinary core community of active stakeholders (and a core crowd reachable through it). This objective will be pursued by steadily expanding the network while maintaining a balance of member characteristics), through the Working Groups and through innovative dissemination and exploitation activities devised to foster collaborations, with special focus on ECIs (Early Career Investigators).
Results. Within its first two years, enetCollect has successfully managed to create a novel and growing balanced community of stakeholders moving forward the new R&I trend on the combination of Language Learning and Crowdsourcing. Throughout the third and fourth year the efforts continued and the number and variety of members increased further. In particular, a relevant number of new members and speakers could be involved for the 4th annual Action meeting planned for March 2020, which had to be cancelled at the very last moment due to the beginning of the pandemic. After that and throughout the final year of the Action the pandemic continued and the increase of members was far more modest.
The balancing of the community has been achieved reasonably well in terms of COST criteria related to gender (38% male / 62% female), professional maturity (24% ECI / 72% non-ECI), and country distribution (all but one COST countries covered, 41% ITC / 57 non-ITC / 1% NNC). The development of the network is documented by the statistics computed on a regular basis and provided on the website.[13]
The balancing in terms of interdisciplinarity and research vs. non-research players
could not be fully achieved. While members of all stakeholder groups (content-creation and content-usage experts, crowdsourcing experts and content management system developers) could be involved into enetCollect their proportions greatly differed, with the best represented groups being NLP stakeholders, linguistics professionals and teachers and the smallest groups being non-research content creation experts and content management system developers. Also, the majority of enetCollect members is situated in research and only a fraction comes from public and private institutions or from industry.
The novel nature of this community is attested by publications for different target audiences[14], and several successfully funded project proposals on topics related to enetCollect[15], co-authored by members who had not collaborated before participating in the Action.
The growing nature of this community can be observed through the increasing number of members registering to the mailing lists and on the intranet (around 270 emails registered to the general mailing list and 205 profiles registered on the intranet at the end of the Action[16], as well the growing number of people attending the Annual Meetings: 54 for the 1st in Bolzano (Italy), 68 for the 2nd in Iasi (Romania) and 105 for the 3rd in Lisbon (Portugal).
This objective of creating a balanced interdisciplinary core community of active stakeholders has been tackled successfully throughout the Action with some shortcomings in terms of the balancing of stakeholders. Overall, it reached noteworthy results despite the huge obstruction introduced by the pandemic.
[13] http://enetcollect.eurac.edu/about-enetcollect/statistics/
[14] https://enetcollect.eurac.edu/results/r_i_dissemination/ and https://enetcollect.eurac.edu/results/general_dissemination/
[15] https://enetcollect.eurac.edu/enetcollect-related-projects/
[16] http://enetcollect.eurac.edu/about-enetcollect/statistics/
————————-
Objective 5: Establishing communication channels. This objective aims to ensure that the created community has well-identified communication channels allowing to easily share information and reach relevant members or stakeholders.
Results. EnetCollect created several Action-internal mailing lists for general matters and every working group, as well as for special project-related activities[17] (e.g. an Erasmus+ proposal writing group).
In addition, enetCollect’s website[18] was constantly updated with content on upcoming and concluded activities, and reports on featured activities within enetCollect’s blog[19]. Facebook and Twitter accounts, as well as a project entry on researchgate were set up and help members and interested stakeholders to stay connected.
[17] https://enetcollect.eurac.edu/results/communication-channels/
[18] https://enetcollect.eurac.edu
[19] https://enetcollect.eurac.edu/blog/
————————-
Objective 6: Fostering new funded initiatives. This objective aims to foster new initiatives complementing the Action. This objective will be pursued by using the Action’s achievements to plan
parallel or follow-up projects and its indicator of completion will be the successful preparation of two large proposals for European follow-up projects as well as several proposals to different regional and national funding schemes.
Results. The objective to foster new funded activities has been achieved to a great extent, as enetCollect’s core group has promoted several activities to foster new funded initiatives (see deliverable 7.2 (Dlv. 16), exploitation plan) throughout the lifetime of the Action. This does not mean that all initiatives led to project applications or successfully funded
proposals, even though a substantial number of related funding applications can be reported.
Several initiatives to foster funding acquisition were started. First, a section on the enetCollect website informs members about relevant funding schemes[20] . Second, Marie Curie Individual Fellowship applications related to enetCollect thematics were fostered by an applicant and host matching initiative in the form of a poster session at the 2nd Annual meeting in Iasi. Furthermore, experienced members offered an information session on the funding scheme and application procedure. Third, an open call for preparing one or several Erasmus+ KA2 proposals were circulated among enetCollect members and led to several online sessions for identifying relevant topics and building potential consortia. While several members were interested to participate, this initiative did not evolve into the submission of any proposal. Indeed, by the time the subject could have been fostered more, the pandemic deviated energies of the participating members to more urgent activities. Finally, the chairs of the Action recurrently encouraged the application of enetCollect-related funding proposals and gave feedback and support to several funding applications prepared by enetCollect members.
Overall, funding for 7 related projects were successfully acquired by enetCollect members between 2017 and 2021 and some more are under evaluation or in preparation for re-submission[21].
[20] https://enetcollect.eurac.edu/autonomous-funding-opportunities/
[21] https://enetcollect.eurac.edu/enetcollect-related-projects/
————————-
Objective 7: Creating an association. This objective aims at creating an association with the purpose of continuing the work started in enetCollect and playing a driving role for the aforementioned community after the conclusion of the Action. This association will follow a long-term agenda, take over and maintain the outputs of the Action and handle subsequent editions of a recurrent scientific event. The indicator of completion for this objective will be the successful creation of a non-profit association and the number of its members.
Results. The creation of a DARIAH working group is under processing. The decision of applying for a DARIAH working group has jointly been taken by the core group. The application to be evaluated by the DARIAH board of directors[22] is in preparation.
[22] https://www.dariah.eu/wp-content/uploads/2019/09/DARIAH-Working-Groups-Policy-Statement_v5.pdf, Section 4
————————-