Keywords

1 Introduction

WYRED (netWorked Youth Research for Empowerment in the Digital society) is a European Project funded by the Horizon 2020 programme in its “Europe in a changing world – inclusive, innovative and reflective Societies (HORIZON 2020: REV-INEQUAL-10-2016: Multi-stakeholder platform for enhancing youth digital opportunities)” Call. It is coordinated by the GRIAL Research Group [1] of the University of Salamanca (Spain) and it started at November 2016 and will be developed along three years, until October 2019.

The project aims to provide a framework for research in which children and young people can express and explore their perspectives and interests in relation to digital society, but also a platform from which they can communicate their perspectives to other stakeholders effectively through innovative engagement processes, WYRED is informed by the recognition that young people of all ages have the right to participation and engagement. It has a strong focus on inclusion, diversity and the empowerment of the marginalized. The aim is to replace the disempowering scrutiny of conventional research processes with the empowerment of self-scrutiny and self-organization through the social dialogue and participatory research [2].

To support the framework for research from a technological point of view, a WYRED technological ecosystem has been defined [3]. The WYRED Platform is the component of this complex technological solution that is focused on supporting the social dialogues that take place between children, young people and stakeholders.

The Platform is already developed but it is important to ensure the acceptance by the final users, the children and young people mainly. To achieve this goal, a usability test with real users was carried out. A group of undergraduate students between 18 to 25 years old used the Platform during 3 weeks with different profiles. After the pilot the participants filled the System Usability Scale (SUS) questionnaire anonymously. This paper describes the usability study and analyzes the SUS test results.

Finally, the paper is set out in seven sections. The second and third sections provide a brief description about the WYRED ecosystem and the WYRED project. The fourth section outlines the research methodology; it explains the materials and methods used to evaluate WYRED Platform, the evaluation procedures and the involved participants. The fifth section presents the results. The sixth section is the discussion and the final section concludes the paper with its more significant contributions.

2 The WYRED Ecosystem

There are a large number of definition related to the technological ecosystem concept, but the WYRED ecosystem is based on the natural ecosystem metaphor applied to the technological context. A technological ecosystem is a set of users and software components related to each other through information flows in a physical environment that provides the support for those flows [4,5,6,7]. There are two main differences between this technological approach and the traditional information systems: the technological ecosystems have the ability to evolve in different dimensions and people are an element of the ecosystem as important as software components.

The WYRED ecosystem is composed by a set of Open Source tools and the people involved in the project, not only the partners, but also the stakeholders, children and young people. Figure 1 shows the architecture of the ecosystem and the connections among the components. This is not a final solution, the WYRED ecosystem will evolve over time to cover the new requirements provided by the project partners and solve the problems detected during the usability studies, for instance, the results of the study described in this paper.

Fig. 1.
figure 1

Architecture of the WYRED technological ecosystem.

Full size image

The architecture is based on the architectural proposal for technological ecosystems defined by García-Holgado [5, 8, 9]. There are three layers that organize the different software components according to their role in the ecosystem - infrastructure, data and services - and two input streams which introduce the human factor as another element of the technological ecosystem. Currently, some components of the architecture are in development phase, in particular the document management system based on OwnCloud and the indexing service based on ApacheSolr.

3 WYRED Platform

The WYRED Platform (https://platform.wyredproject.eu) is one of the services included in the WYRED ecosystem. A key aim of this project is to engage children and young people in a process of social dialogue giving them a voice to share their thoughts, fears and feelings in relation to digital society. The aim of the WYRED Platform is provided a technological support to these social dialogues.

The WYRED Platform allows an easy interaction process among the involved stakeholders; high-level engagement conditions; a secure environment in which all the participants feel comfortable enough with the privacy issues, but with a special attention to the underage ones; and a suitable dashboard for data analytics [10].

A Content Management System (CMS) provides a solid base to build online tools focused on managing information and supporting the interaction among the users. Nowadays, there are three CMS that have been established themselves as the leaders: Joomla, Drupal and WordPress. The WYRED Platform is based on Drupal (release 7.x) because is the most technically advanced [11], with a powerful development framework, and it is the most recommended for community platform sites with multiple users [12] due to its own experience providing a space for working groups (https://groups.drupal.org). Although Drupal released the version 8 on November 2015, not all the modules that extend its core functionality are migrated to the new version, in particular, the base module to develop the WYRED communities (Organic Groups module) has a development version for Drupal 8 but not a stable version that can be used in a production site.

Regarding the functionality of the Platform, the interaction revolves around the communities, where the social dialogues take place. Each community provides a safety space to share ideas, research, build projects, etc. using three types of contents: forums, events and projects. The users with a facilitator role inside a community are their administrators, they can manage user’s roles and subscriptions, moderate the forum threads, create projects and invite new users to the community both registered and non-registered users. Moreover, there are two types of communities, public communities that are accessible for all registered users and anyone can subscribe; and private communities only visible to their members.

The Platform is a private space where only registered user can access. The registration process is restricted too, new users must receive an invitation with a unique link to have access to the registration form. The only contents that anonymous users can visit are the help section with videos and training actions and the terms of use section. Just like the communities, users can have three different roles at the Platform level, one for normal user and two for users with privileges - facilitator and administrator -. A user with administrator role is the only one that can change the role to other users.

Finally, all registered users can answer a socio-demographic questionnaire about diversity and inclusion to get an overall perspective of the children and young people involved in the project. One of the most important things in the WYRED Platform is to maximize the users’ privacy, for this reason the user’s private information collected during the registration process and with the questionnaire is saved in a different place, not in the Platform.

4 Methodology

4.1 Participants

The study was carried out in the subject of “Communication Techniques and Skills”, a mandatory subject for undergraduate students that is taught in the first semester of the first year of the Degree in Social Education of the University of Salamanca (Spain) d. There were 80 students enrolled in the subject in the 2017–18 school year and the experimental group was formed by 77 students divided in 12 work groups. Only 14.29% of the students are men and 85.71% are women, these figures are due to in Spain the number of female tertiary students that choose studies in the field of social sciences is 60% [13]. Regarding the country of birth, 68 students are from Spain and 9 are foreign (2 Dutch, 1 Argentinian, 1 Belgian, 1 Brazilian, 1 Chinese, 1 Italian, 1 Jamaican and 1 Portuguese); all of them speak Spanish. Table 1 shows the distribution of participants by role in the WYRED Platform, gender and average age.

Table 1. Participants in the usability test
Full size table

4.2 Instrumentation

The chosen tool to measure the usability of the Platform was the System Usability Scale (SUS). The SUS questionnaire provides an effective, valid and reliable [14, 15] way to score the usability of a system. In addition, it is an efficient test, given the fact that it only consists of 10 items, and it can be applicable over a wide range of systems [16].

The items of the questionnaires are positive and negative alternated statements (in order to avoid response biases) rated on a 1 to 5 Likert scale (from “strongly disagree” to “strongly agree”, respectively) [17].

The validity and reliability provided by this test and the ease of its implementation made the System Usability Scale suitable for the WYRED Platform usability testing.

In addition to the 10 items of the SUS questionnaire, a set of demographic variables were also collected by the instrument: year of birth, gender, birthplace, parents’ birthplace, the family’s main spoken language, any eye diseases of the user that could affect the experience, the language used in the Platform and the role of the user (student or facilitator). Besides these demographic variables, an open field was provided at the end of the survey to let the users remark any relevant experience during the testing of the Platform. This field adds qualitative feedback to the SUS score (as the score itself it’s not diagnostic, only provides an overall usability rating).

The instrument was implemented using a customized version of LimeSurvey (https://www.limesurvey.org), an Open Source on-line statistical survey web application. The instrument was applied in Spanish, but it is also available in English to be used for future usability studies in other partner countries.

4.3 Study Design and Data Collection

The pilot experience was carried in the last weeks of face-to-face classes of the “Communication Techniques and Skills” subject of the Degree in Social Education of the University of Salamanca (Spain) during the 2017–18 school year. One of the main goals of the subject is to address conflict resolution.

During the pilot, the students were divided in 12 work group to promote a series of social dialogues on topics that students considered of their interest and fit into the topics of the WYRED project.

The selection and configuration process of the social dialogues started with the presentation of the proposed topics in a collaborative board using the free version of Padlet (https://padlet.com), a multi-device application to make and share content with others. Then, the topics was explained in a face-to-face class with the purpose of guarantee the widest range of possible topics. Each group selected a different topic and a group member as facilitator.

The aim of the activity was argued about the selected topic using the WYRED Platform to prepare a report in which the following aspects will be reflected:

  • Definition and description of the selected topic.

  • To what extend the selected topic concern to young people.

  • Proposals to solve or improve the state of the problem addressed.

  • User experience and proposals to improve the WYRED Platform.

The topics selected by the students to work in the Platform were:

  • Gender stereotypes and discrimination.

  • Cyberbullying, humiliation and sexting.

  • Construction and knowledge of ourselves through education and new technologies.

  • Privacy on Internet.

  • Stress reasons among young people.

  • Young people’s access to Deep Web.

  • The new influencers in social networks and the false myths.

  • Personality in the context of social networks.

  • Gender and the digital society.

  • Sexting in relation to gender violence and cyberbullying.

  • Machismo in the social networks.

  • Dangers of the Internet for young people.

After the first phase in classroom, the administrator of the WYRED Platform sent a registration invitation to the selected facilitators (12 students) with the data provided by the teacher. When each one of them finished the registration process, the administrator gave him/her a facilitator role in the Platform because the role cannot be assigned before the user exists in the Platform.

Each facilitator had two different tasks regarding their group mates. First, he/she had to create a community inside the Platform to carry out the activity. Second, the facilitator had to send a registration invitation to each of his/her group mates so that they had to register and start to work in the community.

The pilot experience lasted three weeks: one week to organize the activity and two weeks to use the WYRED Platform to prepare the report.

Regarding the data collection, the teacher sent by email the link to answer the questionnaire two days before the pilot experience ended, and the questionnaire was available for two weeks. During that period, two reminders were sent by the teacher to request more answers.

Moreover, the user interaction of the students was collected using Google Analytics (https://analytics.google.com), a web analytics service offered by Google to track website traffic. The WYRED Platform uses this tool principally to get information about the interactions from the different countries involved in the social dialogues and the used devices to work with the Platform. This information is complemented with the information extracted from the WYRED Platform database in order to get the number of forum threads and comments.

4.4 Analysis

The data collected by the instrument were downloaded to obtain a structured dataset with all the users’ answers. This dataset was cleaned to calculate subsequently the SUS score. Although the score calculation is relatively simple [17], the analysis of the responses has been made through the Python Pandas [18] library given its high performance and easy-to-use data structures.

The individual SUS score was calculated for every participant, to finally obtain the average SUS score of the WYRED Platform.

In addition, another analysis has been made taking into account that a technical error was found in the system during the testing. The average SUS score was also calculated for both the group of users exposed to the error and the group of users that tested the Platform after the error was fixed.

The interpretation of the results is based on previous System Usability Scale studies and benchmarks [19, 20] which allow SUS score comparisons and provide insights about the perceived usability of the WYRED system.

All the source code developed for this analysis is available at https://github.com/andvazquez/wyred-sus-hci-2018 [21].

5 Results

5.1 Socio-Demographic Results

From the 70 participants that entered the questionnaire, 43 of them finalized it, with the following socio-demographic characteristics:

  • 86.05% students are female and 13.95% male.

  • 41 students were born in Spain, 1 in China, 1 in Belgium and 1 in Netherlands.

  • 22 students have between 15 to 19 years (51.16%), 20 students have between 20 to 24 years (46.51%), 1 student has between 25 to 29 (2.33%). The average age is 21 years, approximately.

  • 33 participants used the Platform in Spanish (76.75%), 8 participants in English (18.60%) and 2 participants both in Spanish and English (4.65%).

  • No users pointed out significant sight diseases that could affect their experience with the platform.

5.2 WYRED Usability Test (SUS) Results

As mentioned before 43 users completed the entire set of questions regarding the usability of the WYRED Platform. According to the literature, the System Usability Scale (SUS) is reliable with a minimum sample size of 12 participants [15]. Consequently, the 43 responses received could yield fairly reliable results.

Although the SUS was originally developed to provide a single score that indicates the (perceived) usability of a system [14], subsequent studies pointed out the two-dimensional nature of this scale [22], allowing the calculation of the system’s learnability score (in addition to the usability score). Particularly, from the 10 items that conform the SUS questionnaire, items 4 and 10 can be used to score the learnability of the system being tested, while the remaining items are used to obtain its perceived usability [22].

Taking this into account, the SUS score was calculated following the scoring instructions [17] for every participant’s responses. Additionally, the learnability score (from items 4 and 10) and usability score (from items 1, 2, 3, 5, 6, 7, 8 and 9) were also calculated and transformed to fit in a scale from 0 to 100 (as in the original SUS scoring method, in order to allow comparisons).

The calculations yielded the following results [21]:

  • The average perceived usability of the WYRED Platform is 65.23, which can be considered as an acceptable SUS score, as it is close to the average SUS score (68.00) and falls around the 50th percentile (interpretation based on the studies done in [19, 20]).

  • On the other hand, the perceived learnability seems to be slightly higher (66.28) than the usability (64.97), both being acceptable scores.

The Fig. 2. summarizes the outcomes of the SUS questionnaire, also including the individual scores for every participant (represented by overlapping circles) across the three dimensions considered: SUS score, Usability and Learnability.

However, as mentioned in the previous sections, there are some considerations regarding these results: during the evaluation of the WYRED Platform a technical error was found. This issue was resolved over the next 8 days, meaning that some participants were exposed to that error.

Considering this situation, the users were divided into two groups (users who tested the Platform before the fix and users who tested the Platform after the fix). The SUS score was calculated for each group, reporting new results:

  • Group 1 (users who tested the Platform before the fix): the average SUS score for the 30 participants belonging to this group is 64.67, which is lower than the general one. Learnability and Usability scores are 65.27 and 64.48, respectively.

  • Group 2 (users who tested the Platform after the fix): in this case, the 13 users from this group gave an average SUS score of 66.54 (slightly higher than the SUS score obtained from the whole set of users). For these users, Learnability and Usability scores (68.27 and 66.11, respectively) fall closer to the average SUS score, based on the literature.

    Fig. 2.
    figure 2

    Visual representation of the SUS questionnaire results regarding the WYRED Platform usability and learnability scores.

    Full size image

6 Discussion

The average usability of the WYRED Platform for 43 students is 65.23, which, as commented in the previous section, is an acceptable score, although it is below the average of perceived usability in web systems (68) [16]. Specifically, this score is in the “marginal high” in terms of the acceptability ranges, and falls in the “OK” and “Good” interval as defined in the SUS adjective ratings [19]. The same interpretation applies to the individual Learnability and Usability scores.

While the results can be considered as acceptable, it also reveals that the WYRED Platform usability can be improved. The SUS score gives only an overall rating for the system’s usability, but the comments collected along with the survey give hints about what features to be improved or about what factors lowered the scores.

For example, there were complaints about the language of the Platform, for which translation is in progress.

In addition, some participants mentioned the technical error found during the testing (an issue that affected the login tool). This issue was mitigated and the users that completed the questionnaire after the fix rated the usability of the system better (with an average score of 66.54). Although the sample size of this set of users is lower than group that tested the Platform before the fix (13 users against 30 users, respectively), the results obtained from this segment of users is equally valid, because the sample size is enough to apply the SUS test (at least 12 participants [22]).

However, the collected comments also reflected that although some users rated low the Platform, they felt that the WYRED Platform is a practical and valuable tool.

To sum up, the application of the SUS test provided insights about the usability of the WYRED Platform and placed the system in the acceptable range of usability, with room for improvement based on the questionnaire feedback.

7 Conclusions

The WYRED Platform is a software component of the WYRED technological ecosystem that is focused on supporting the social dialogues about the digital society between children and young people. A first version of the Platform is already developed, but it is important to ensure the acceptance by the final users and to know their opinions in order to evolve the Platform and, therefore the ecosystem.

The System Usability Score has been applied to the WYRED platform in order to obtain insights about its usability. This test is not diagnostic; it gives an overview of the usability of a system. The WYRED Platform obtained an average score of 65.23, which is a score below the average (68) but is considered a decent result.

However, adding an open field in the questionnaire for comments gave hints about the improvements to be made in order to increase the usability. Also, it provided feedback about the general thoughts about the Platform, being the majority of them positive.

There are a number of important changes which need to be made in the WYRED Platform to improve the SUS score. Future works will be focus on solving some technical problems detected by the participants and the experts during the pilot experience, which could influence in the study results.