Posted on 07th January 2016
Insights from evaluating the Autism&Uni toolkit prototype
In December 2015 we ran an evaluation workshop at Leeds Beckett University, UK, to get feedback on the current version of our Online Toolkit. The Toolkit aims to help students navigate their journey from thinking about what to study, over applying to university, arriving and finally settling into their studies.
The workshop followed on from a co-design workshop we held in summer 2015, during which content and interface ideas were tested, discussed and further developed. Since the summer, we added a professionally designed visual identity to the toolkit, and we also created many ‘toolkit items’ – pieces of information that deal with a particular topic. Examples include:
- What university is really like
- How to reduce anxiety and stress
- Getting the right support when entering university
- What is Disabled Student Allowance (UK-specific)
- Telling people at university about your autism
(visual design by Washhouse)
For the workshop, we welcomed 5 autistic students, all currently studying either at Leeds University or at Leeds Beckett University. Marc Fabri and Penny Andrews facilitated the event at the Carnegie Pavilion building. On the picture below you can see that the room looked directly over the famous Headingley cricket ground, with a magnificent view (which did not distract any of the participants, we checked that!).
(participants gave permission to be photographed)
Workshop aims
We had some specific research questions for the workshop which informed the schedule and activities for the day:
- How do participants respond to the visual design of the toolkit?
- Which interactions lead to errors or confusion?
- How do users navigate to relevant information?
- How can related toolkit items be linked or grouped?
- How relevant is the content?
- How focused and detailed should a toolkit item be?
- Do users understand the narrative structure of a toolkit item?
- How specific to a country or institution do toolkit items need to be?
Results
Participants generally liked the interface design and the choices they had for customising the display of information. There were no negative comments about the length of items as such, although some said they liked to have control over how they viewed long pieces of information. We provided some choices for that and these were welcomed.
There was unanimous agreement that the toolkit must specifically tackle the key challenges students face. However, negative depictions of autism and a focus on difficulties were not welcome. Doom-and-gloom statements are definitely off-putting and increase worry, rather than alleviate it. Equally inappropriate are statements that minimise or gloss over potential challenges. Students told us clearly that striking the right balance is important here.
Participants were very adept at spotting typos, odd sentences, duplicate content, wrongly ordered information etc. – having such imperfections lies in the nature of a prototype, but it reminded us that the final product has to be spot on and carefully proof-read. Otherwise these little errors distract from the content and undermine the overall perception – and effectiveness – of the toolkit.
On the homepage of the toolkit we trialled a reminder bar that will help students get to their next lecture or tutorial on time, with the option to get directions and find out more about the type of session. This was positively received, and participants could imagine how this feature would help them get to places on time, safely and confidently.
(a mock up of the Reminder Bar)
There were quite a few comments on grouping and linking different toolkit items. The current prototype does this in a rudimentary way only, and during the workshop we collaboratively explored ways to improve those links. Discussions with the designer and developer have since started and this will be one of the key changes for the next version of the toolkit.
After the workshop we asked participants to score the usability of the Autism&Uni toolkit, using the established System Usability Scale. The toolkit scored 71 out of 100 which is encouraging as it is considered slightly above average in terms of usability – even at this prototyping stage (NB: SUS scores are not percentages; a system of average usability would be expected to score 68 out of 100).
Participants also told us that they felt comfortable to share their experiences and that their opinions were valued on the day. There was anxiousness initially but that generally disappeared during the course of the workshop. Students said they would be happy to share the toolkit with other autistic students, which is a great endorsement. Also, the majority of participants felt that they could identify with the student experiences described in the toolkit.
What’s next?
Overall we felt the workshop was a success and it has given us some very useful pointers for how to improve the toolkit. It has also reassured us that we are on the right path with both the content and the look and feel of the service.
Other project partners will now run their own evaluation workshops during January 2016, and once we have collated and analysed feedback from those we will organise a final evaluation event in Leeds.
We would like to thank the workshop participants for their time and their valuable comments.
Get involved
If you are an autistic student in the West Yorkshire area and would like to take part in our final evaluation workshop, complete the volunteer form.
If you work at a Higher Education Institution and would like to adopt the free toolkit at your university, please get in touch with Marc Fabri (m.fabri@leedsbeckett.ac.uk). The toolkit will be completed by April 2016. We can help with the set up and give advice on how best to support your students on the autism spectrum.
