User Evaluation – How to Proceed
Over the past two weeks I have been able to get a few user evaluation sessions completed, though I’ve sady had some bad luck organising it in the university itself. However, I have been able to feedback from several new and old computer science students, a history student and even a drama student. This gives me a wider array of feedback that hopefully grants more insight into whether this application has wider uses than just computer science algorithms, which is merely an example application of it. I will continue to attempt to get more feedback over the coming few weeks but holidays and exams will naturally make this more difficult to set up and execute. Because of this, I also do not have enough filled out SUS forms to really make useful graphs so, unless I no longer have the time for more evaluations, I will wait to post those results until later.
Following is a list of various problems that cropped up through the informal methods of evaluation; the scenario and informal questions. To each of these bullets I will attach the solution I have devised for the problem or various possibilities that I am still weighing against one another.
Problems with the UI
- The lack of an EDIT button: This arose both in the evaluation by my mentors and the evaluation with nearly every user. When prompted to post a new iteration, the first instinct of most of the users was to try and edit their existing solution with an improved iteration.
Solution: The solution is, naturally, quite simple: A button to quickly edit posts has been added to the interface.
- Creating a new iteration is too abstract: Together with the missing edit button, it was also pointed out numerous times that creating a new iteration is too difficult and not intuitive at all. The normal procedure is to create a new post and to click an Iteration dropbox. Selecting the post of which the new one is a new iteration will result in incrementing the iteration and posting it. In hindsight, the whole process is also faulty from a programming standpoint and far too complicated for what is actually just a different form of a “reply”.
Solution: The solution I have chosen will have another button added to the interface on the page of a post that allows a user to quickly generate a new iteration. Much of the information of the original post, such as the description and tags, are auto-copied into the new post so the user need only add-on new text and information.
- Notifications in awkward spot: Especially the history student had pointed out the awkward position of the notifications bubble. While it does flash red and attract attention in this way, the lower left corner of the screen isn’t really the place for an application element that is supposed to notifiy a user. I had initially chosen this location because it was empty space and was rarely, if ever, filled by anything significant.
Solution The solution is two-fold. First of all, notifications will be changed into a clickable button instead of that bubble, and the button will be located in the upper right corner, which is also often space that isn’t used. Because a lot of information such as the user console and the post headers are located nearby, the user’s eyes will be attracted to the notification tab much more often.
- Searching shouldn’t float: A complaint of many of the users is that some of the menu functions don’t belong in a floating tab that collapses when the user clicks anywhere else on the screen. The feedback is rooted in the fact that it’s common to accidentally click next to a button and if the advanced search functions, that might take some time to fill out properly, collapses whenever this happens, redundant and annoying repeat work will be required of the user. This complaint can extent to other functionalities in the future, garnering the need for a solution to this.
Solution: Possible solutions are many, but I chose to go with the simplest and most common practice one. Moving the advanced search function to a different screen. When the user wants to do anything more complicated than looking for posts containing one or more words, they are taken to a different screen of the application that does not invalidate their work by collapsing. It may have seemed like a minor complaint, but an application must do everything in its power to ‘not’ be frustrating.
- User Summaries aren’t useful: This was an issue with a few of the users. A user’s smaller profile displayed near posts, comments or in the top left corner of the screen displays various achievements and expertiese and, when clicked, expands to show a summary of that information in greater detail in the form of a list. This was deemed not useful because it is common for more experienced users to have a lot of achievements and would likely also have a lot of tags for which they have at least bronze expertise. The list would get needlessly big and thus fail to summarize anything.
Solution: The suggested solution was the make clicking it simply lead to the person’s profile, but this is redundant given clicking the person’s name already does this. I decided to simply alter the way the summary is laid out. When giving a summary of tags, only the most prominent tags and their expertise are displayed, leaving out very broad ones such as “Computer Science” simply because this tag would be present in just about all posts by an author specialized in that field of study. Achievements are a bit trickier and I am still mulling over ways to make them more organized and displayed efficiently for the user to gleam information from at a glance.
- Lack of Formatting options: This was a common issue among most users, either because programming code as flat test isn’t particularly interesting to read or because formating the problem domain can lead to a better understanding of the material. While basic formating options will be available, none of them will be related to code because the application isn’t specifically targetting computer science. Advanced formating options will not be included simply due to time constraints and because they are not integral to seeing whether or not the application does what it is supposed to.
Solution: There is none, adding more of these options will not aid the application in reaching its goals beyond basic usability. Time constraints won’t allow me to put so much work into text manipulation.
- “Friending” makes no sense: This one was only pointed out by my mentors during the user evaluation I did with them, but I found it a very valid point nonetheless. I was intent on providing an interface for friends and the like, but such request make little sense because there is no real shielding of information from non-friends and, if there was, it would destroy the purpose of the application as it must share solutions equally among all users. they suggested a Twitter-like follow mechanism instead that changes little to the interface but makes much more sense conceptually. A notification is given to anyone who is followed and following is just a matter of clicking a button for the interested user.
- Will Challenge Tracks be used? Once again, a problem pointed out by my mentors, but not the other users. Why would someone create a challenge track for himself when it serves no other purpose than granting achievements?
Solution: The solution lies in the blog posts I have made before about user-generated content and meaningful achievement. Many people have an urge to share and created something yourself and subsequently sharing it with other users, most notably followers and people of equal interest, would yielf a beneficial effect and that alone would urge people to use the feature. Furthermore, it can serve as a more long-term, advanced challenge that promotes greater learning through the same iteration functionality that is supported by the application. Each step is clearly defined and a reward is given along the way. One thing I will add to the application’s explanation is that a teacher can use challenge tracks to help explain the material in an iterative fashion, preventing a complicated information dump and preferring to take the approach of letting students figure things out for themselves with the milestones and achievement “hints” as help.
- Useful on its own? A concern expressed by several test users was that, especially for certain fields of study, the application may not be useful on its own. People will not be inclined to use it without a first push into its direction. Unless the user is tremendously interested in the material, students will not necessarily see this application and think, right away, “this can help me”.
Solution: This concern is an important one as it may indicate a fundamental flaw in my application. While the general concensus so far is that it would be useful and students would use it if the motivational aspects work in the long run, the very real problem exists that it would be overlooked in favor of simply studying the text book on the subject matter and nothing else. This indicates that an initial push towards the application must be made so the student can experience it and properly make out whether or not it is something that can help them. This can be achieved by making this application related to one or more subjects at the university in question. Professors would post some challenge tracks or point out the application for students so they know of its presence and are more inclined to try it out. It also indicates the typical problem of having a cold start. In the beginning, the application will be empty and it will take some time for a good repository of solutions, tags and problems to fill the database and make the application truly useful. While the above solution seems like something that could work, further study will be needed to point out whether this fundamental flaw is truly an issue or just a minor concern.
Next I will go over most of the informal questions I asked each of the users and summarize the general feedback and answers I received from them and how I will use that feedback:
- How is the user interface experienced, in general? Is everything readily and easily accessible? Are there any immediate impairments?
The user interface was, in general, positively received. Most of the elements, apart from those discussed above, were visible, centralized and easy to access. This made the application intuitive, quick and easy to use.
- How are the game elements experienced at first glance? Do they serve their purpose, even in an early stage? Which are pointless? Which simply need refinement?
The reactions were largely inconclusive here. In general, the elements were well received and they would urge users on to be involved with the application. Often, posts are made and published but then forgotten and not much else happens with the interplay between the users. But the challenges and other game elements urge the user to not forget about his or her posts, or those of others. However, no in-depth feedback could be given because the game elements did not really do what they were supposed to in such a short test. Only long term or fully implemented tests can show whether they need refinement, change or removal.
- If you were following a course that had, at the very least, a significant section on computer algorithms, would you use this application to better understand the material as well as broaden your knowledge beyond the material?
Reactions to this were hard to come by. The non-computer science test users could not really imagine how the application would work because of their lack of knowledge on the subject, meaning the question itself was unfortunately formulated by myself. But two of the computer science students, one of which majors in video game design, had to admit it could have made their life a lot easier as their current occupations often have them altering existing algorithms to suit their needs and this was a difficult transition at first due to them having mostly book knowledge and not practical, flexible application know-how. This is a good result, but not at all conclusive, but it does hint at how the application may be capable of doing what it needs to do.
- Is the process of challenges, rating and creating your own content to be considered enjoyable, even though this is on the short term?
As with the second question, most users could not give an answer to this question. The interface surrounding these elements was fluent and easy-to-use, but whether it was enjoyable could not be gleamed from the short evaluation scenario.
- Are user-generated achievements relevant and something you would be willing/happy to show on your profile, or are they meaningless?
Some of the users stated that they would indeed show achievements generated by challenge tracks they would hand-pick on their profiles. The main reason for this being that they would often be achievements they could be proud of, actual proof of achieving. Others, however, did not see why these elements were even present in the application and were largely uninterested in them. This does show that it is indeed important to include various game elements as people’s tastes differ widely. Those users could perhaps enjoy other elements more and it is my job to try and include something for everyone (within the realm of the possible, of course).
- If given a choice, what platform would you use this application on the most?
As I expected, all users chose a pc as a target platform due to the heavy text related core of the application. While a mobile version to view new posts would be useful, typing up a full solution on a mobile platform that isn’t at least as sophisticated as a touchpad, would be impractical and sometimes impossible. Given these unanymous answers, I will develop it for desk- and laptops.
- Would you use any results, achievements or expertise gained while using the application on other online social sites, if possible, such as LinkedIn or Facebook?
This idea came from a suggestion of one of my mentors and got me thinking about how important it is for students to show their skills to the industry. One of the best ways of doing so is to take advantage of the pervasive presence of social networking. A lot of employers check applicants’ facebook and other social sites to see what sort of person it is and how their skills relate to what is demanded by the job. If expertise and relevant achievements, with links to actual, tangible and reviewable solutions are present on professional sites such as LinkedIn, it could prove the break a student needs to get a certain job. All but one user agreed that this would be a useful addition to the application.
These are all the results I have compounded so far. Naturally, I still lack proper feedback from people in my specific branch of computer science, which would prove most valuable for the example I am going with. But I do have people from very varying backgrounds, each of which pointed out issues others had not and thereby giving me food for thought. All in all, it was very educational and I can finally move on to implementation and getting a first iteration up and running that will show, digitally, how the program will work.