Rocket surgery for PickCells

To run our usability evaluation, we followed Steve Krug's highly-recommended and very readable book "Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems". This book describes a practical way to carry out usability evaluations with minimal overhead.

Participants

We recruited six researchers to help us with our evaluation. Three participants were regular users of PickCells on a daily, weekly or monthly basis ("experts") and three had never used PickCells before ("novices"). We refer to these researchers as participants, since the subject of the evaluation was PickCells, which they were helping us to evaluate.

Were six participants enough? Steve Krug comments, in his complementary book "Don't Make Me Think", that even "Testing one user is 100 per cent better than testing none". Craig Tomlin agrees that "ANY usability testing is better than NO usability testing". These comments are based on findings that even a small set of users will encounter the most serious usability issues and also that we are not trying to "prove" anything, we are not undertaking research, but we are trying to get a list of problems that need to be fixed!

Tasks

Three tasks were selected for the participants, which would exercise the common features of PickCells:

1. Analysis of nuclear markers. Quantify the level of expression of differentiation and stemness markers in control versus treated conditions to determine if any difference can be observed.
2. Analysis of neural rosettes. Determine the number of neural rosettes in a sample image and the average number of cells in each rosette, then extract the percentage of cells engaged in a neural rosette within the sample image.
3. Correlation between protein level and number of RNA molecules. Given a PickCells experiment with a work-in-progress analysis to investigate if a pro-differentiation gene of interest is post-transcriptionally regulated, determine how the number of mRNA molecules of the gene correlates with the level of protein in stem cells as they differentiate and then complete the analysis.

Someone who has been trained to use PickCells should have no problem completing Task 1. However a novice user, with no prior experience, may have problems unless they follow one of the website tutorials. Tasks 2 and 3 expect more creativity from users, as they require knowledge both of image analysis and PickCells.

Each task was broken down into steps of discrete activities to be done using PickCells, but without mentioning features of PickCells specifically.

Participants were assigned to tasks based on their expertise. The three novices and one expert did task 1. One novice did task 2. And two experts did tasks 2 and 3.

Method

We spent 2 hours with each participant, who was asked to work through their assigned tasks using PickCells on a laptop. They were also referred to the PickCells website and forum, which they could use if they so chose.

Each participant was asked to say what they were doing or trying to do, what they were looking at and what they were thinking (a "think-aloud" protocol). We sat with each participant, taking pen-and-paper notes. The laptop was hooked up to two large screens to make it easy for us to see what they were doing, without the need for us to crowd around the laptop (and the participant).

We captured both their comments and the screen display using the open source Open Broadcaster Software (OBS) and a USB microphone to serve as a record of the evaluation for future reference.

We had a few problems with our setup, which we addressed as they arose:

• OBS would lose audio for long recordings. The loss coincided with times when PickCells executed CPU-intensive operations. We stopped and restarted OBS more regularly.
• Multiple workspaces were used on the laptop and OBS captured the console from which PickCells was launched and its splash screen only, rather than its user interface. When discovered, we ran everything from within a single workspace.
• Some task descriptions were too vague or ambiguous and some participants felt there was pressure to complete each task in full within their evaluation slot. We refined the task descriptions, provided clarification when asked for, and reassured participants that it was not a time test, that we were interested in their experiences with PickCells.

Observations

Both experts and novices remarked on the utility of PickCells. A novice commented that they could see that "it's going to be very useful for my job" and an expert commented that it is "powerful".

All three novices commented on the need for tutorials. One said that PickCells has "lots of potential" but that "honestly, I need a tutorial". Another commented they "should have" watched the online tutorial, and the third that "I was understanding what I was doing but there was a lot of information" and that they "ha[d] to keep going back to the tutorial.".

Similarly, both novices and experts commented on the value of seeking help from others. One expert remarked that while the learning curve is high, users can always get help from other researchers. Similarly, a novice commented that "if I don't know how to use [it] I can either turn to [a colleague] or turn to the web".

During task 1, an expert entered segmentation parameters based on their prior experience doing similar tasks, even though they weren't familiar with the example image set used for that task. However, for task 2, which they were not familiar with, they opted to use the default parameters as they "assume[d] [they were] somewhat optimised". Similarly, a novice who had no prior experience of PickCells, relied upon its default values.

Experts often expressed their intentions and interpretations in domain-specific terms (eg image, staining, segmentation, nucleus, radius, volume) rather than software-specific terms (eg buttons, fields, dialog boxes etc.)

Usability problems

Over 50 distinct usability problems were identified with PickCells, and 10 with its complementary website and forum. The most significant of these issues, which were encountered by more than one of the participants, were as follows.

Creating and loading experiments

Both experts and novices had problems in creating new experiments and loading existing experiments within PickCells. For example, both an expert and a novice tried double-clicking on an experiment name to load the experiment (which does not work), before right-clicking to select the required command from a context-sensitive menu. The file browser dialog box was also problematic. For example, it has a drop-down list labelled "All Files", when the user needs to select an experiment folder. As creating and loading experiments are the first actions any user will do, it is important that these problems be addressed.

Viewing images

All the novices and one expert had issues with loading both raw and segmented images. They all expected to be able to view the raw images and segmented images once loaded. In many cases, they repeatedly expressed this desire, checking tooltips, menus and the website for how to view the images. One novice expressed an intention to reload the images, which, if they had done so, would have caused problems in the subsequent analysis. Another novice tried to view the images via a third-party application. These suggest that images should be viewable within PickCells.

Updating the metamodel

At the heart of the PickCells is a metamodel, a graph-structure which holds information about images and the analyses performed on them. The metamodel is visualised within PickCells, in a metamodel view.

When images are loaded, or other operations complete, the metamodel is updated, and the metamodel view is refreshed within the user interface. However, this was not always noticed by the participants, for example when images had loaded (as above) or when they'd triggered other options. One novice commented that "I'm lost...I thought I measured the intensity but nothing happened and I'm not sure what to do now to get the results if something was done". Rather than relying on a user noticing that the metamodel view has updated to determine if an action has completed, PickCells could show a dialog box stating that the action has completed and refer the user to the metamodel view.

Progress reporting

Related problems arose in terms of how PickCells informs users that it is currently running some operation. Some progress bars are shown within dialog boxes, but others are shown in a border at bottom of the PickCells user interface. Novice users were not always clear whether PickCells had started, or was continuing, some operation. Two novices also mistook a bar that displays PickCells' heap usage as a progress bar.

One image analysis module has a Preview button which updates an image in light of changes to parameters they have made. When the image is updated, a "Done!" label appears. However, if the user updates their parameters and forgets to click the Preview button, the "Done!" label can be persist from the previous invocation, giving the user the false impression that the image has updated, when, in fact, it hasn't.

Selecting modules via icons and tooltips

Both experts and novices had problems with PickCells vertical toolbar, which has icons to invoke PickCells functional modules. This toolbar was cropped by the bottom of the screen so that not all of its icons were visible and there was no scrollbar to view those that were obscured. Once undocked, the toolbar's tooltips are cropped to the boundary of the undocked toolbar, "Really annoying!" one expert commented.

While icons in both the vertical and horizontal (visualisation modules) toolbars are distinct, one novice commented that they are too small to be able to usefully discern their differences and quickly see their meaning. This, in part, may explain why both novices and experts made extensive use of tooltips on these icons, moving the mouse pointer over the icons and skimming the tooltip text. This was especially the case for the vertical toolbar, with the icons for functional components. However, this activity was inhibited by the fact that, as one expert observed, the tooltips "are quite lengthy" with "references and stuff" and that a user doesn't need so much detail. Some tooltips are so wide they span the width of the display. In addition, not all icons, buttons or fields have tooltips.

Richness and complexity

Users can perform analysis and visualise data by selecting nodes and paths within the metamodel view. One expert commented that the complexity of the underlying metamodel is worth it given PickCells's functionality. But, they cautioned that while they understand the concept of paths in the metamodel, defining paths just to visualise data is not especially intuitive.

Another expert made extensive use of the metamodel view, and data tables accessible via its nodes. The third expert commented that when they had first used the metamodel, it was "extremely complicated" but, over time, it proved to be "quite useful" especially if using an experiment someone else has started. They commented that it was "quite intuitive, once you know how to read it". In contrast, a novice commented that they had "no idea" what the metamodel was about and that they should have read the website first.

Creating even simple plots also proved to be very challenging. All participants experienced numerous issues with filtering and plotting data, due to the richness of the metamodel view and the PickCells query builder, with which a user selects data from the metamodel for plotting.

More generally, the functional richness of PickCells was a source of issues for the participants. While experts were correct in the actions the intended to perform to complete a task, they could not easily identify how to map their intended actions to the functionality offered by PickCells. For example, one exprty commented that they were "trying to figure out how to make the correct associations between the nuclei and the clustered centrosomes" and that "I think I've made the correct entry for clustering the centrosomes", but "don't know how to just pull out the nuclei that correspond to the clusters."

This suggests that additional guidance, in the form of tutorials, advice on common workflows and HOW-TOs for tasks within these workflows (e.g. select a path within the metamodel view, segment images, compute intensity, create 2D bubble plot) should be provided on the website.

Within PickCells, embedded help, in the form of labels and tooltips could be provided, along with context-sensitive help via, for example, floating dialog boxes (as done in Eclipse) or via links into complementary pages on the website (as done in Microsoft Word). One expert "wish[ed] there was a little button called 'help' that would take you to a place in the website that you can read about it".

A related issue is that all PickCells functionality is enabled by default. Consequently, users can suffer from decision paralysis or uncertainty in terms of which function to invoke. One possible solution would be to disable features by default and only enable them when PickCells is in a state where the feature has all the information it needs to run.

The website itself needs revised. One novice commented that they felt that its content was a "bit cyclical" and that "to do A I need to do B which needs A" and that there were "layers and layers of stuff...more more and more doors...you do this because this is what you do" but that "there is no explanation of what you do."

Consistency

Inconsistencies were noted in terms of labelling and phrasing both within PickCells and between PickCells and its website. One expert expressed surprise that when they clicked the 2D bubble plot icon a plot titled "2D scatter plot" appeared. Another expert commented that they "struggle with the words 'linked' and 'connected' because that's the same thing". A novice commented that the website shows a "define dimension dialog" but a tooltip showed "change dimensions". Care should be taken to ensure that PickCells is consistent both within the software and between the software and the website.

What next

We are now implementing changes to PickCells, its website, and forum to address the significant issues arising from the evaluation. We will also produce questionnaires for users and developers to allow us to capture usability and other PickCells-related information from PickCells' user community. We will run another usability evaluation in the New Year, and will blog about our experiences in a future post.

You can also read an earlier blog post about PickCells: PickCells and exploratory image analysis in cell biology