Designers attitudes towards steps in the design process

⚠ Exploratory Research ahead

This is a little study that explores Designer’s attitudes to steps in the design process

The idea was based on the observation that some steps are part of almost any design process while some are left out. So I wanted some data on this. In addition there were two additional hypothesis: Designs steps which are less fun are done less often and design steps in which the designer does not feel in control are less fun.

Gathering data

From books and some interviews I created a list of design steps, and created a survey. For each of the design steps I asked

  • how frequently the participant does the step in design (ranging from »never« to »in every project«)
  • how much fun the activities for this step in the design process are for the participant.
  • how much control the participant feels to have in this step of the design process.

These attributes were rated on a 7-step scale. Higher values indicated more experienced fun, more perceived control and doing the activity more often. In a little text before the survey questions, I explained what I understood as »(not) having fun« and »(not) being in control«.



To find out how core activities of design fare compared to research activities, three Designers coded the steps as being: research, design-itself, something else¹. I created the average of the steps which were coded »design« or »research« by all three designers.


Design-itself activities are done more often, seem more controllable and more fun to do. Some ideas in relation to this are using methods for research that feel more productive and controllable. Co-Design and research-by-mapping (mapping emotions and social life, describing workflows) may be ways to do this.

There is a significant correlation between control, fun and frequency – so if designers felt they are in control they had (in average) more fun and if they had more fun, they did the step more frequently.

Since the research was not experimental, we can not draw conclusions about causation here. This would need further experiments, though drawing from interviews I made with graphic designers and participants of a human centered design class, it seems to be a promising direction.

One interpretation that can be drawn is this: In design-itself-tasks, the designer feel in control of the situation and that the outcome is mainly depended on ones own actions, while in research tasks the outcome is not sure (you can ensure a scientifically sound experiment or the like, but the outcome is open).


As I said the research was not experimental and explored only correlations. In addition there are some shortcomings:

  • The data is based on answers of only 20 people about half being graphic designs, half being product designers.
  • The order of the questions was not randomized
  • The questions were ordered by the design process steps, and asked for the three attributes of a design task (Control, Fun, Frequency)  in each step. If someone would choose the same rating in each of these subsequent questions  (out of boredom or to complete quicker) there would be a seemingly perfect correlation.

When I continue the research I would thus

  • Get a bigger sample
  • get a more diverse sample
  • Order by attributes as main category first, than ask for the attribute for each step. the order should be randomized.

If you are interested in doing research on the topic too, I’d be greatly interested in cooperating.

¹ »something else« was often described as »communication«.

Activity co-documentation

In a previous blogpost I demonstrated some methods for mapping social relations as well as emotions/liked-and-less-liked-parts-of-a-process. In this blogpost, I will show some ideas on documenting the processes with your participants by writing step-by-step instructions together.

I’m not the first one to try something this, for example CARD¹  or Semistructured Models². But I wanted something more lightweight.

Here is what I came up with:

Step 1: List tasks

First we need an overview of what the participant does. It would be ideal to collect these activities³ together with the participant by observing him/her, however this is not always possible and even if we, lets say, would do a few hours of observation, we still might miss important activities (like the setup when starting work or the cleanup routine when ending it).

My first ideas was to ask what the participant does and to write it down with the participant.


Simple list of tasks. Reads: Washing dishes, choose music, clean tables, put up chairs (?), open bottles, mix drinks, advice for choosing drinks, count money

However, this was a bit too general in my eyes. To make it more concrete, I asked for things that the participant was doing just now and some minutes ago, too. This makes it somehow more reliable since it is likely that recent activities are remembered. In addition, these remembered recent activities have actually been done in (possible) contrast to activities in general which may be written down because they should have been done.


To ease this for me, I created a little template:


task list template. Reads: Tasks and Activities; What? What happens if it is not done?; Just done; Done in the last 30min; Activities in general; [Some example]


task list template in use.

You see that there is a second column, asking for “effects if the activity in not done”. This can help to focus on the (seemingly) most relevant activities first.

Before creating the list, I show an example how the result may look like. The example is concerned with a very different domain (to avoid biasing the participant), so people working in a cafe got a »How I repair a computer«-Example from someone who volunteers in a hackerspace repairing computers (hmm, that someone is me).

Step 2: Describing the activity(s)


process documentation template

After having an overview of activities I choose the one that appears most relevant and ask the participant to document this activity with me. I frame it as possible instructions which could be used to teach somebody else the activity. I show an example for this too.

I provide a template, with an »activity name« and a »notes/sketches« column, and fields for basic infos like the name of the activity, the trigger, the result, the risks/problems and the motivations to do the activity.

When documenting the process, the participants only write down the general steps (I hoped they would include some sketches, hints etc. too). After they are satisfied I go through their list and ask questions:

You wrote “Distributing team work” – how do you do this?


This point “Let the milk cool down” – what happens if you skip this?


documentation & observation combined

In addition, I may ask for a demonstration:

Can you show me that coffee-container-thingy you need to clean?

…How is that thing attached to the coffee machine?

I add the information I asked for as notes and sketches, either in another color or by creating references to another note sheet using (latin) numbers.

Prozessbeschreibung U9


The latin numbers link to tasks the participant wrote on the template sheet. We ran out of space, so we used another sheet.

At the end we created a documentation of the work process on one or two pages.

The data is mainly written text (as opposed to audio or video or researcher’s memories)  so there is no  transcription required. However, I strongly recommend to review your notes afterwards, add possibly forgotten data from your memory and to rewrite hard-to-read words so that the document is useful at some future point.

I still think that a interview/observation gives more opportunities for experienced researchers. However, the method described above may provide beginners with some structure and still leaves room for own questions and ideas. A motivational advantage might be that an actual artifact is created (the sheet of paper on which the process is documented) instead of just having collected “more data”.


- List of processes
- Documentation of an activity

Under CC 4.0 BY

Both in German. Write me a comment, if you need a template in English.
Linked templates are now in English.


  1. Tudor, Leslie Gayle, et al. "A participatory design technique for high-level task analysis, critique, and redesign: The CARD method." Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 37. No. 4. SAGE Publications, 1993.
  2. Herrmann, Thomas, et al. "Semistructured models are surprisingly useful for user-centered design." Designing Cooperative Systems (Coop 2000) (2000): 159-174.
  3. »Activity« is used here like in everyday language, it does (most likely) not exactly match the definition of an »activity« in activity theory

Text and images are licensed under a Creative Commons BY 4.0 License

The Educational Power of Examples

I think that teaching-by-examples is an awesome and empowering method. Instead of a universal principle given by some authority a way to solve a problem is suggested.

Example for an Example (meta, yey!):
While the interface design principle  »Visibility of the system status« may be a great advice for interface designers, the statement can be interpreted in many ways (what is this "status" actually? How can it be shown?). But by showing examples of the principle in action, it becomes clear what is meant by the principle.


shows status of the system


does not show the system’s status well


The solution seen in the example can be  applied to one’s own work. Thus the learner has a sense of achievement (since one can easily make one’s own example-derived solution work). If the suggested method works, the learner can start to experiment and find other or better solutions and can compare to the method that was taught in the  example.

FlickrUserAllen LI CCBy20163551685_fa4cb8ed2c_z

Can you know the design process just by discussing the final product? (Picture by Alan LI,Creative Commons BY 2.0)

In particular in design where much of the knowledge that should be acquired by future designers is tacit and thus can’t be just learned by heart from books. Teaching by example makes much sense and is already practised. However, often this is done by dissecting and  and discussing great, finished designs: A particular chair, a building etc. This has advantages over only discussing abstract principles of good design but for teaching how to design it has a major shortcoming: Great designs don’t just fall from the sky. They are created in a process (as messy as such a  process might be). Such a process is can’t be (easily) reconstructed by looking at a final product.


To learn »how« to design it makes sense to show how other people design not just  what in particular they came up with.

Building upon the »show system status« example above: It is not only interesting to see an example of the principle itself but of the context in which such knowledge may be important. Let’s say, an interface is tested an a certain function is not found, in which case it could be worth a try to check if there is  visibility of the system’s status.


Users forget which file exactly they uploaded. How should they name it then?


As a possible solution, a thumbnail of the uploaded image as an application of the "visibility of system status" principle.

While finished designs are shared in great number and on several platforms, work-in-progress examples are rare. Often it is only stated that this-and-that process works great; In-progress-designs, visual examples, solutions developed in parallel or discarded ideas remain hidden.

There are some reasons why we have few  examples in design which show work in progress and explain the decisions and actions involved in progressing in the design process:

1) The Genius Designer
Design is sometimes framed like art and presented as a mythical process consisting mainly of incubation of ingenious thought, thus there is no process to present, at least not one from which non-geniuses may learn.

2) Portfolio culture and perfectionism
Caring for every detail is sometimes said to be the hallmark of great design. But in progress work is not work has perfect details yet.

3) Investments and Benefits
While sharing a great example may benefit many, the benefits for the one who shares it are small: A design- in progress does not look great and it does not make a great page in the portfolio.

However, depending on the area one wants to work on, these obstacles may be overcome. Partly by a culture in which designers are no longer associated with supernatural genius-powers and working always perfectly; partly by employers who care for the awareness of process, the reflection of ones work process and the peer-to-peer education among designers.

I think that the advantages of sharing examples are to big and the available material to few to ignore the unused potential in sharing more of our designs and thus educating each other in a way which is instructive as well as empowering for the learners.

Worked Example Effect (Wikipedia)

Visual Interviews and Visual analysis

a beginner friendly method for user need research

While interviews and observations are popular methods for data gathering they can be hard to handle for beginners: Asking open questions, avoiding influences and managing the flow is hard. Graphical templates on which the reserach participant sketches or writes can help the researcher and the participant alike by providing a scaffold for the process.


The basic process

One example would be graphing good and bad phases of an experience over time – like this:

The template is just a sheet of paper with the axis and their labels (and possibly a miniature example of a finished graph)


The researcher needs to introduce the process to the participant:

I am interested in the activities and experiences you like or don’t like and how they line up.
Could you draw a graph of how you felt during the project and write which activities were connected to these feelings? A finished diagram can look like this [shows example]. While you draw, just explain me what you draw so I can understand it better.

Or more abstract:

  • State your interest
  • Explain how to use the diagram
  • Ask for explanations during the drawing

While the diagrams itself are an important outcome, don’t forget to note the participants utterances and/or to make an audio recording.

Make it suit your needs

There are many different forms of mappings which you can use for user research.

Some examples:

Diagramming Feelings

Just like in the example above you can ask your participants to draw the course of their feelings over a specific time and to annotate this graph.



Social diagram

Ask to map the connections and tasks of people the participant works with.



Note that the diagram does not just include individuals the participant directly worked with; Books are mentioned too. If participants ask if they may include something (e.g. books instead of only persons) encourage them to do it and use the additional data.

Process diagram


Ask to draw and annotate a diagram of the workflow: What tasks need to be done? Why? How are decisions made?


If possible ask for demonstrations of the activities while they are added.

… your own creations

It is a great idea to create your own template is existing ones do not suit your needs. Just keep in mind that it should easy to fill out and test it at least by using it yourself or ideally in a pre-reserach session.


I think that this has several advantages for beginners in user research since the template provides some predefined structure:

  • Participant and Researcher alike feel more secure it is rather clear what can happen and what is expected
  • Asking about processes and motivations can be tough for beginners; the template can help to get to know about these.

In addition there are some advantages because of the graphical nature of research

  • It is easy to point to data like »you wrote/said/drew… this – what does it mean?«
  • The data can be analysed graphically by comparing patters.


Annotate the diagrams

Annotate the mappings directly after the reserach session when your memory is still fresh. Supplement the drawings from you memory and add utterances which you remember and rewrite annotations which are unclear. If you don’t you miss out some data and the diagram will be hard to understand when you can’t decipher the handwriting of the participant.

Search for patterns

To find patterns across participants, put all diagrams of the same kind side by side. See if there are similarities or contrasting patterns, find reoccuring data as well as unique events.

Example Analysis on the diagrams above (click to enlarge)

  • In all but one diagram the onset of a project seems to be a good experience
  • In three of the five diagrams there is a significant decline after the projects first, motivated phase. The reasons are: Seemingly unsolvable problems, project not going according to the plan and unhappy clients. A commonality is that the named reasons for the bad mood are seemingly out of the control of the participants.
  • for all participants the project ends good, while four of them seem to be very happy at the end: Finishing seems to be a good experience.

reasons for being happy:

  • project onset
  • ideation
  • design works like hoped or expected
  • project end

reasons for being unhappy

  • »unsolvable problems«
  • client does not like the design
  • self critique
  • insufficient starting material

Wrap up

I hope that the examples for possible diagrams and their analysis gives you an idea of how to use diagrams for your own research. If you do, write some comments on your experiences.

Edit (9.2.2015): The page I hosted the .svgs on is down. But they found a new home, so download the social map and the feelings map form the Medienwiki.

Edit (10.1.2015): Since the images are only .png, here the original svgs. Have fun using and  improving them (according to Creative Commons 4.0 BY).

 Edit (2.1.2015): The text and the photos and illustrations may be used according to the Creative Commons 4.0 BY License



Online: A Beginner’s Guide to Finding User Needs

My Beginner’s Guide to Finding User Needs is online.

The text is aimed at students and professionals who want to learn about qualitative user research. It is written in a hands-on manner and the described methods factor in that you might neither have a big team nor a big budget.

If you want to suggest improvements, you can write me (dittrich.c.jan AT or file an issue on github.

Update: 2.11.'14: Tippo

Design as a Structured Process?

I have a Love/Hate relationship with design method frameworks, these diagrams and flows showing what happens afters which step in a design process: »Human Centered Design« or »Design Thinking« to name two.


A Human Centered Design Cycle

Phases in Design Thinking

Design method frameworks provide an overview of the process and give suggestions of how design can (or should) be approached. This includes researching user needs before creating solutions and testing prototypes before investing lots of time. They might  be particularly useful for beginners who can use them as a guide for their design work.

But these models may not match what designers actually do: Having a model of how a process can or should be like is one thing, putting it into practice is another. Thus it would be interesting to know if these suggestions are followed and if not why.

To find this out one could observe designers and protocol their actions. Such »protocol studies« are an established method in the field of empirical design research. Helen Sharp and her co-authors used such a method to study novice interaction designers. All students were taught »the definition of usability and  user experience goals, exploring the problem space and challenging any underlying assumptions before starting to produce a solution«. Despite of this, the findings suggest that »Participants immediately suggest a solution
when they are given a design problem.« [1]

The study is not big, but it fits into the findings concerning other creative and problem-solving activities: Mechanical Engineering, Circuit Design, Architecture etc. In these fields, studies suggest that design does not happen in clear cut, sequential steps. In particular, the problems which are solved are often seen as »moving targets« : The problem which is to be solved is often only preliminary defined, and not exactly depended on the external requirements (Client x needs y) but co-evolves with the solutions [3]. So design is highly opportunistic and it is opportunistic not just in areas where something should look stylish but in seemingly »hard« areas like mechanical engineering or programming [2].

The usefulness of models is thus debatable: They may help (novice) designers to include all necessary steps in a meaningful order, so that they don’t get stuck (e.g. in deciding which colors to use before they know if they want to design a magazine or a website) [3] but on the other hand they may wrongly suggest that steps in design are not interwoven. In addition, following a model may add mental bookkeeping costs [3] (e.g. we are now in the prototyping phase with part x but part y is still… so we need…).

Design method frameworks can help (novice) designers to give their project a meaningful structure. But to be more useful they should consider as well the opportunistic and and intertwined kind of work that seems to be part of most design practice.

  1. Helen Sharp, Nicole Lotz, Richard Blyth, Mark Woodroffe, Dino Rajah, and Turugare Ranganai. 2013. A protocol study of novice interaction design behaviour in Botswana: solution-driven interaction design. In Proceedings of the 27th International BCS Human Computer Interaction Conference (BCS-HCI '13), Steve Love, Kate Hone, and Tom McEwan (Eds.). British Computer Society, Swinton, UK, UK, , Article 18 , 10 pages.
  2. Willemien Visser. 1990. More or less following a plan during design: opportunistic deviations in specification. Int. J. Man-Mach. Stud. 33, 3 (August 1990), 247-278. DOI=10.1016/S0020-7373(05)80119-1
  3. Cross, Nigel: Design Thinking : Understanding How Designers Think and Work. Bloomsbury Academic, New York, 2011. ISBN 978-1-847-88846-4.

- Added References on 28.10.2014


Sketchy Theme for Prototyping in Code

A major problem of  prototyping in code is that the prototype may be mistaken for a (nearly)  finished product and that the focus shifts from the interactions towards colors and fonts.

Some mockup tools (Pencil, Balsamiq) provide sketchy stencils to create mockups which look not finished. I tried to recreated similar style which can be applied to Bootstrap Designs as a theme, including styles for buttons and widgets.


…testing as drop-in with the Freifunk-Splashpage


some generic bootstrap example

UPDATES: 21.10.14: Added Freifunk-Example

Audio-Recordings of Interviews: Indispensable or Nice Supplement?

Reading in the literature on Interviewing almost everybody recommends audio recording [1]  as well as additional video [2][3]. Nevertheless I wondered how important these recordings might be especially considering that some people never learn to value a user centered design process if they find their first steps too tedious.

So: Is it worth it? (considered that you might have few time/money)

The answer is »it depends« (as usual) but I wanted to get some insight into what it depends on. Sadly I could not find the key to the rocket science building, so all you get are my experiences in several small case studies.

What I did

I did several interviews as part of three different projects. In each interview I took notes and recorded the audio. After all interviews I complemented the notes from memory as soon as possible. I  filled in gaps and extended bullet points to more verbose descriptions of what was explained to me or observed. Than I transferred the information in a text file.

I listened to the recording as well and wrote all coherences, statements and explanations  from the audio in a text file too (thus, no word-by-word transcript). Having my in-interview notes and my from-memory-notes and what I got by going through the recordings I could compare the results of each step.

What I found

Going though the audio made a difference – but not as big as I assumed. What was added was mostly minor details. Among the five Interviews there was one in which I relied heavily on the recordings. In one of the interviews I had seemingly conflicting statements; I was able to understand and clarify this by listening to the recordings.

The main points were already in the written notes and/or their complements.


Overall, the notes and their completions after the interview already provide a usable basis for user research, even if no audio was recorded. However, if it is possible, you should record nevertheless. It can happen that your notes are not useful (as it happened to me in one of the interviews) and you may need to review some of the recording to resolve conflicting statements or get a better understanding.

Open Source Social Reserach

I rely mainly on open source Software. I would use proprietary software as well but I think open source has  some advantages:

  • to be able to just install software on several computers
  • being able to share data with a team easily.
  • ease the entry: nobody is enthusiastic about trying something that comes with a hefty price tag.

However while there is open source software for almost anything it gets a bit sparse in regard to software for qualitative methods. Here is what I use:

Easytranscript: A little software for transcribing audio. Audio can be controlled by shortcuts. Especially useful is the setting of timestamps in the text and that clicking on that parts of the text go to the according part in the audio file.
Before using Easytranscript I used VLC together with some editor. When doing this, best use global shortcuts in VLC (e.g. some F-Key for start/stop and jump back) and word autocompletion in the editor.

RQDA: A R-based software for coding texts and retrieving text parts which have been coded. It has a simple GUI although its not totally conforming to standards. The installation requires R and GTK+ (the installation is described here –  needs some dependencies, but no manual config required)
RQDA only takes plain text and not the .rtf written by Easytranscript, but copy and paste to a text editor can resolve the problem.
An alternative to RQDA might be CATMA  ( which runs on every computer that has Java installed. I have not tried it yet.

Open Office Writer: Serves me well for writing reports, consent forms and whatever else I want to print on paper.
Together with Zotero it is quite good for writing scientific text as well. Many Linux-Users will opt to LaTex for that though it is harder to get into.

little hack

When shopping in my local supermarket I always wondered about one thing at the checkout: The cashier was always pushing one of these separation-bars back, along the movement direction of the conveyor belt. It seemed tedious: after a few movements of the belt the separation-bar was at the belt’s end and was pushed back again.

So I asked why. It turned out is it a little hack: The conveyor belt stops automatically, when something passes a light barrier.

Probably this is implemented in order to make moving the belt easier (no need for finding a good point to stop); it will prevent as well that items get accidentally pushed on the scanner.

However, the light barrier is not very reliable. To assure that the barrier is triggered the cashiers use a plastic separation stick that assures blocking the light barrier. When it blocked the light barrier, it is pushed back again.

A nice example how people overcome insufficient technology  with a simple but effective solution.

Bildschirmfoto vom 2014-03-01 14:40:01

Brainstorming: it sucks (says science)

Brainstorming is a rather well known Method for generating ideas. The participants generate as many as possibly ideas without judging the ideas.  Since Osborn introduced it in his work »Applied Imagination« it is in use and its rather popular [1] and part of the design thinking approach [2].

But already a few years after the technique has been introduced, empirical tests casted doubt on the method’s usefulnesses for generating ideas: Neither in terms of quality nor quantity it could could compete with ideas generated by single individuals [3].

This has been attributed by other researchers to several reasons (all from [4]):

  • In a group, just one person can talk, while others have to wait (supported by [5])
  • Fear of evaluation (despite of brainstorming’s credo »defer judgement«) (not supported by [5])
  • Group members don’t actually inspire each other; instead a group is prone to discuss more of the same.

Nevertheless, methods for creating alternative ideas are needed – at least research on prototyping strongly suggests that generating and testing multiple alternatives improves designs [6]. So what are viable alternatives for brainstorming? There are a lot of creativity techniques out there and I sadly can’t tell which is THE BEST. However, I made good experiences with the design studio method. (which is explained by Todd Warfel here). It is a process that uses sketching, individual idea generation and group feedback as its foundations; my experiences so far have been very well. As the process is less known than brainstorming and has admittedly a more complex structure, I made a little web app (At the moment only in German;  availiable on github too; you can compile it into a phonegap app) which hopefully guides a newbie through the process. Have fun being creative!


[1] Idea Generation Techniques among Creative Professionals by Scarlett R. Herring, Brett R. Jones, Brian P. Bailey, hicss, pp.1-10, 42nd Hawaii International Conference on System Sciences, 2009
[2] Use our methods,
[3] Taylor, D., Berry, P., Block, C.: Does Group Participation When Using Brainstorming Facilitate or Inhibit Creative Thinking? Adm. Sci. Q. 3, 23–47 (1958).

[4] Lamm, H., Trommsdorff, G.: Group versus individual performance on tasks requiring ideational proficiency (brainstorming): A review. Eur. J. Soc. Psychol. 3, 361–388 (1973).
[5] Diehl, M., Stroebe, W.: Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 497–509 (1987).
[6] Dow, S.P., Heddleston, K., Klemmer, S.R.: The Efficacy of Prototyping Under Time Constraints. Proceedings of the Seventh ACM Conference on Creativity and Cognition. pp. 165–174. ACM, New York, NY, USA (2009).

learning with videos

I love to use (and to contribute) to free educational resources. It was and is a big trend to use videos as a ›modern‹ way to get information across. There are quite a lot of these online. Since some time, you can download lectures from several (high-class) universities, later on we got the MOOCs like on coursera and on udacity.

As much as I still like these resources – after being overly motivated I came to use the video lectures very few. I e.g. tried to learn some more about data analysis. I quickly switched to using just the slides and the example data.

Why? Because some stuff I did already know. It was annoying to hear it again. So I needed to find where that-what-I-know ends. Most stuff I did not know, so I needed to find where what-I-don't-know starts. Not too difficult. But as soon as I came to a concept that was new interesting but hard to understand for me I needed to pause – otherwise I hear that lecturer going on about something else. The constant speed of a video does not match the naturally rather variable speed learning.

Thats not surprising, but a good reason for me to prefer (online) books with images, simulations or even short clips (like the classes from CMU or the online stat book). No need to pause, easy to skim.

However, I do not in general think videos are bad even if they are used as the main way to get knowledge across. I love Khan Academy. The videos are very short, so I immediately can choose what I need to know (So like: »I got that log-something here in the equation. What does that mean again?«). Thus I only get interesting content which develops step by step so I get what I need to. Because it is so focused I seldom had the feeling that it developed too fast

So in brief: For lecture style kind of information with multiple concepts of varying complexity I think a text with accompanying material suites – at least my style of learning – far better that a video. For very focused, bit-sized learning, short videos shine.

security and usability: message encryption

Recently I began to do a bit of research on security and usability in order to make it a little project for students to work on. I was well aware that the most secure system is not effective if people can't use it and that security (lets say: a very long password) and human preferences (a rather short one if at all) are not matching. However I was amazed how big the problems are.

First I was looking at message encryption. It seemed the most likely scenario: Write something that only you and a particular other person should see in plaintext. Well known for archiving this is PGP. During the recent revelations about the NSA’s  practices (time of writing: Oktober '13)  Crypto-Partys were flourishing all over the place and teaching people how to use PGP was seemingly the way out of the trouble. But as far as research and my personal experience is concerned it leads to another problem: the one of using PGP. In a classic study on PGP it took the participants quite a long time to get encryption working, several people broke the security by sending their private, secret key by accident. Security Guru Bruce Schneier says: »[My tips for online security] are not things the average person can use. […] Basically, the average user is screwed.«

I started to look at an alternative method for message encryption: Off-The-Record-Instant-Messaging. The protocol is designed with usability in mind (and there is a little paper on the topic) . You don’t need to manage keys by yourself, authentication works via a answering a question and the like. However, I still run into problems: To negotiate keys you both need to be online, so just sending an encrypted message into the blue does not work; To tell somebody that »encryption works but, let’s authenticate« results in a »WTF?« on chat (if you get your non-nerd-friends to play the game that far).

And despite of some background knowledge I myself am still confused about keys and their management– and that is just the part which has some interaction with the users: Which keys may be exchanged, which may never be exchanges and which are… whatever, it is complicated and it is no wonder that the mental model and the actual system diverge. (You get in contact with keys even on OTR if you can’t use the authentication-by-shared-answer [added])

What works probably fairly well is locking data via a password and unlocking it (Though I have no empirical study on the subject).  The user’s likely mental model of a locked box matches the one of the system fairly well. And that box is not some mysterious whatsoever but a file, something that most users will know. If they send it, the password needs to be shared via a second channel – but that’s it. But for sending many messages it is not very comfortable.

So encrypting messages is rather difficult, and even a relatively usability-aware protocol like OTR is noticeable less easy to use than plain text. To say that this is how it is and that one should RTFM is no solution. In general, because than, every unusable thing could be justified with that. And, more specific, because if only those who have something to hide encrypt, they are easily spotted and if one side can use encryption, but it seems to much for the other, both will stay without.

Update 05.11.13: Spelling and typo. Key management on OTR clarification. Link to the definition of a crypto-party,

from research to requirements: user need tables

There is a lot written about qualitative user studies but in my eyes there is few material out there which helps a practitioner to apply the methods. Even the more practical works depend on a large team and lots of time and resources. So the interested designer or researcher may decide to skip formal user studies. But I think there is a need for doing that in order to have a valid foundation for discussing possible features and how they must play together.

Because of the difficulties a practitioner may face when attempting to do user studies, I was pretty happy when I recently found the research of S. Kujala, a researcher whose research deals (among other fields) with methods for user studies. The research uses multiple case studies in which several methods and involvement of industrial partners is used to determine suitable methods of gathering and analysing needs and generating requirements for the product be be created.

In the suggested method, data is gathered as usual in
interviews and observations with focus on predefined critical topics (e.g. context of work, existing tools etc.)
In addition two other complementing methods are suggested:

  • Think-aloud usage of a (possible imaginary or prop-like)  product
  • The "interactive feature conceptualization": Items and Processes the user mentions in the interview are written on sticky notes; the user that is asked to arrange the notes in a system meaningful for him/her.

The interesting point for me is how the data is analysed. There is already a kind of pre-analysis existing: The categories from the main interview and observation. However, this structure was not very practical, so Kujala e.t al. developed an interesting tool, called "User Needs Table". (I hope my depiction is coherent with the authors suggestions; This is a description of my understanding of it)

It is aimed at bridging the gap between user studied and user requirements. The table has two columns, one for the tasks of a sequence, one for problems an possibilities associated with this task. In Kujala’s work they especially serve the construction of use cases which can be easily derived, since sequence and possible problems are linked together in the user need table.

I found the suggestions and the research design very interesting and useful. Neither does that author do an exclusively theoretical analysis of methods nor a quantitative comparison of  »treatment groups« using different methods (whether the latter would be any practical may be questioned). Instead the research is based on several case studies building upon each other. This, I find, is a very useful approach to develop empirically based and practical methods – and I have rarely seen such an approach. The most literature I read on qualitative methods is rather theoretical, so I think such a empirical and practical approach should be praised.

What remains unsolved for me is still a better and more practical transition from data to an analysis. As far as I understood, the user needs tables are created after the first report has been written. The predefinition of a focus is an important step, but at least I'd love to have a analysis methods beyond that.

Nevertheless, check out Kujala’s Papers, they are really interesting and illuminating and contain useful information on why the methods are designed the way they are:

Example user need table for a file upload to a learning management system (created for illustrative purposes, so it is not based on a 'real' project of mine)

Task Sequence Problems and Possibilities
Step 1: Selecting an “area” to which the upload belongs to (e.g. study group) Problem: The difference between the private file collection and the public files is frequently misunderstood.

Problem: For teachers who manage only one course, it is seemingly useless work

Step 2: Choosing a file Problem: The folder structures on the web portal and the local computer may be not the same; disorientation can occur.

Problem: Without instant feedback, Users may upload files twice

Step 3: … … (etc.)