Theme 6: Interaction Issues

This theme is concerned with the flow of events taking place in an interactive visualization session - concern is to preserve human cycles at the expense of machine cycles.

(6.1) Interaction flow

It is important to recognize that in designing a visualization one is both creating an interface to information and shaping or structuring the interaction which the user of the visualization has with the information or data being visualized. In other words, it is important to "design with the mind in mind (Hewett, 1997)."

(6.1.1) The characteristics of this interaction

will facilitate and limit what the user is able to do and the types of information which can be extracted from the visualization. The critical issue is that in many cases there will be a flow of events taking place overtime. The concern is to preserve human cycles at the expense of machine cycles as generally human cycles cost more than machine cycles.

(6.1.2) Relationship between interface and interaction

In thinking about visualization design, it is important to recognize that there are two different levels of design criteria that are important. One level is the interface or"tactical" level of design. The second level is the interaction or "strategic" level. At the interface level, design criteria address issues such as how interface components should be built and how they should be shaped. For example, aircraft cockpit instruments need to show clearly certain necessary information. Similarly, in a wide range of circumstances, knobs, switches and dials need to be easily discriminable by sight and/or by touch. Interaction level design criteria address the overall goals of the user and take into account the way in which the user thinks about the task. Concern with interface features is important since badly designed components can make the user’s task more difficult, lead to unnecessary errors, and even prevent successful task accomplishment. However, optimizing interface features without paying close attention to the overall flow of interaction can lead to unworkable or unusable systems.

(6.1.3) Example

(6.2) Interaction design

As illustrated above, even with locally well designed interface components a tool can fail to support the user’s interaction needs unless care is also taken to understand and support the user’s task goals. Focusing on the problem of interaction design, Norman (1988) has offered a series of ideas about how to design artifacts that are both usable and useful.

(6.2.1) Seven stages of action

The model Norman has developed involves what he calls the "seven stages of action." To complete an action successfully, the user must first formulate one or more goals. Once a goal has been established there are then six additional steps which must be taken for the user to determine whether or not the goal has been accomplished. The user must: 1) formulate an intention to act; 2) specify the action(s) to make; 3) execute the actions successfully; 4) perceive the state of the world;5) interpret the state of the world, and 6) evaluate the outcome.

(6.2.2) The gulfs of execution and evaluation

The first three of these steps (1, 2, &3) Norman calls, "The Gulf of Execution," and the second set of three (4, 5, &6) he calls, "The Gulf of Evaluation." If any step doesn't work properly either the user won't make progress towards the goal, or else the user won't be able to determine the outcome of actions taken. The visualization designer should facilitate the user's tasks by helping to make it clear how to bridge the Gulf of Execution and by providing appropriate feedback to make it possible to bridge the Gulf of Evaluation. As noted above, one prerequisite to designing a visualization which facilitates tasks is understanding what those tasks are and how the user thinks about them.

(6.2.3) Some design principles

At a general level, some guidance in how a visualization designer might help the user bridge the Gulfs of Execution and Evaluation can be found in a set of human-computer interaction design principles proposed by Norman.

(6.2.3.1) Principle 1

The first design principle is that the designer should enable the user to make use of both knowledge in the head and knowledge in the world. Knowledge in the head is that knowledge which is stored in memory. However, we often can and do store only partial knowledge in memory and make use of the presence of knowledge in the world to assist in guiding our actions. For example, most people have little trouble dialing a phone number such as555-PUCK to get hockey tickets. However, with no telephone dial or set of pushbuttons visible, many of those same people have to reconstruct rather than remember the letters associated with each number. (They also have difficulty identifying the two letters of the alphabet which do not appear.) Similarly, the typical graphical user interface provides memory cues which reduce the user’s memory burden since it stores many commands on the screen where they can be easily located or found with minimal search time.

(6.2.3.2) Principle 2

A second design principle identified by Norman is that the designer should seek to simplify the structure of the tasks which the user needs to perform. This simplification can involve several different options. One is to keep the task the same and provide one or more mental aids (e.g., the calendar for people who have to do scheduling; e.g., a symbolic computing engine which can send numeric output to other tools). Another option is to improve feedback and the user’s ability to maintain control by using technology to make things visible that would otherwise not be visible (e.g., instruments in the automobile; e.g., processing flow in a parallel program). A third option is to automate portions of the task without radically altering the actual flow of the task (e.g., provide turn signals for automobiles rather than require arm and hand signals; e.g., provide a consistency checker which determines if there are missing parentheses in a complex equation; e.g., develop a notation translator which facilitates conversion from one notation to another). The fourth option described by Norman for simplifying the structure of the user’s tasks involves actually changing the nature of the task to make it more easily performed (e.g., learning to tell time is much easier with digital watches than with analog; e.g., some navigational instruments require positioning of a slide and reading of results rather than calculation of numbers).

(6.2.3.3) Some additional principles

Norman’s third design principle is to make things visible so that the user can more easily bridge the Gulfs of Execution and Evaluation (e.g., the Captain of an ocean going ferry boat should have an indicator light on the bridge which clearly signals that the front deck doors which prevent large waves from washing over the automobile deck have been closed). A fourth design principle is to get the mappings right. That is, the results of users’ actions should match their expectations. For example, the motion of a throttle should require a forward push, as should the motion of other controls associated with an increase in speed. Controls to rotate 3D visualizations should move or point in the direction of intended motion. One of the more important of Norman’s principles is that the designer should design for error. The designer should assume human error will occur and allow for error by making it possible for the user to be able to identify when an error has occurred and to be able to cope with or even avoid the undesirable consequences of the error.

(6.3) Human performance characteristics and limitations

One of the earliest books on human-computer interaction to address human performance characteristics and limitations was written by Card, Moran, & Newell (1983). In this book the authors introduce the notion of, "the model human processor," as a way establishing a conceptual framework within which to discuss the operating characteristics and performance limitations which might be assumed to be true of any human interacting with a computing system. Their discussion ranges from the level of individual key strokes and the components of reaction times involved in those keystrokes all the way up to the level of talking about the Goals, Operators, Methods, and Selection rules(GOMS) that might be involved in a contextually appropriate model of human problem solving. In the intervening years, there have been a number of other books published which discuss the issue of how to factor human capabilities and limitations into the problems of interface and interaction design. Some which provide broad coverage and which should be readily available include Baecker & Buxton (1987); Baecker, Grudin, Buxton &Greenberg (1995); Booth (1989); and Helander (1989). Others provide an introduction to some of the more controversial issues in the field of human-computer interaction (e.g., Carroll, 1987, 1991). The purpose of this discussion is to provide a summary overview of some selected topics and their relevance to visualization design.

(6.3.1) Demands on working memory

The contents of working memory are often thought of as having been selected for further processing by an attentional mechanism which is guided either by changes in sensory stimulation, by expectations resulting from immediately prior processing, by expectations resulting from information already stored memory, or by some combination of the three.

(6.3.1.1) Working memory

has a limited capacity and duration. Usually, working memory is described as having a limited storage capacity (seven plus or minus two chunks)for a relatively brief duration (estimates range from 12 to 30 seconds without rehearsal) before information is lost through simple decay. (i.e., After about 12seconds these chunks have disappeared and are no longer available for use in working memory.). Another way in which information gets lost from working memory is when new information displaces older information. (Think of watching a moving train from your office window, you can see only a limited number of boxcars moving along the track at any given time, those that have already passed out of view have been displaced by cars currently visible through the window. You know that the earlier boxcars were there, but....) Although new chunks can be added to working memory, either by introduction from the perceptual world or by activation of a node in long term memory, the capacity limitation will result in some chunks being lost if the number of chunks being juggled exceeds the capacity of working memory. Finally, it should be noted that a failure to recall something from working memory can also occur as the result of interference, i.e., the retrieval of an item which is not the correct one but which has similar characteristics. Even though interference between items does not strictly represent a loss of information from memory, if you query memory and retrieve an item similar to but not identical with one which was put in, there is are call failure.

(6.3.1.2) Keeping memories

alive in working memory. Decay in working memory can be prevented by rehearsal of the chunk, i.e., by simple repetition. Information can be maintained in working memory for periods of time longer than 12-20 seconds with maintenance rehearsal (MR). However, this simple repetition of material does not appear to be very efficient at transferring information into long-term memory (LTM). Rather, elaborative rehearsal (ER), which requires working with the information in some way, appears to be the most effective set of processes for the transfer of information into long-term storage. Considered as a unit of storage the chunk is somewhat problematic in that it turns out that sometimes chunks contain only one of something and sometimes chunks contain much more. For example, a chunk may be only a single one digit number from a randomly presented string. Or alternatively a chunk might well contain several digits, as in the case of storing 555-PUCK into working memory so that one can easily recall the number to call for hockey tickets. There are reasons to believe that any principle or basis for the organization or structuring of long term memory can serve as the basis for chunking. This also means the capacity of a chunk is variable, depending upon the complexity of the knowledge structure upon which it is based.

(6.3.1.3) Some implications

While it is not possible to provide an exhaustive description of the implications of the various strengths and limitations of working memory, it is probably useful to describe a few examples which go beyond the effects of obvious capacity limitation created by the number of chunks which can be maintained in working memory.

(6.3.1.3.1) Given the apparent failure

of maintenance reversal at transferring information into long-term memory, would you expect the user of an interactive visualization system to be able to recall command strings where practice involves simple repetition of commands? What type of experience with commands is needed? What implications are there for the design of tutorial manuals and for the structure and choice of exercises?

(6.3.1.3.2) When looking at

and working with a visualization, domain experts and novices will probably have very different bases for chunking information and therefore will work with a particular representation in very different ways. For example, we would expect novices to focus on and extract relatively superficial or trivial features from the visualization since their chunks. What type of implications does this have for error messages?

(6.3.2) Strengths and limitations of long term memory

In general long term memory refers to those memories which may persist of many years but are not active in working memory. LTM, which in many ways is an enigma, has an indefinitely large capacity for storage of information for long periods of time. For example, it is useful to hink of the organization of long term memory as being something like a complex semantic network of concepts and relationships among concepts. Activation of one node in the network can result in an increased likelihood of activation of nearby nodes.

(6.3.2.1) Capacity of long term memory

There is, however, no easy or obvious way to determine the limits of how much information can be stored, or for how long it can be stored. Determination of capacity requires identification of a unit of storage. The range of things stored in long term memory is quite broad and includes such things as representations of events that have been experienced and a variety of knowledge structures that have been acquired by reading or thinking. Among the types of information represented in LTM are such things as facts and events, motor and perceptual skills, knowledge of physical laws and systems of mathematics, a spatial model of the world around us, attitudes and beliefs about ourselves and others, etc. This information is more or less well organized, in a variety of ways, and varies in its accessibility as a function of several factors.

(6.3.2.2) Access to the contents of long term memory

The factors determining accessibility of the information in LTM include such things as the conditions which existed at the time the information was stored, the regency of its last use, its degree of inter-relationship with other knowledge, its degree of uniqueness relative to other information, etc. While most authors stop short of claiming that information once stored in LTM is not forgotten, most discussions of failure to recall information from LTM focus on explanations such as interference, the absence or inappropriateness of retrieval cues, or some type of organic dysfunction such as brain damage. While there are many researchers who argue strongly for the existence of a process of decay of information in long term memory there is still controversy of the degree to which failure to recall information from LTM should be attributed to decay rather than interference or lack of appropriate retrieval cues.

(6.3.2.3) Some implications

It is clear that humans interacting with computers and with visual representations of information can and do use existing memories and memory structures to assign a meaning or interpretation to a wide variety of things, regardless of whether that meaning was the one intended by the designers. Consequently, it is important to understand and take account of existing knowledge structures and how people think about domain problems and the problems of interaction. As with working memory it is not possible to provide an exhaustive description of the implications of the various strengths and limitations of long term memory. However, it is again useful to describe a few examples which go beyond the obvious fact if people do not have the appropriate knowledge structures they will not be able to correctly interpret what they see, even if told what they are seeing.

(6.3.2.3.1) If getting the appropriate meaning for a word

or if making sense out of a procedural description or visualization depends upon having the appropriate knowledge structure in a person’s head, how helpful will it be to provide someone with set of instructions about what to look for in a visualization, or with a detailed set of step by step instructions for accomplishing a particular interaction goal, when they do not have a clear and appropriate mental model and of what they looking at or what they are doing and why? Can such a mental model be established simply by following instructions?

(6.3.2.3.2) Imagine you are part

of a product development team. Your company thrives on the sales of a visualization software toolkit. It is your team’s job to migrate the software from Version 3.x, which has been quite successful in a non-GUI environment, into Version 4.0 which is expected to run under a GUI. Metaphorically speaking you have tied around your neck that large, cumbersome sea anchor known as, "an installed user base." This user base includes both end-users of 3.x visualizations and the large number of developers employed by your customer companies and/or research laboratories. The role of these developers is to use your tool kit to create in-house visualization environments for use by the end-users (their user group). Given that a GUI involves a new model of interaction which is similar in some respects and which differs markedly in others from what the developers and end-users are accustomed to, part of your task is to build Version 4.0 in such a way that your installed user base can migrate with the greatest possible ease and least possible pain. Can you think of any reasons why it might be important to convince your management to allow you to do frequent early iterative evaluation of Version 4.0 with developers who use your existing toolkit for building visualizations? Would knowing something about the existing knowledge structures of developers who use Version 3.x (e.g., how they think about the current functionality, etc.) be helpful in finding a way to organize the features, structure and functionality of Version 4.0 so as to ease their migration to the new system?

(6.3.3) Human problem solving

A major turning point in the literature on problem solving occurred with the work of Newell and Simon (1972). Newell and Simon proposed that a problem be analyzed in terms of a"problem space" representing various states of knowledge of the problem solver, a series of transformations between states, and a set of operators which produce those transformations. In other words, a problem exists when we have a gap between an initial state and a goal state. The means of solving the problem involves selecting and applying the appropriate set of operators required to complete a series of state transformations that will eliminate the gap. These transformations must be accomplished without violating any of the conditions on the operators. It is this conception of human problem solving which motivated the GOMS model proposed by Card, Moran & Newell (1988)

(6.3.3.1) Problem solving schemas

As a result of repeated experience with a series of identical or very similar problems, problem solvers build up an organized body of knowledge or information about the properties of a particular type of problem. Such an organized body of knowledge is usually referred to as a schema (e.g., Norman, 1982). There are a wide variety of familiar problem schemas (Hayes, 1981), and the typical individual has built up a stock of problem solving schemas which come into play in solving problems. Some of these schemas are so familiar they are activated almost automatically and without thought. Typically, the main effect of problem solving schemas is to provide us with reasonably efficient methods of solving frequently encountered kinds of problems. However, sometimes a schema can interfere with problem solving. Some of these side effects of schemas can be illustrated by considering two related concepts described in the older literature on problem solving. These concepts are problem solving set and functional fixedness.

(6.3.3.1.1) Problem solving set

The work of Luchins (1942) provides a reasonably clear example of the concept of a problem solving set, i.e., the effect of prior experience in limiting or constraining solution procedures. Luchins asked people to solve a series of problems, many of which required the exactly the same procedures. He found that several of his participants continued to use the solution procedure developed on the early problems even when they came to problems later in the set which would solve by a simpler or more efficient procedure.

(6.3.3.1.2) Functional fixedness

Functional fixedness is a second concept from the older literature on problem solving which can be seen as being related to the concept of a schema. Coined by Duncker (1945), the term "functional fixedness" is used to refer to the inhibition created when an object with a familiar function needs to be used in a new and different way to solve a problem.

(6.3.3.1.2.1) A standard example of functional fixedness

is Maier's (1931) two-string problem. For this problem, you are in a room with two strings tied to the ceiling. Both strings are of equal length. Your objective is to tie the ends of the two strings together. The problem is that while the strings are long enough to be tied together they are short enough that you are unable to just take hold of one string, walk over to the other string, and tie them together. Scattered around the room there are a number of objects. These objects include a plate, some books, a chair, a pair of pliers, and a book of matches.

(6.3.3.1.2.2) About 60% of the participants in Maier's study

failed to find a solution within a 10 minute time limit. These individuals saw the pliers only as a tool, not recognizing that the pliers could be used as a pendulum bob, swinging at the end of one of the two strings. (Significantly more participants were able to find a solution to the two string problem in an alternative situation. In this alternative situation there was an open tool box in the room. In the room were several hand tools, including a pair of pliers. In addition, visible in the tool box was a plumber’s bob.)

(6.3.3.1.3) Representation and re-representation

Since it influences the problem solver's early analysis of the problem and determines which problem solving schema, or schemas, will be brought to bear, the way in which a problem is represented can make a vital difference in how easily it can be solved. Sometimes what is needed to facilitate problem solution is an alternative representation (a problem isomorph or a problem analog).

(6.3.3.1.3.1) Representations clearly make a difference

in the ease with which a problem can be solved. As Simon(1981) points out, "Solving a problem means representing it so as to make the solution transparent." For example, sometimes people debate which programming language is the best. Given any five programmers there it is not unusual for there to be at least six answers. Does the multiplicity of answers suggest that the choice of a "best" language depends on what one wants to accomplish? Occasionally someone will claim that anything which can be done in one language can be done in any other. But, would anyone choose to do intensive scientific computation in LISP or Prolog? Would anyone choose to do Artificial Intelligence programming in FORTRAN?

(6.3.3.1.3.2) As Norman (1993) points out,

the "powers of cognition come from abstraction and representation." Once one has a representation one can use that representation to think about a situation. The difficulty is that one must get the essential structure of the situation into the representation, otherwise the representation becomes misleading or a hindrance to solving the problem. When a problem that looks solvable is not solved as easily as expected it is often quite fruitful to look to the adequacy or the nature of the representation of the problem, and to give consideration to changing that representation to a new one.

(6.3.3.2) The importance of problem solving in visualization

In considering the nature of human problem solving it is clear that visualization designers need to take account of the fact that users will bring to bear established problem solving strategies which are developed from having solved problems that have been frequently encountered in the past. While these established strategies can be used to facilitate usage they can also create process blockages which may only be solved by changing the way in which the problems of interaction are presented to the user or by creating a working environment in which the user has the flexibility to redefine, reconceptualize or re-represent the tasks, the problems, and/or the visualizations which the software is being used to solve. In particular, the software should make it possible, without penalty, for the user to suspend current activities and be able:

• a) to see more information about the software or the problem domain;
• b) to re-examine or re-formulate some aspect of the visualization, the interaction problem or the domain problem, with an eye towards modifying that problem in some way;
• c) to completely re-represent or re-formulate the entire interaction or domain problem; and
• d) to break up the interaction or domain problem into manageable sub-problems in ways that allow for productive recombination of elements or components.

(6.2.4) Other factors impacting the interaction flow

Clearly there will be, depending upon the circumstances, a number of other variables which may impact the flow of the interaction between the user and a visualization and therefore become relevant in thinking about the problems of interface and interaction design. For example, given that both memory structures and changes in external levels of stimulation can impact attentional mechanisms and the focus of a user's attention there are both internal and external factors with which to contend.

(6.2.4.1) What will be the effect

of repeated shifts of attention (on a user’s memory burden, say) of taking items which are conceptually related in the user’s mind or which are integral sub-tasks of a single task and putting them in widely separated screen locations, in screen locations which differ from one screen to the next ,or on different screens entirely?

(6.2.4.2) Suppose that a user is working

in an environment where there is a continuous flow of information and a number of distracting events. Does this suggest anything about the kind of memory aids one might want to design for users who will be working with visualizations in a context where they will continually have to be shifting their attention from one information stream to another?

(6.2.4.3) Suppose that in my working memory

I have two pieces of information which I will need to use on the next task in a series of tasks on which I am working. Now I must shift the focus my attention to working out a series of voice commands to prepare the computer software for the next task. What might happen to the contents of working memory?

(6.4) Implementation

(6.4.1) Interaction Techniques

the three big issues here are:

• (6.4.1.1) Direct manipulation (interacting with visual depictions of data directly).

• (6.4.1.2) Integration of visualization and computation in a tight interaction loop for computational steering.

• (6.4.1.3) Shared, collaborative visualizations and interactions, where users at different screens see the same visualizations and share interactive control of them.

• (6.4.1.4) Individual techniques supporting these issues are

  • (6.4.1.4.1) selection

  • (6.4.1.4.2) moving

  • (6.4.1.4.3) editing

  • (6.4.1.4.4) probing

  • (6.4.1.4.5) other tasks (print file, delete file, ...)

(6.4.2) The Interface

• (6.4.2.1) Can facilitate interaction

• (6.4.2.2) Can inhibit interaction

(6.4.3) Interaction Components

• (6.4.3.1) Visual and non-visual ICs

• (6.4.3.2) Well designed ICs

• (6.4.3.3) Ill designed ICs

(6.4.4) Consistency

Consistency in interface and interaction design is often said to be a good thing. But, consistency is a relational concept and has little or no meaning unless we answer the question, "Consistent with what?" Suppose we are developing software with a new range of functionality. How useful will it be to put a higher premium on consistency with a style guide than on consistency with the mental model we want the user to develop or use in thinking about the task, the visualization, the software and the interaction?

(6.4.4.1) Standards

(6.5) Evaluation

The development of visualizations and of interactive computing systems in general should be an iterative process of design and redesign which involves users as participants in the process of development and refinement. For example, Hewett(1986, 1995) argued for the importance of user-driven, iterative evaluation-evaluation as part of each design cycle-as a major factor underlying the process of successful interactive computing system design. One type of evaluation formative involves monitoring the process and products of development and gathering user feedback for use in refinement and further development of the materials. A second type of evaluation summative involves assessing impact, usability and effectiveness of the materials. Different stages and components in the development of visualizations an interactive computing materials require different types of evaluation.

(6.5.1) The role of evaluation

The first, and most important, thing to recognize about evaluation of visualizations and of interactive computing systems is that designers do it. The second thing to recognize is that the users do it. The only uncertainties are when, and how systematically, the evaluation will be done, and whether the evaluation results will be used to guide redesign of the interface, the interaction, and visualization materials. Consequently, it is essential that the evaluation process be structured to create and maintain a clear focus on the goals of the project and to provide useful, meaningful information to guide the improvement of the visualization and the flow of interaction. Similarly, evaluation procedures themselves sometimes need to be designed, evaluated, and redesigned.

(6.5.2) Formative and summative evaluation

In developing a systematic evaluation plan for any visualization or interactive computing environment it is essential to recognize the basic difference between two types of evaluation formative and summative. This distinction was first discussed by Scriven (1967) in the context of the evaluation of educational and social action programs. These two types of evaluation have very different goals and it is important to keep them separate.

(6.5.3) Formative evaluation

As originally described by Scriven, formative evaluations the assessment done to evaluate progress towards completion of the project, prior to actually implementing it. That is, formative evaluation is an assessment of the state of development of the relative to intended final form. This type of evaluations initially focused on determination of what the various components of the social action program should be. During development it is focused on whether or not the pieces of the project are being brought together as they should be. Is the project moving towards the stated conditions which must be in place before the program or intervention is actually carried out? Are there unanticipated factors which necessitate a re-design of the plans? Was the original goal misconceived and in need of be in re-thought? As applied to development of visualization materials and interactive computing systems, formative evaluation involves monitoring the process and products of development and gathering user feedback for use in refinement and further development of the visualization, the interface, and the interaction flow. In other words, what are the goals of the project and is the project moving towards those goals?

(6.5.4) Summative evaluation

Scriven described summative evaluation as that type of evaluation which is conducted when what is wanted is an assessment of the impact of the social action program once it has been put in place. Does intervention have its desired effect upon the people it is intended to benefit? Is the social action program more successful than the alternatives already in place? As applied to development of visualization materials and interactive computing systems, summative evaluation involves assessing the impact, usability and effectiveness of the materials-the overall task performance of user given the visualization. In other words, does the visualization make the user's task easier, harder, or is there no difference when compared with the alternative ways of interaction with the data or information and its representation.

(6.5.5) Formulating evaluation criteria

In thinking about conducting evaluations, perhaps the most critical component of a successful evaluation is the selection or formulation of appropriate criteria upon which to base an evaluative judgment. What follows is a sample list of some of the types of evaluation criteria and judgments which might be appropriate for data and information visualizations.

• (6.5.5.1) Is the visualization more efficient than the alternatives?

• (6.5.5.2) Does the visualization allow the user to do or see something new and different?

• (6.5.5.3) Is the visualization easily transportable, i.e., can it be used on more than one machine or in more than one environment?

• (6.5.5.4) Does the software which controls the visualization allow for "what if" manipulations of the representation?

• (6.5.5.5) Does the software make adequate use of unique capabilities of the technology (e.g., Hypertext)?

• (6.5.5.6) How much documentation is required to use the software?

• (6.5.5.7) Can users who have browsed easily find out how to revisit points of interest?

• (6.5.5.8) Does the software allow a user to interact directly with information?

• (6.5.5.9) Does the software represent a special purpose tool or a generic tool with capabilities for extension?

• (6.5.5.10) Does the software seem cobbled together or does it exhibit a clean, coherent concept.

• (6.5.5.11) Does the software offer important functionality not found elsewhere?

• (6.5.5.12) Does the visualization enhance the user’s ability to learn from and about the material being visualized?

• (6.5.5.13) Does the infrastructure operate against natural convenient use of the visualization or the software?

(6.5.6) Some general guidelines

Typically the development of a visualization or interactive computing system will require more than one design-evaluate cycle. There are some useful guidelines to keep in mind during the iterative process of re-design and revaluation. For examples from the design of interactive computing systems, see Hewett (1986, 1995).

(6.5.6.1) Guideline One: "Know the User."

Probably the single most basic admonition inhuman factors and ergonomics is, "Know the user." However, this admonition might better be formulated, "You have a model of the user, what is it?" Designers of visualizations and interactive software should seek early involvement with users as part of the design team (cf. Gould & Lewis, 1985; Hewett & Meadow, 1986) so as to avoid being trapped into erroneous assumptions about their visualization, the computer interface, and the interaction flow.

(6.5.6.2) Guideline Two: "Seek Out a Variety of Users."

Once a prototype of the visualization and interaction software is available, and on a continuing basis there after, the designers should seek out as wide a variety of users as possible. For almost any group of users who get actively involved with evaluation of the materials there are strengths and weaknesses associated with the type of information they can provide. For example, volunteers will often go out of their way to be helpful to the designers, even to the extent of compensating for problems in the computer interface or not mentioning problems encountered with the visualization. Consequently, designers should recognize that it may be necessary in actual practice to repeat this type of formative evaluation more than once to insure a sufficient variety of users.

(6.5.6.3) Guideline Three: "Pilot Test Summative Evaluation."

There are important benefits to pilot testing summative evaluation procedures whenever possible. Designing an evaluation is much the same as designing anything else. It is possible to overlook important factors until one is actually engaged in putting the evaluation procedures into practice. Also, a summative evaluation pilot test can have a very powerful formative evaluation effect by forcing clarification and reassessment of overall project goals and how best to determine whether or not they have been achieved. In day-to-day concern with the details of visualization and interaction development it is possible to lose sight of the overall project goals, allowing the direction of the project to undergo subtle shifts. Facing up to the question of, "Compared to what?" can help one to realize that the visualization may not be as helpful as it ought to be and that project goals need to be reformulated.

(6.5.6.4) Guideline Four: "Prepare for Contingencies and Trade-offs."

Justas design, development, and delivery of a visualization and interactive software materials involves making contingency plans and trade-offs, so too a summative evaluation of the effectiveness of those materials requires making contingency plans and trade-offs. Such planning can help minimize the impact of possible disruptions of evaluation studies. Similarly, it can help maximize the gain from unexpected opportunities. The kinds of trade-offs required in summative evaluation include: the type of users to test in the evaluation studies, the size of the samples to be used, the kind and amount of data to collect, when and how often to test during the development of the participants' abilities, whether to use single or multiple tasks, and whether to use a "real" or a standardized task in the evaluation.

(6.5.7) Some procedures for conducting formative evaluations

While any good book on usability testing (e.g., Hartson & Hix, 1993, Lindgaard, 1994) will describe a variety of techniques for gathering the information needed to improve the design of a visualization, most of these techniques can be summarized by noting that they have a number of basic things in common. One commonality is that the procedures for conducting usability tests typically are open-ended enough to allow for the unexpected, i.e., to allow the users to behave in relatively natural ways which provide useful information about their interaction with the visualization and software. A second, and more important commonality is that the procedures are designed to reduce or compensate certain kinds of biases which can eliminate the benefits of conducting an evaluation. A reasonably accurate way of thinking the bias problem is to consider a developer's quite natural desire to see a visualization appreciated and to want to coach the user through difficulties which might be eliminated by redesign. Such coaching is only really helpful if the designer is willing to provide such assistance any user of the visualization or the software.(Anyone who thinks that carefully conducted formative evaluation work is just too expensive should take a look at work reported in Bias & Mayhew (1993) before investing too much time and money in a project where no resources are being allocated for usability testing.)

(6.5.7.1) An example

The following example, based upon work reported in Hewett & DePaul(under review), illustrates the use of three different formative evaluation techniques to develop an understanding of the visualization needs of a research mathematician and of the inadequacies of the visualization software to which he currently has access. The techniques used were interviewing, simple observation of behavior, and a variation of the thinking out loud and protocol analysis procedure described by Ericsson & Simon (1996).

(6.5.7.2) The nature of the problem

The mathematician wants to produce geometric representations of a particular class of equations known as soliton equations. About twenty five years ago, certain equations which are highly complicated were found to be exactly solvable. These types of equations, complex equations with simple behavior, are referred to as soliton equations. There are a number of such equations known. Two examples of equations which behave in this manner are the equations for fiber optics and the equations for smoke rings (or tornadoes).The mathematician being studied has found that these different equations share subtle correspondences. Furthermore, he believes that the soliton equations are so similar that there must be a generalized geometric analog, a curve moving in some space (within or beyond the third dimension), which is common to all soliton equations. Currently, to solve the representation problem, the mathematician takes the information provided in the solutions of the soliton equations, the bend and the amount it leaves a plane, and converts that information to create a geometric representation.

(6.5.7.3) Information gathered from interviews

Through a series of interviews, conducted as preliminary information gathering sessions before directly observing problem solving activities, some interesting things were learned about the problem solving behavior of the mathematician. Although the information provided here was self-reported by the mathematician and was reported post-hoc, there is reason to believe that the issues discussed in the interviews are representative of the actual behaviors. For example, the problem solving behaviors described by this mathematician parallel those found in experts in other fields (e.g., Candy & Edmonds, 1995; Clement, 1988; Dunbar,1995).

(6.5.7.4) Self-reported visualization needs

In the past year, the mathematician has put together a paper in which he argued that higher dimensions play a role in forming the representations of some soliton equations. With this knowledge he then found a possible representation for the particular soliton equation he was working on solving, are presentation which exists in some n-dimensional space. Since this type of solution is very difficult to work with and since most people have trouble comprehending higher dimensions, the mathematician’s next step is to find either another soliton equation which could occur in 3Dspace, or another space in which a simplified version of the current representation could occur. To begin this next step of the problem, the mathematician is looking at the possibility of representing these equations in the three sphere, hyperbolic space, or one of several known space models where light does not travel in a straight line, but nevertheless, travels in a controlled or understood manner. His reason for turning to these representation spaces is that his knowledge of the behavior of the structure of these spaces has led him to believe the curves or surfaces of the soliton equations could potentially be characterized by the same type of behavior.

(6.5.7.5) A need for three dimensional interactive visualizations

In terms of his 3Dvisualization needs, the mathematician reports that he needs to be able to discretize the equations in such a way that he is able to directly perceive some of their structure. The starting object in the visualization is a curve in the plane. He reports that he utilizes rotation regularly to help gain a sense of structure and often finds that a switch in visualization modes (from solid to grid or from grid to solid) is helpful in "seeing" what happens. He reports being frustrated by the fact that the intersections of the surfaces are important and are often particularly hard to see. He also reports that it is important to him both to see the full result surface and to see how the curve evolves frame by frame. Consequently he expresses a desire to have tools which he can use with a 3D visualization which will allow him to create an intersection focus, an edge enhancement, and a local isolation of pieces of the picture for manipulation.

(6.5.7.6) Sources of frustration

Since the internal structure of the 3Dvisual representations with which the mathematician works is often concealed by a surface, even with a grid visualization, he thinks he would also like to have some special purpose tools which would enable him to enhance the visibility of regions of the internal structure. Analogies which seem to represent these needs, and which the mathematician accepts, include the notions of "pearl growing," of "onion peeling," and of "surface crawling" or "paint spreading." Surface crawling or paint spreading would involve being able to endow a region of an internal portion of the surface with a different color or texture and then being able to move or spread this visible difference around or across regions of the internal structure, thereby "highlighting" and making those regions more discriminable visually. The instantiation of pearl growing might involve taking a small portion of the internal structure as the starting point and then being able to watch as the surface "grows" into its full complexity. Similarly, onion peeling in this context might involve taking the full 3D representation and selectively peeling off and discarding external portions of the surface, thereby bringing internal surface into view. Interestingly, the analogies of pearl growing and onion peeling are conceptually the same as those investigated by Hewett & Scott(1987) in the context of bibliographic database searching. These two analogies were shown to be appropriate ways to conceptualize two frequently conducted types of bibliographic database search. In the one case a searcher starts with a singe article and"grows" it into a larger set of related articles. In the other case the searcher starts with a large set of references and selectively "peels" away subsets to achieve a desired result.

(6.5.7.7) Observations of behavior and thinking out loud

During sessions when the mathematician's problem solving behavior was observed, a number of other interesting factors became clear about his current working environment and the types of features which would improve his ability to do his work an appropriately re-designed visualization environment. Using a slightly modified version of Ericsson and Simon's (1996) instructions, the mathematician was asked to "think aloud" while working on a professionally relevant problem. The problem was typical of those in which the his research interests lie. He and a colleague were about to publish a paper when they recognized that there were errors in the mathematics. The observations of his problem solving behavior were conducted while he was working to find the errors.

(6.5.7.8) Current visualization tools

The current "problem solving environment" employed by the mathematician includes a common symbolic computing program with visualization capability, a textbook, personal notes, and a copy of the paper which contained the errors. Utilizing the textbook to find the formulas he needs, the mathematician then writes them out in standard form before modifying them with features specific to the current problem. This may well be related to the fact that he uses one notation in his research papers, the textbook uses a different notation system, and the symbolic computing engine requires him to use still another notation system. When he is using formulas with variable notation the mathematician often gets confused and plugs in numbers for the variables to be able to see a solution in terms of numbers rather than variables.

(6.5.7.9) Memory demands which could be avoided

The mathematician also ran into difficulty in recalling the names he had assigned to specific formulas in the symbolic computing software. The software allows great flexibility in naming equations and he seemed to assign the same labels over and over again, or to randomly assign a new label. He would follow a consistent trend in his equation naming convention until her an into a formula which was similar to earlier ones. In this situation he would produce a "random" name and then later have difficulty recalling the name. This difficulty with names is not surprising, given that typically the mathematician tended to proceed through his search for errors in the mathematics in the paper in a careful stepwise manner, systematically considering one possibility for a solution to a piece of the problem and then moving on the next step. This careful checking process continued as he both confirmed and disconfirmed his assumptions regarding the behavior of the equations.

(6.5.7.10) Criteria for a new visualization support environment

From consideration of the mathematician's current working environment and observation of his behavior, it is possible to identify some characteristics which should be built into a visualization tool designed for such a person. He clearly needs simultaneous access to multiple sources of information. He periodically needs to concretize abstract aspects of the problem as a way of checking on his progress. The fact that he had difficulty in recalling labels for formulas suggests that a shorthand naming directory would be helpful. In addition, the symbolic computing software, rather than simply replacing a previous formula with the same name as a new formula should keep a record of all formulas assigned that label. This would allow retrieval of a potentially lost formula. The existence and common usage of multiple notation systems suggests that a visualization support environment should include a notation translation feature. Another helpful feature would be a syntax checker for the programs developed in the symbolic computing software. It was not unusual for him to drop a closing parenthesis, make a typographical error, or introduce a syntax error which interfered with the program execution.

(6.5.7.11) A final note

In the example described above it is instructive to consider how difficult it is to determine the utility of the visualizations actually produced by the symbolic computing software. Much of the mathematician's interactions with the software involve preparation and work required to cope with problems generated by the software interface and interaction rather than the mathematics and a visual representation.

(6.5.8) Some procedures for conducting a summative evaluation

The practice of conducting summative evaluations basically requires some knowledge of and experience with behavioral science experimental methods. While any good book on research design (e.g., Kerlinger, 1973; Runkel & McGrath, 1972) or quasi-experimental design (e.g., Cook & Campbell, 1979) can provide detailed descriptions of such methodologies, most of these techniques can be summarized by noting that they have a number of basic things in common. One commonality is that the procedures for conducting a summative evaluation typically involve the clear specification of one or more manipulated independent variables, and careful measurement of the dependent variables they are thought to affect. Another important commonality in the various procedures for setting up a summative evaluation is the exercise of the procedures necessary to ensure an appropriate control group through the use of randomization, or the exercise of the procedures necessary to ensure an appropriate comparison group when randomization is not feasible. The basic guideline for conducting an appropriately designed summative evaluation is to utilize the skills of someone who has expertise in experimental design and the statistical analysis of experimental results.

(6.6) References

• (6.6.1) Baecker, R. M. & Buxton, W.(Eds.) (1987). Readings in Human-Computer Interaction: A Multidisciplinary Approach. Los Altos, CA: Morgan Kaufmann.

• (6.6.2) Baecker, R. M., Grudin, J., Buxton, W. A. S., & Greenberg, S. (Eds.)(1995). Human-Computer Interaction: Toward the Year 2000. Los Altos, CA: Morgan Kaufmann.

• (6.6.3) Bias, R. & Mayhew, D. (Eds.) (1994). Cost-justifying usability. Boston: Academic Press.

• (6.6.4) Booth, P. (1989). An Introduction to Human-Computer Interaction. Hillsdale, NJ: Erlbaum.

• (6.6.5) Candy, L. & Edmonds, E. A. (1995). Creativity in knowledge work: A process model and requirements for support. Proceedings OZCHI '95, pp.242-248.

• (6.6.6) Card, S. K., Moran, T. P., & Newell, A. (1983). The psychology of human-computer interaction. Hillsdale, NJ: Erlbaum.

• (6.6.7) Carroll, J. M. (Ed.) (1987). Interfacing Thought: Cognitive Aspects of Human-Computer Interaction. Cambridge, MA: MIT Press.

• (6.6.8) Carroll, J. M. (Ed.) (1991). Designing Interaction: Psychology at the Human-Computer Interface. Cambridge: Cambridge University Press.

• (6.6.9) Clement, J. (1988). Observed Methods for Generating Analogies in Scientific Problem Solving," Cognitive Science, 12, pp. 563-586.

• (6.6.10) Cook, T. D. & Campbell, D. T. (1979). Quasi-Eperimentation: Design & analysis issues for field settings. Chicago: Rand McNally.

• (6.6.11) Dunbar, K. (1995). How scientists really reason: Scientific reasoning in real-world laboratories. In R. J. Sternberg & N. Davidson (Eds.)The Nature of Insight. The MIT Press, Cambridge, pp. 365-395.

• (6.6.12) Duncker, K. (1945). On problem solving. Psychological Monographs,58, Whole No. 270.

• (6.6.13) Ericsson, K. A. & Simon, H. A. (1996). Protocol Analysis: Verbal Reports as Data (rev ed.). Cambridge, MA: The MIT Press.

• (6.6.14) Gardiner, M. M., & Christie, B. (1987). Applying Cognitive Psychology to User-Interface Design. New York: John Wiley & Sons.

• (6.6.15) Gould, J. D. and Lewis, C. (1985). Designing for usability: Key principles and what designers think. Communications of the ACM, 28,300-311.

• (6.6.16) Hix, D. & Hartson, H. R. (1993). Developing user interfaces: ensuring usability through product and process. New York: John Wiley &Sons.

• (6.6.17) Hayes, J. R. (1981). The Complete Problem Solver. Philadelphia: The Franklin Institute Press.

• (6.6.18) Helander, M. (1989). Handbook of Human-Computer Interaction. Amsterdam: North-Holland.

• (6.6.19) Hewett, T. T. (1986). The role of iterative evaluation in designing systems for usability. In M. Harrison & A. Monk (Eds.) People and Computers: Designing for Usability. Cambridge: Cambridge University Press.

• (6.6.20) Hewett, T. T. (1995). Towards a generic strategy for empirical evaluation of interactive computing systems. In G. Perlman, G. K. Green, & M. Wolgalter (Eds.). Human Factors Perspectives on Human-Computer Interaction (167-171). Santa Monica, CA: Human Factors and Ergonomics Society.

• (6.6.21) Hewett, T. T. (1997). Designing with the mind in mind: Basic phenomena in human memory and problem solving. Tutorial notes for User Interface '97, Boston, MA: User Interface Engineering.

• (6.6.22) Hewett, T. T. & DePaul, J. L. (under review). General characteristics of a human centered scientific problem solving environment. Submitted for possible presentation at the ACM SIGCHI CHI 98 Conference on Human Factors in Computing Systems, April, 1998,Los Angeles, CA.

• (6.6.23) Hewett, T. T. and Meadow, C. T. (1986). On designing for usability: An application of four key principles. In Proceedings CHI '86Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery.

• (6.6.24) Hewett, T. T. & Scott, S. (1987). The use of thinking-out-loud and protocol analysis in development of a process model of interactive database searching. In H.-J. Bullinger & B. Shackel (Eds.), Human Computer Interaction-INTERACT '87,51-56.

• (6.6.25) Kerlinger, F. N. (1973). Foundations of behavioral research. (2nded.) New York: Hold, Rinehart & Winston.

• (6.6.26) Lindgaard, G. (1994) Usability testing and system evaluation. London: Chapman & Hall.

• (6.6.27) Luchins, A. S. (1942). Mechanization in problem solving. Psychological Monographs, 54, Whole No. 248.

• (6.6.28) Maier, N. R. F. (1931). Reasoning in humans II: The solution of a problem and its appearance in consciousness. Journal of Comparative Psychology,12, 181-194.

• (6.6.29) Newell, A., & Simon, H. A. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.

• (6.6.30) Norman, D. A. (1982). Learning and Memory New York: W. H. Freeman.

• (6.6.31) Norman, D. A. (1988). The Psychology of Everyday Things, Basic Books: New York.

• (6.6.32) Norman, D. A. (1993). Things That Make Us Smart. Reading, MA: Addison-Wesley.

• (6.6.33) Runkel, P. J. & McGrath, J. E. (1972). Research on human behavior: A systematic guide to method. New York: Holt, Rinehart & Winston.

• (6.6.34) Scriven, M. (1967). The methodology of evaluation. In Tyler, R.W., Gagne, R. M., & Scriven (Eds.) Perspectives on Curriculum Evaluation. Chicago: Rand McNally.

• (6.6.35) Simon, H. A. (1981). The Sciences of the Artificial. (2nd ed.)Cambridge, MA: The MIT Press.