Theme 3: User and Tasks

This theme explores limitations and abilities of the user as related to interpreting images. The influence of the task(s) to be accomplished by the user during the interpretation of images is given considerable emphasis.

(3.1) Human Performance Issues

The following discussion is a broad overview of some of the basic characteristics of the human visual system and of perception and how it is thought to work. For amore in depth discussion, the reader is referred to such sources as a good Introductory Psychology text book (e.g., Sternberg, 1995), a good text book on Perception (e.g.,Maitlin, 1997), a good book on Cognition (e.g., Reisberg, 1997), and Solso's(1994) work exploring the intersection of cognitive psychology and art.

(3.1.1) Perception

Perception is both a Top Down and a Bottom Up process involving contributions by the sensory systems, and the ways in which the neurons in the human nervous system interact with each other, as well as contributions from prior experience and learned expectations.

(3.1.1.1) The human visual system

The primary subsystem of the nervous system which will be of interest in dealing with visualizations is that of the human visual system.

(3.1.1.1.1) Biological

At the basic biological level there are neurons and the type of stimuli which affect them to be considered.

(3.1.1.1.1.1) Effective sensory stimuli

When we are looking at the visual system it is clear that the physical stimulus of interest will be light and some aspect that physical phenomenon. It is important, however, to recognize that the effective stimulus which activate various components of the visual system will depend in part on the types of neurons being considered and the response of those neurons to some physical characteristic of light. In general, since neurons adapt to continuous stimulation what is critical in their interaction with the world is a change in stimulation, i.e., the effective sensory stimulus is a changing one.

(3.1.1.1.1.2) Types of neurons

Within the visual system there are several types of neurons. In the eye alone there are rods, three different types of cones, bipolar cells, horizontal cells, amacrine cells, and ganglion cells. For all practical purposes, however, it is possible to treat all of these cells as a "black box system" and concern ourselves with the fact that there are two types of functions performed by these cells, some of them are receptor cells and others are transmitter cells.

(3.1.1.1.1.2.1) Receptor cells

The receptors cells, which constitute the first layer of cells in the visual system, generally fall into two categories, the rods and the cones. Both of these types of cells convert light energy into patterns of neural firings.

(3.1.1.1.1.2.1.1) Rods

The rods are longer and narrower than cones and respond selectively only to changes in the intensity of light. These cells are so sensitive that they have been shown to respond to the presence of only a single photon. Generally the are large numbers of rods collecting light and providing stimulation eventually to a single ganglion cell, the type of cell which has a fiber leaving the eye and becoming part of the optic nerve. This generally means that there are assemblies of cells in the eye which are quite sensitive to the presence of light but which lack the ability to discriminate finely where the boundaries of that light may be located in the world.

(3.1.1.1.1.2.1.2) Cones

There are three populations of cones in the eye, those that are maximally sensitive to red light, those that are maximally sensitive to green light, and those that are maximally sensitive to blue light. In fact wavelength characteristics of these populations is known quite well from their absorption spectra, i.e., the wavelengths of light which are not reflected back out of the eye to the degree which other wavelengths are reflected. Generally there are relatively few cones collecting light and providing stimulation eventually to a single ganglion cell. This generally means that there are assemblies of cells in the eye which are quite sensitive to colored edges, boundaries, etc.

(3.1.1.1.1.2.2) Transmitter cells

There are a variety of transmitter cells in the eye. If the receptor cells are thought of as the first layer of cells, the bipolar and the ganglion cells are the second and third layers, respectively. The horizontal cells make connections among the receptor cells and modify the activity which takes place between the receptors and the bipolar cells. The amacrine cells provide a similar horizontal organization at the layer of the connections between the bipolar and ganglion cells. It is the ganglion cells which contribute the fibers which join together to form the optic nerve.

(3.1.1.1.2) Psychophysical

In looking at the relationship between the physical characteristics of light and the psychological response to them it is clear that there are three fundamental psychological dimensions of light which show a correspondence with physical characteristics of light, hue saturation, and brightness. The relationship between the physical and psychological characteristics of light is non-linear (e.g., it takes considerably less light energy make a light source "twice as bright" when that light source is in a darkened room that when that light source is in a brightly lit room).

(3.1.1.1.2.1) Hue

The psychological dimension of "hue" corresponds to what is popularly referred to as the color of the light reflected by an object or produced by alight source. The effective physical characteristic of the light that produces variations in hue is its wavelength. It is primarily on the basis of differences in wavelength that we assign color names across the visible spectrum from Red (through Orange, Green, Blue, Indigo) to Violet.

(3.1.1.1.2.2) Saturation

The saturation of a color corresponds to what the relative proportion of that particular wavelength reaching the eye from a particular light source. For example, when most of light reflected from a patch of cloth is red the object is seen as having a very intense red color, that is saturated. If the full range of wavelengths is present with red having only a slight predominance the color will appear quite pale and be less saturated.

(3.1.1.1.2.3) Brightness

The brightness of a light is the psychological dimension of light which corresponds to the intensity of the light, i.e., the number of photons falling on a particular region of the retina at a particular moment in time.

(3.1.1.1.2.4) Preattentive visual

(3.1.1.1.3) Visual Phenomena

When examining the visual system and this level of analysis there is an important process operating which is responsible for a number of particularly critical visual phenomena.

(3.1.1.1.3.1) Lateral Inhibition

Atone level of analysis there are two basic process by which neurons influence each other: excitation and inhibition. The process of excitation has occurred when an increase in the firing rate of one neuron leads to an increase in the firing rate of another neuron upon which it is having it’s effect. The process of inhibition has occurred when an increase in the firing rate of one neuron leads to a decrease in the firing rate of another neuron upon which it is having its effect. For example, when measured by the effect produced upon a ganglion cell leaving the eye, stimulation of a particular set of receptor cells can result in an increased firing rate in one ganglion cell and a reduced firing rate in some other ganglion cell since. Similarly, the output of a ganglion cell leaving the eye may be the net result of the effects of several inputs, some of which are inhibitory, some of which are excitatory, but all of which are the result of stimulation of multiple receptor cells.

(3.1.1.1.3.2) Color Afterimages

It is this process of lateral inhibition which is responsible for several visual effects, for example, color afterimages. Imagine a very simple network consisting of three cells, two of them receptors and the third a transmitter. Let one of the receptors be maximally sensitive to red light and let the other be maximally sensitive to green. Furthermore, let the transmitter cell have anon-zero base firing rate even when none of the receptors are being stimulated. So that this neural circuit behaves in a way that is consistent with the eye, let both receptors be connected to the transmitter cell, but let one of the receptors have an excitatory effect and the other an inhibitory effect. For purposes of discussion, assume that the red receptor has an excitatory effect when stimulated and that the green receptor has an inhibitory effect when stimulated. Furthermore, assuming equal amounts of red and green light, let the about of excitation produced by the excitatory receptor be equal to the amount of inhibition produced by the inhibitory. Under these circumstances stimulating the neural circuit with white light, which contains equal amounts of red and green light results in no change in the firing rate of the transmitter cell. Now expose the receptors to a couple of mines of lots or red light. The red receptor gets a considerable work out, the green is receiving little or no stimulation, and the net result is that the ganglion cell firing rate goes up markedly as a result of the excitation effect produced by the red receptor. Meanwhile the red receptor has used up any reserve of stored chemicals and is firing only as rapidly as it can regenerate the chemicals it needs to produce an electrical impulse. Next the receptors are exposed once again to white light, containing equal amounts of red and green light. The red receptor continues firing and producing its excitatory effect on the ganglion cell. However, the green receptor, being fresh and having a stored reserve of the chemicals it needs to produce electrical impulses and have an inhibitory effect on the ganglion cell, is fresher and fires more frequently than does the red receptor. Even though the light hitting the receptors is white, the red receptor is not able to keep pace with the green receptor and there is more inhibition being produced on the ganglion cell than there is excitation. The net effect is that the ganglion cell fires below its normal white light base rate and so sends a message to the brain that the incoming light is green.

(3.1.1.1.3.3) Opponent process color coding

The color afterimage effect described above illustrates the opponent process nature of the coding of color as it leaves the retina of the eye. The interconnections of the various cells and the patterns of excitation and inhibition have created a neural circuits such that some ganglion cells behave in one way to red light and in the opposite way to green light. That is the cells respond selectively and differentially to complementary colors. Similarly, other ganglion cells display an opponent process response to blue and yellow light, thereby illustrating some of the complexity of the connections and effects since there are no receptors in the eye that are maximally sensitive to yellow light, but yellow and blue are complementary colors and each produce the other as an afterimage under appropriate conditions.

(3.1.1.1.3.4) Edge Enhancement

This same process of lateral inhibition operates to sharpen edges and to increase the contrast between areas of lightness and darkness. When ganglion cells right along either edge of a patch of light are receiving inhibition and excitation from only from a portion of the field of receptors which normally influence them. Those on the dark side of the edge are receiving some inhibitory input without any excitatory input and those along the light side of the edge are receiving excitatory input without some of the inhibitory input which would normally be present if the full receptive field were being stimulated. Thus the ganglion cell on the dark side of the edge fires more slowly than do the rest of the ganglion cells whose receptive fields are in darkness. The ganglion cell on the light side of the edge fires more rapidly than do the rest of the ganglion cells whose receptive fields are fully impacted by the light.

(3.1.1.1.3.5) Black, white and shades of gray

Many of these basic stimuli involve the shades of brightness of different objects in the world and the contribution of the neural processes which extract information from the world and process it.

(3.1.1.2) The human memory system

While a more extensive discussion of memory appears in section 6, it is helpful here to provide a brief review of the characteristics of human memory since that memory is where the complex knowledge structures which guide perception are stored. The major types of memory to be concerned with here are working memory and long term memory.

(3.1.1.2.1) Working memory

Working memory is a limited capacity system which can keep about 7 plus or minus two "chunks' active for approximately 12 seconds. After about 12 seconds these chunks have decayed and are no longer available for use in working memory. Decay in working memory can be prevented by rehearsal of the chunk, i.e., by simple repetition. Although new chunks can be added to working memory, either by introduction from the perceptual world or by activation of a node in long term memory, the capacity limitation will result in some chunks being lost if the number of chunks being juggled exceeds the capacity of working memory. As a unit of storage the chunk is somewhat problematic in that when one considers some basic physical units it turns out that sometimes chunks contain only one of something and sometimes chunks contain much more. For example, a chunk may be only a single one digit number from a randomly presented string. Or alternatively a chunk might well contain several digits, as in the case of moving 555-PUCK into working memory so that one can call for hockey tickets. There are reasons to believe that any principle or basis for the organization or structuring of long term memory can serve as the basis for chunking.

(3.1.1.2.2) Long term memory

In general long term memory refers to those memories which may persist of many years but are not active in working memory. It is useful to think of the organization of long term memory as being something like a complex semantic network of concepts and relationships among concepts. Activation of one node in the network can result in an increased likelihood of activation of nearby nodes. The range of things stored in long term memory is quite broad and includes such things as representations of events that have been experienced and a variety of knowledge structures that have been acquired by reading or thinking.

(3.1.1.3) Theories of Perception

(3.1.1.3.1) Bottom up processing

The most widely accepted view of how perception works is that there are a number of basic physical stimuli which provide input to the sensory system. These include the various characteristics of the world which are responsible for color and for black, white, shades of gray. These physical characteristics of the world induce activity in the nervous system and that activity is influenced and patterned by the interconnections of neurons and processes (excitatory and inhibitory) which take place at those interconnections.

(3.1.1.3.1.1) Depth, motion, and others

In addition to the raw elements of sensory stimulation created by the physical characteristics of patterns of light energy, there are consistent relationships between some of these patterns which are either extracted directly or which are interpreted quite directly and immediately on the basis of prior learning and experience with the world. For example, there are a variety of depth perception cues which come into play in making judgments of distance. Some of these, such as the relative size of two objects or whether or not one object appears to be interposed between the observer and a second object do not depend upon having two eyes. Other cues such as those associated with binocular disparity result from the brain having to integrate two different sets of information from the optic nerve.

(3.1.1.3.1.2) Pattern recognition

Perceived forms fall into a variety of patterns. Letters, faces, and visual relationships all constitute examples of patterns are easily recognized on the basis of experience. Some of the information used to recognize patterns is directly contained in the patterns of sensory input. However, there are a number of convincing demonstrations which show that the recognition of patterns is also guided by one's prior experiences and expectations.

(3.1.1.3.2) Top down processing

Patterns, people, and other.

(3.1.1.4) Perceptual Phenomena

There are a wide variety of perceptual phenomena which operate to determine an observer's interpretation of a visual scene. These include such things as the factors described by the Gestalt Psychologists, the Gestalt Laws of perception, the movement of objects, either real or apparent, and various types of perceptual constancies.

(3.1.1.4.1) Gestalt laws of perception

The Gestalt Psychologists identified several laws of perception which could be determined from the behavior of observers. These laws are: figure ground perception, proximity, similarity, closure, continuity, and symmetry. In the case of figure ground perception, as in the classic example of the vase and faces, one tends to see either the vase or the faces as the figure with the other as background. The law of proximity describes the observation that humans tend to see objects as a group when they are physically close to each other. The law of similarity describes the observation that similar objects tend to be grouped with each other. The laws of continuity and closure describe the observation that humans tend to perceive as a single object things which have smoothly flowing forms rather than disrupted ones and that almost complete objects tend to be seen as being complete. The law of symmetry describes the observation that human tend to perceive some mirror image forms as being part of a whole object.

(3.1.1.4.2) Integration of images across time

The perception of motion, whether actual or induced, illustrates that the visual nervous system tends to integrate information across time. Movies or cartoons, which make use of the principle of stroboscopic motion, are actually composed of a series of still pictures (frames) which are shown at a rate of about 24 frames per second. The eye, however, sees the appearance of movement rather than a series of still pictures. There are some who have argued that all motion perception is actually the result of a similar phenomenon that results from the way in which the eyes interact with the continuously varying world.

(3.1.1.4.3) Perceptual constancies

A perceptual constancy exists when there is an unnoticed discrepancy of a particular kind between the stimulus provided by the world and the observers interpretation. In the case of size constancy, an observer who knows the individual in question will attribute the same height to another person regardless of whether or not that person is standing 5 feet away or 200 feet away. Similarly know objects tend to be thought of as having the same shape regardless of the angle from which they are viewed. In the case of color constancy, despite changes in the hue of light shining on known objects the observer continues to perceive the color as being the same as when the objects are illuminated with white light.

(3.1.1.5) Avoiding Traps

(3.1.1.5.1) Color blindness

Certain types of color blindness result from a deficiency in one or more populations of cones or the chemical processes which make them work. Any visualization which codes information which requires the ability to discriminate various colors from one another may very well contribute to creating a new category of disadvantaged users. Statistically, males are more likely to be represented in this group than females.

(3.1.1.5.2) Adhere to the conventional meanings of colors

While this may seem like a trivially obvious recommendation, there are many instances where it has been ignored. For example, in most western societies red lights are associated with danger, green with safety, and yellow indicates a situation in which caution should be exercised. None the less, it is not unheard of for a blue light to be used to signal that a particular subsystem is over heating and needs cooled off. While this may seem reasonable in that blue is normally a color associated with coolness, flashing yellow is the color typically associated with a warning or cautionary situation.

(3.1.1.5.3) Some other guidelines for the use of colors

(from a presentation at the Delaware Valley Chapter of the Human Factors and Ergonomics Society, Principles of Color and Colorimetry by William A. Breitmaier, 30 Apr.,1997).

(3.1.1.5.3.1) Use white or green letters on a black background.

(3.1.1.5.3.2) Use black letters on a white or desaturated cyan background.

(3.1.1.5.3.3) Do not use more than six colors for information coding and no more than four colors for text.

(3.1.1.5.3.4) Do not use pure blue for fine detail or small objects.

(3.1.1.5.3.5) Unless you want to show depth, avoid using saturated red and blue together. These colors cause chromosteropsis.

(3.1.1.5.3.6) Use complementary colors for maximum color contrast.

(3.1.1.5.3.7) Some color "meanings" change with context. Most people are familiar with the fact that the colors used for letters and background can influence the difficulty with which text is read. There are also cases where the luminance of colors in a display can make the interpretation of what is seen in the display more or less difficult. (de Weert, 1988, has provided some examples).

(3.1.1.6) Contributions of senses other than vision

(3.1.1.6.1) Sound

(3.1.2) Design Issues

(3.1.2.1) Visual context (scale bars, annotations, ...)

(3.2) Visualization and task goals

In developing a visualization and the mechanisms for interaction with that visualization, it is important to remember that the user is interested in solving some particular problem in a particular problem domain. The focus of a user on a particular domain of interest while deriving meaning from a visual representation is called his/her visualization or task goal. Both the visualization and the tools available to work with it must be suited to the user's task. To convince students of the importance to understand goals when mapping from data to visual representations, examples make visible the vast difference of effective versus ineffective representations. While visualization and task goals are application dependent, a feeling for possible categories of tasks, such as comparing, identifying, associating, or correlating, need to be conveyed to students. Finally, procedures to evaluate progress on the interpretation of an image by a human need to be introduced.

(3.2.1) An example

Imagine a high-end visualization program. Within the visualization window a 3D object is represented, e.g., a complex surface in grid mode with portions of the surface wrapping around, intersecting with, and concealing other portions of the surface. In the lower region of the scroll bar on the right hand side of the window there is a rotator wheel. The user can "grasp" the wheel with the pointer and move either upwards or downwards to signal the computer to rotate the visual object around its horizontal axis. In the scroll bar across the bottom of the window is another wheel controlling rotation about the vertical axis. Each wheel is carefully designed to recreate the appearance of physical rotator wheels, even to creating the visual effect of having the top and bottom (or right and left) edges of the wheel appear to recede behind the plane of the scroll bar. The objects are clearly rotator wheels and their intended function is obvious even to an inexperienced user. Now consider the following scenario. Assume the 3D representation is computationally intensive and that the user wants to be able to discriminate an intersection or some internal aspect of the structure which is partially concealed by outer portions of the surface. There is a time lag between moving the rotator wheel and when there drawn figure appears on the screen. Further, there is no visible cue by which the computer signals the user, "I’m processing." Uncertain of the degree of rotation effected by his first attempt to rotate the 3D object, and seeing no visible result of having moved the rotator wheel, the user "grasps" the wheel and tries again. Naturally this action aborts the earlier processing and tells the machine to compute the new position of the object. Still having had no feedback after a second and third attempt to rotate the figure, the user next tests the horizontal rotator wheel. Again there appears to be no direct result. (Meanwhile the computer has been busily and patiently re-computing a new position of the 3D object each time it has been instructed to do so.) Finally, the user pauses in his manipulation of rotator wheels to try and determine what he is doing wrong. Suddenly, with no apparent transition, the 3Dobject appears in its most recently computed position. While it is true that a really nice interface on an under powered machine won’t serve the user’s needs, a bigger, faster machine won’t solve the real problem here. The user’s frustration is not determined solely by the fact that the machine takes time to compute each new view of the 3D object. Within limits, time is a minor irritant compared to the fact that the user is unable to see what he wants to see in the 3D representation. The control afforded by the rotator wheel is not fine enough to suit his needs. There is no simple and direct way for him to calibrate his movements to results. Too many cognitive operations intervene between views.

(3.2.2) Effect of problem representation upon ease of solution

Just as the flow of interaction with a visual representation can be disrupted by failing to provide the interaction tools needed to support the user's goals, so to the representation itself can make a difference in the ease with which the user can make use of the data or information being visualized. Even though this is ultimately the reason for the existence of visualizations, it is sometime possible to forget that not all representations are created equal and that some representations are better than others. "Better," however, needs to be at least partially defined in terms of the user's goals and task-oriented needs.

(3.2.3) Task Analysis and problem need

(i.e., what are the demands of the task).

(3.2.3.1) Focus on Cognitive Aspects of user needs and goals

The importance of focusing on the cognitive aspects of the user's task oriented needs and goals is a recurring theme in the literature of human-computer interaction (See, for example Norman, 1993; and Preece, Rogers, Sharp, Benyon, Holland & Carey, 1994),human factors and ergonomics (See for example, Kirwan & Ainsworth,1992; Rasmussen, 1986; and Reason, 1990), and a variety of design disciplines (See for example, Cross, 1984; Dreyfuss, 1955; Heskett, 1980). Indeed, this concern for the user's task can be seen as the basic idea behind many of the recommendations made by Tufte (1983, 1990, 1997).

(3.2.3.2) Different types of user tasks

User tasks may be understood as a general criteria to fulfill (e.g. "granting exploration of data") or as a very specific assignment (e.g." is the flow of water symmetrical to any of the axes"), and my be stated independent or dependent of problem domains. Examples of general and domain independent tasks are:

(3.2.3.2.1) Exploration: What is out there? What if a certain parameter changed?

(3.2.3.2.2) Confirmation: Is there evidence? How much evidence is there?

(3.2.3.2.3) Presentation: Is the point to be made clear? Is the point to be made simple?

(3.2.3.2.4) Time Critical Decision Making. Can a decision be reached within the allowed time? (e.g. cockpit displays, nuclear plan emergency, real time reactive systems).

(3.3) Evaluation

There is need to assess the quality of the visualization at least in dependency of user and task. There are two types of evaluation-formative and summative. Summative evaluation is the competitive evaluation conducted when two or more alternatives are compared in an attempt to determine which is the better alternative. For example, a summative evaluation of a graph might involve having some students look at a non-visual presentation of data and some at the visual presentation in order to assess which is most effective. Formative evaluation is the type of evaluation conducted when one is attempting to determine whether or not the graph contains all information and whether or not it is ready for the summative evaluation. For a further discussion of formative and summative evaluation and an extended example of how a variety of evaluation procedures were put top practical use in designing and interactive computing system, see Hewett (1986). Evaluation involves at least the following steps:

(3.3.1) Establishing a goal statement which can be assessed

Effectively, the most critical part of formative evaluation is that one must have one or more clear cut goals which have been set as a basis for guiding the design and construction of the program or artifact.

(3.3.2) Determining progress towards that goal

The evaluation then involves determining the degree to which one is moving towards either the original goal or a towards a new goal which has been established at the result of the type of clarification that results from learning more about what one is doing in dealing with unstructured problems. One of the benefits of frequent periodic formative evaluation is that it helps to conserve resources by preventing one from drifting too far off course.

(3.3.3) Some procedures for formative evaluation

There are a variety of techniques for doing a formative evaluation and some of these are discussed below in section 6. In the context of working with users from one's target population these basic techniques are typically the more open ended ones where one leaves room for the unexpected to happen. That is one observes users in some way and listens to them describe or talk about things which they are doing and/or thinking. Often these open ended procedures will reveal the nature of user's underlying confusion about the meaning or interpretation of a visualization. furthermore, they will often result in potential useful suggestions about how to do things more clearly or simply.