L503 Project Materials

System Evaluation

The Pyramid of Evaluation
General Description of the Client's Environment

Goals of this Stage
Possible Methods for this Stage

Scanning
Ethnography
Focus Groups
Structured Interviews
Surveys

Goals of Task Analysis
Possible Methods for Task Analysis

Structured Interviews
Timeline Interviews
Think Aloud (Cognitive Walk Through)
Surveys

The Pyramid of Evaluation

The process of evaluation is like an inverted pyramid.  You start by getting a general overview of the system and how it fits within the goals and structure of the organization. You then proceed to a more detailed evaluation of how well the system is supporting the tasks it is supposed to support.
 

General Description of the System and its Organizational Environment

Goals of This Stage:

• Establish trust between you and the client. If working in a group setting, establish trust among all of the participants.

• Understand the role of the system in the organization, identify stakeholders and the economic, political and social assets and issues surrounding the system, and grasp how the system is perceived by various stakeholders,  To do this, answer the following questions (see The Social Matrix of User Needs and Behavior:  Organizations as well as other course readings for more information about the description of the organization):

System Name:  What is the system called?

Organizational Environment:

 
Name and Location-- What is the name of the organization that is responsible for the system?  Where is it located?

Mission -- What is the mission of the organization?

Organizational aspects of the Information Use Environment:
 

People (Stakeholders) -- What groups and individuals have a stake in the success of the system? (Who manages the system? Who maintains the system? Who uses the system, either directly or indirectly? Are there other organizations, such as outside contractors,  that are involved in providing the service?)

Settings -- Where are tasks to be carried out by different stakeholders?  How do these settings impact the system and its use?  Settings are not just physical space, but how this space is shared and organizational rules for its use.

Problems --  For each of the stakeholding groups (whose use of the system may vary), what kind of problem-solving is the system designed to help with?  How do the various stakeholding groups judge the effectiveness of the system?

Resolution -- For each of the stakeholding groups, what is (are) considered desirable resolutions to the problems it faces?  What are the organizational values and customs pertaining to problem-solving?

Economic -- What financial resources are available for operating, maintaining, evaluating, and revising the system (money, time, supplies, equipment, manpower, equipment,  cost of training)?

Political -- Who makes the decisions about how the system should work, how it should be evaluated, who should operate it, etc.?

Social -- Are there special needs of people inside or outside the organization that impact on their use or management of the system (e.g., needs based on gender, ethnicity, religion, age, disability)?

Disciplinary -- Are there special standards that your discipline or profession sets for this system's performance?

Possible Methods for this Stage of the Evaluation

No matter how informal the methods you choose to use in any research activity, you should ALWAYS have prepared a statement of the goals of the session and a list of the questions that need to be answered.  This document is called a research PROTOCOL.  Examples are given in the linked documents for specific methods.  These are only for illustration. Use your judgment of what kind of questions need to be asked, and how, depending on the organization and the stage of research you are in. (For more about methods, see: Methods for Analyzing Users' Needs and Behavior as well as other course readings on that topic.)
1. Scanning:  Use this method to explore the environment of the client.  You can do this with little annoyance to the client and still learn a good deal about the client's environment, mission, tasks, etc.  Scan the electronic and print documents the stakeholders consult and produce that are related to the operation and management of the system.  If there are surveys or other evaluations of the system that have been done, be sure to scan these.  Write down questions that arise from reading these documents and note where they answer some of the questions mentioned above.
1. Ethnography:  Visits to observe the client's environment involve more intrusion and are harder to arrange, but, depending on the environmnent, can often be accomplished with ease.  There is no substitute for even casually observing people doing their tasks, seeing how they interact with one another, getting to know them and letting them get to know you, etc.  Ethnography can include observations of naturally-occuring activities, and conversations.  For more information on using ethnography, see:  Ethnography.

 
2. Focus Groups:  Group conversations or brainstorming sessions may be a good way for you to get to know your client, especially if you are working with a group, but they cannot replace seeing the client and relevant stakeholders in the environment where tasks will have to be carried out.  These sessions may be used as a part of ethnographic visits.  For more information on using focus groups, see:  Focus Groups.

 
1. Structured Interviews:  Structured interviews with key stakeholders are a good way to get their take on the system, its mission, challenges, etc.  People will often talk one-to-one in a way that they will not when in a group situation, when political and status relationships may prevent them from being completely candid.  For more information on structured interviews, see: Structured Interviews.
1. Surveys: It is unlikely that you will have the time or expertise to use a survey in this course.  However, if you think that a survey of stakeholders would be an appropriate method to use for your study, please consult the Instructor for guidance before proceeding.
Top of Page

Task Analysis

Goals of Task Analysis

During the preceding stage, you learned something about what the system is supposed to help different stakeholders do.  In this stage, your goal is to understand in detail:
 
• the tasks that the system is supposed to support
• how stakeholders interact with the system
• how stakeholders evaluate the system


Start with a description of the most important tasks and work down to the less important ones.  Answer the following questions for each task:

1. What is the name of the task?
2. Why is the task important to stakeholders (mission)?
3. What stakeholders are involved with performing this task?
4. Where is the task performed (= location)?
5. What special resources are needed this task (economic, political, social)?
6. What sub-tasks are required to complete the task?
7. What information acts are involved in performing this task?
8. How do stakeholders interact with the system to perform the task (= system interaction)?
9. What do stakeholders view as a successful outcome for this task (= resolution)?
10. How do stakeholders evaluate the effectiveness of their use of the system for various tasks (=evaluation)?

Possible Methods for Task Analysis

Structured Interviews: You can use structured interviews to get answers to the questions listed above.  These interviews may be combined with the Think Aloud method (see below).  If these methods are combined, be sure to start the session with the think aloud method, then follow up with the structured interview to find out how the stakeholder views his/her interaction with the system, the mission, etc.
 
Timeline interviews:  Timeline interviews are a form of structured interview.  They are designed to enable the researcher to identify, in a chronological fashion, the information gaps and bridges that a user faces in trying to perform a task.  Timeline interviews may be used to supplement the Talk Aloud method.  Timeline interviews rely on the memory of the user, and are therefore subject to omissions due to selective memory.  Their strength resides in the fact that they show you what the stakeholders view as important information acts and system interactions, and they provide you with the language that stakeholders use to refer to the system and the stages of performing a task.  For more information on timeline interviews, see: Timeline Interviews.

Think Aloud (Cognitive Walk Through): This is a technique that is useful for finding out what is going through a person's mind when they are performing a task, how they approach problem solving, how easy it is to use the system, how they decide when a search is complete, how they evaluate information, etc.  It may be used to help you understand how someone interacts with an information system, search for information, cope with errors, interpret a computer interface, etc.  For more information on this method, see:  Think Aloud (Cognitive Walk Through) Method.

Surveys:  Surveys are not a good method for studying the details of task performance.  However, surveys of user satisfaction might give you some idea of how representative your findings using qualitative methods are.  As noted above, it is unlikely that you will have the time or expertise to use surveys in this class.  Please consult the Instructor if you wish to use them.

Comparing Usability Targets and System Performance

Goals of This Step:
 

1. List the usability targets for each task, including targets for:

• Utility
• Integrity
• Aesthetics
• Ease of learning
• Ease of use
• Error handling


If possible, set numerical values as the targets for each task.

Based on your research so far, take each task and describe the usability targets that stakeholders have noted are desirable ones if the system is to be considered successful.  These targets should be as specific and measurable as possible.  Vague targets (e.g., "should be easy to use") will not be helpful when it comes time to describe how you would evaluate the system.  The more specific you can be, the better.
 

1. Compare system performance with usability targets.

 

 

Based on your observations, how close does the system come to meeting all the targets on the list?  In what ways do they fall short? Why?

Written Evaluation Report (max. 10 double-spaced pages)

Summarize your descriptions of:

• the organizational context of the system
• major system tasks
• usability targets for major tasks
• ways in which the system does and does not come close to meeting usability targets

• Date of session
• Method(s) used
• Goal of session
• Number of stakeholders taking part in a session
• Protocol used (label by date, time, location)