evaluating systems, methods and there pro's and con's!

Blog 07.12.06

Today’s lecture was about the different methods of evaluating systems.

Observation and monitoring

· Watching the user interact with the system

· The user is either performing special tasks or performing normal work

· The users performance is recorded in some way

Direct Observation Limitations

· Can only record a proportion of the events

· We can only record certain types of events

· We only have one attempt at the observation

· We may simply miss events

· We might make recording errors

· User performance may be influenced by observation

Benefits

· Cheap

· We can concentrate on the information you are interested in

· Useful early in project for an informal feedback

· We can use a checklist to make data collection quicker and simpler

Video recorder limitations

· Users can be self conscious

· Can create more distance between users and evaluators

· Can be difficult to synchronise recording with eachother

· Analysis of video is time consuming 1 hour of tape can take over 5 hours to analyse

Benefits

· Can be synchronised with automatic logging

· Creates more distance between the users and evaluators

· Multiple cameras can capture different aspects

· Tools available for simplifying analysis

· We can revisit the data

Verbal protocols

· Where users talk through what they are doing

· Thinking allowed

Limitations

· Thinking aloud is difficult and can interfere with performance

· Thinking allowed is of dubious value

· User may need promoting which can disrupt them

Post event protocol

· Users view video of their actions and provides a commentary on it

· Good for situations in which the users has to concentrate when using the interface

· Doubts as to the abilities of users to recall their actions

· Users may rationalise their actions with benefit of hindsight

Software logging

· Does not require the researcher to be present

· Analysis of data can be automated to some extent

· Can be synchronised with video and audio recording

· Should make users aware they are being logged

· Expensive

· Large quantities of data

Users opinions

· Users interact with the system and then there subjective opinions are elicited

· Users attitudes are important and will effect the acceptance of a system

Structured interviews

· Predetermined questions

· Asked in a strict order

· No exploration of individual attitudes

· Good for comparing the responses of different subjects

· Good for doing statistical analysis

Flexible interviews

· Have some set topics

· No set sequence

· Interviewer can follow up on interviewees comments

· Relies on the skill of the interviewer to get the interview and make the person feel easy

· Interviewee may be reluctant to criticise the system

Structured verus flexable

· Structured are easier

· Easier to analyse

· Less scope for picking up relevant issues

· Both benefit from pilot studies

Checklist for semi structured interviews

· Why do you do this?-to get the users goal

· How do you do it?

· Why didn’t you do it this way?

Questionnaires and surveys

· Preparation of questions

· Can be used to reach a large number of people

· Closed questions(select from a group of options)

· Open questions(responds with personal answer)

We were also shown some examples of good and bad questionnaires

Analysis

· Mean

· Standard deviation

· More advanced statistics

Experimentation

· Users asked to do tasks in a controlled environment

· User performance is analysed

· Purpose of the experiment

· a hypothesis that can be tested

· what statistical methods will be used? which are best?

Results

· size of the effect

· alternative interpretations

· consistency

· generalisation

interpretive evaluation

· users perform natural tasks in the working environment

· often there is user participation in collecting analysing or interpreting the data

Context

· behaviour is only useful in context

· work context

· time context

· motivational context

· social context

predictive evaluation

· use a team of experts to review the system to predict the system

· present there findings as a report typically with a prescriptive element

limitations

· experts tend to have strong opinions, so having a few makes it more equal

· finding the right experts can be hard

· can require a alot of detailed knowledge

· experts cannot capture the behaviour of all the users

those were the notes i took based on the slides and comments of the lecturer(whos name i always forget.lol)in todays lecture!well thats it from me

be the music, not the scene

x