evaluating systems, methods and there pro's and con's!

Blog 07.12.06

Today’s lecture was about the different methods of evaluating systems.

Observation and monitoring

· Watching the user interact with the system

· The user is either performing special tasks or performing normal work

· The users performance is recorded in some way

Direct Observation Limitations

· Can only record a proportion of the events

· We can only record certain types of events

· We only have one attempt at the observation

· We may simply miss events

· We might make recording errors

· User performance may be influenced by observation

Benefits

· Cheap

· We can concentrate on the information you are interested in

· Useful early in project for an informal feedback

· We can use a checklist to make data collection quicker and simpler

Video recorder limitations

· Users can be self conscious

· Can create more distance between users and evaluators

· Can be difficult to synchronise recording with eachother

· Analysis of video is time consuming 1 hour of tape can take over 5 hours to analyse

Benefits

· Can be synchronised with automatic logging

· Creates more distance between the users and evaluators

· Multiple cameras can capture different aspects

· Tools available for simplifying analysis

· We can revisit the data

Verbal protocols

· Where users talk through what they are doing

· Thinking allowed

Limitations

· Thinking aloud is difficult and can interfere with performance

· Thinking allowed is of dubious value

· User may need promoting which can disrupt them

Post event protocol

· Users view video of their actions and provides a commentary on it

· Good for situations in which the users has to concentrate when using the interface

· Doubts as to the abilities of users to recall their actions

· Users may rationalise their actions with benefit of hindsight

Software logging

· Does not require the researcher to be present

· Analysis of data can be automated to some extent

· Can be synchronised with video and audio recording

· Should make users aware they are being logged

· Expensive

· Large quantities of data

Users opinions

· Users interact with the system and then there subjective opinions are elicited

· Users attitudes are important and will effect the acceptance of a system

Structured interviews

· Predetermined questions

· Asked in a strict order

· No exploration of individual attitudes

· Good for comparing the responses of different subjects

· Good for doing statistical analysis

Flexible interviews

· Have some set topics

· No set sequence

· Interviewer can follow up on interviewees comments

· Relies on the skill of the interviewer to get the interview and make the person feel easy

· Interviewee may be reluctant to criticise the system

Structured verus flexable

· Structured are easier

· Easier to analyse

· Less scope for picking up relevant issues

· Both benefit from pilot studies

Checklist for semi structured interviews

· Why do you do this?-to get the users goal

· How do you do it?

· Why didn’t you do it this way?

Questionnaires and surveys

· Preparation of questions

· Can be used to reach a large number of people

· Closed questions(select from a group of options)

· Open questions(responds with personal answer)

We were also shown some examples of good and bad questionnaires

Analysis

· Mean

· Standard deviation

· More advanced statistics

Experimentation

· Users asked to do tasks in a controlled environment

· User performance is analysed

· Purpose of the experiment

· a hypothesis that can be tested

· what statistical methods will be used? which are best?

Results

· size of the effect

· alternative interpretations

· consistency

· generalisation

interpretive evaluation

· users perform natural tasks in the working environment

· often there is user participation in collecting analysing or interpreting the data

Context

· behaviour is only useful in context

· work context

· time context

· motivational context

· social context

predictive evaluation

· use a team of experts to review the system to predict the system

· present there findings as a report typically with a prescriptive element

limitations

· experts tend to have strong opinions, so having a few makes it more equal

· finding the right experts can be hard

· can require a alot of detailed knowledge

· experts cannot capture the behaviour of all the users

those were the notes i took based on the slides and comments of the lecturer(whos name i always forget.lol)in todays lecture!well thats it from me

be the music, not the scene

x

Evaluation

Today’s lecture was about evaluation. Evaluation is about gathering data, about the usability or design of a product. We are interested in a particular set of users, related to a particular set of tasks. We are not interested in every user. And we need to evaluate which situation it will be used in and in what location?

Why are we evaluating?

· to understand the real world

· to compare designs

· to engineer the product towards a target

· to check conformance to a standard

· to ask different questions at different points of the process.

Real world

· what does the user actually want?

· Remember the designer is not the end user, what we want is not what they want!

· What problems do users experience? What is wrong with the current system?

· What tasks do users perform

· How do users perform tasks

· What problems do they experience

Comparing designs

· Commercial competition? What do the rivals do, what is good? What can we improve?

· Which design is best and what do we mean by best? What makes it the best? Speed, appeal?

· Answer specific design questions

Engineering towards a target

· Aiming for specific performance measures

· Improving areas of an existing design

· Being better than a competitor

Conformance to a standard

· ISO standards

· Accessibility standards

· Health and safety standards

· In-house standards

· Design guidelines

When to evaluate

· Formulative evaluation

o Evaluation as part of the design process

o Predesign proof and viability

o Initial design with user

o Interactive design

o Acceptance testing

· Predict usability of system or aspects thereof

· Check design teams understanding of user requirements

· Test out ideas quickly and informally

· Summative evaluation

o Evaluation of a essentially finished system

o Interactive design

o Acceptance testing

· Identifying user difficulties so products can be finely tuned

· Upgrading product

Factors

· Kind of information required

· Nature of system or specification under evaluation

· Stage in the life cycle being evaluated

· Whether or not statistical validity is required

· Resources available

Qualitative evaluation

· Descriptive

· Typically non-numerical

· May be subjective

· Introspection

· Direct observation

· Query via questionnaires and interviews

· Continuous evaluation through feedback and field studies

Quantitative evaluation

· Precise measurement

· Numerical values

· Bound on how to correct our statements are

· User performance

· Controlled experiments

· Statistical analysis

Measurable quantities

· Time taken to complete task

· Number of tasks completed within a given time

· Ratio between successful interactions to errors

· Time spent recovering from errors

· Number of errors

· Number of commands or features actually used by the user

· Number of features user can remember after the test

This was basically everything said in the lecture. Not long until the Christmas break!w0o0o0o0o0o0o0o0o0o!!!!!Bye for now!

useability

Today’s lecture was about usability. It was our first lecture with Daniel Cunliffe. Who will be lecturing us for the next 4 weeks. In our lecture we were introduced to the basics of usability. We are going to be focused on multimedia applications such as Macromedia suite or Adobe photoshop. Multimedia applications are used in a variety of contexts. Some are only used once such as software at trade shows and others are used multiple times, such as presentations in shops. They can be used for single users(one user using the software alone) or multiple users. You must consider what the application is used for. Eg. Informing/ teaching/entertaining. And who is going to use the software? The program will have to be different for novice users as opposed to more experienced users as they will not know what to with the software straight away and will struggle to adjust. What location are the users going to be using the software? Online test will have supervision for example and the program could be designed for the specific computers. However on a web based application you have little or no control of the location it will be used in. we can use the CUTS model which stands for Context (how will it be used?, in what situation?),User (who will use it? Are they experienced users or do they need a more simple system? What do they want from the application? What do they need for it? How do you motivate the user to use the application?), Task(what is it made to do? Does it do the same task all the time? How many times a day is it used?) and System (does it require standard equipment such as mouse and keyboard or can you use different input devices? What type of specification is required? Processing speed, how long will it take to do a required task? What about connection speed?) to realise what we need to consider when developing an application. Definition of effectiveness: “The effectiveness efficiency and satisfaction in which specified users can achieve specified goals”. Also the system should be pleasant to use and relatively straight forwards to operate. It should not require too much concentration. The system should have good functionality but also be useable and suit the environment.

Copywrite laws

This weeks lecture was on Copy write laws. It covered many of the basics of copy write law to do with software and multimedia. Copy write effects all of the music you listen to or buy, and most of the programs you will own or “aquire” ;).for example if you pay for a CD album, you are only allowed to listen to that CD privately. Although you have paid for the CD you are not allowed to make a back up of the music, copy it to your computer or even convert it to mp3 to use on your mp3 player. Also you are not allowed to lend the CD to a friend or even relative as this is technically a breach of copy write. Software is also covered by a version of copy write law. However these laws are a little more reasonable as you are allowed to make a back up of the software as long as you keep it to yourself and don’t give it to others. Some software gives you an opportunity to “try before you buy”. This is called Shareware. These programs give you limited features, a certain amount of time to try them or often reminders to purchase the product. If you choose to buy you are given a serial number which can be used to activate the full product. This gives people the chance to see if the program is for them before forking out large amounts of money for software. However these sharewares can be “cracked” to allow the user to use the full product for free. This is done by modifying some of the files or obtaining a serial number from a third party, this is of course a breach of copy write and illegal. There is also open source software which is free from the off. You can also modify this software to make it work how you want it to. I think this will be the future of software as the gap between open source and company made software is closing.

We also were shown the music video for Nizlopi’s shit song “the jcb song”. The video was made in flash and was quite impressive. Its just a shame it was for one of the worst written, most commercially driven song I have ever heard. But personal opinions aside, the video was a work of art.

Computing for all

Today’s lecture was about disabled computer users and how you should design projects to incorporate there needs. It is actually law to accommodate less able users. Users have all different needs. Some have vision difficulties which means you need to choose sensible colour schemes. Others may be deaf so any audio files will need to have subtitles or a description of what they contain as the information would otherwise be useless. Others cannot see at all and use software such as dragon dictate or JAWS to speak text to them. For this reason images need to be captioned and videos annotated. These measures as well as others are all steps towards the internet becoming accessible to all users. You can have your website tested to see how user friendly it is, and if you pass you can put a logo that states it is for all users on your page. We also watched a video which shows how users with different disabilities use computers. Some have a brail pad, which translates text, line by line. Other users who do not have use of there arms move there heads onto sensor pads to interact with the computer. It is amazing what can be done to make technology available to all.

Crap sites of the web unite!

Todays lecture was on Human computer Interfaces(HCI's).
Hci's are designed to aide the user and to make the operation of a computer more simple and user friendly. HCI's range from the Graphical user interface of an operating system, such as Windows Vista or Mac OSX. Features of these HCI's are things like the windows, tabs, drop down and pop up menus, or the much dreaded cascade menu(which single handedly pushed me to uninstall windows media player 10!). Icons are also used in these operating systems to help the user identify files, folders or programs more quickly. These HCI's replaced the DOS operating system, which was command based, instead of the array of colours, sounds and graphics that make up more modern operating systems. Today DOS only really exists as an emulated platform in windows ME,XP and XPpro. Although not user friendly it is very useful for more advanced users and often alot faster than going through all the windows,directories and menus of a HCI. HCI's are also used in websites. Web designers utilise lots of methods to add to the user experience and make there sites more appealing and useable. for instance text can be made bold or underlined to make it stand out, flashing text can also be used as well as using text that stands out from the background colour. Text can also be presented in different ways to add to its apperence. You can use CAPITAL LETTERS to make the words stand out from the page. or you can allign the text to the right with a jagged left edge to make it stand out from other infomation. Also the use of colours can be used to make the page more appealing although you should tread carefully in this area. You cannot choose a colour scheme to please everyone but you should hold certain things in mind when you choose the pallette of your design. As a rule you should limit yourself to 4 colours so the page will not become overdone. You should also bare in mind that not all monitors can display as many colours as others.some for example 256bit monocrome monitors can only display 256 colours and due to this many will not display correctly. for this reason it is advised that projects are designed in monocrome first. It is also important to concider that colours represent different things in different cultures, for example in the western world blue is a calming colour and is quite regal. But in japan it is concidered to represent "dirty work" or generally frowned on activity. As you can see user concideration is paramount in web design. Disabled users with disabilitys such as colourblindness cannot see certain colours and others can only see shades of grey. Even the healthy eye has problems with certain colour schemes. it is very difficult for the human eye to destinguish between the primary colours blue and red quickly.
In the end of the lecture we were shown examples of good and bad websites. radioqualia.va.com is a good example of a terrable website. the site had no real navigation and was just barely viewable icon leading to bland illegable text! it was soo bad it was frustrating to look at. Also arngren.net was an example of a totally overcrowded website! it was as bad as milliondollarhomepage.com, but atleast that guy made a cool million for barly any effort! i learned the web is great if sites are done well, however a poorly designed site is either annoying and frustrating or just simply a laughing stock!

peace
x

stick death!

this weeks lecture was about multimedia design techniques. in the lecture we learned about various techniques you can use to create a multimedia project. we were told about the need to consider the user of the project and not to please your own tastes. we were made aware that certain colour schemes would be hard for certain people to see and were shown first hand how frustrating it is if a flash presentation has over the top navigation. if the navigation is too complicated most users will "switch off" or just give up on the page all together. however we were also showen a very creative flash movie based in the flash2004 program. i stick man is drawn on the screen and is then a target of attacks by various features of the program!the stick man defends himself and it turns into an out and out war with the cursor towards the end of the movie. this video was hilarious, even better than wedding crashers. it was soo impressive because it was a simple idea executed soo well.i was glad that this was an interesting lecture because i was fit to drop after the 5 hours of lectures i had in the day!i have found a good use of flash in a commercial myspace profile for the rap group g-unit. it is a menu at the top of the page, made in flash, to redirect you to the G-unit artists profile you are looking for. the page can be found at http://www.myspace.com/gunit

peace
x