evaluating systems, methods and there pro's and con's!

Blog 07.12.06

Today’s lecture was about the different methods of evaluating systems.

Observation and monitoring

· Watching the user interact with the system

· The user is either performing special tasks or performing normal work

· The users performance is recorded in some way

Direct Observation Limitations

· Can only record a proportion of the events

· We can only record certain types of events

· We only have one attempt at the observation

· We may simply miss events

· We might make recording errors

· User performance may be influenced by observation

Benefits

· Cheap

· We can concentrate on the information you are interested in

· Useful early in project for an informal feedback

· We can use a checklist to make data collection quicker and simpler

Video recorder limitations

· Users can be self conscious

· Can create more distance between users and evaluators

· Can be difficult to synchronise recording with eachother

· Analysis of video is time consuming 1 hour of tape can take over 5 hours to analyse

Benefits

· Can be synchronised with automatic logging

· Creates more distance between the users and evaluators

· Multiple cameras can capture different aspects

· Tools available for simplifying analysis

· We can revisit the data

Verbal protocols

· Where users talk through what they are doing

· Thinking allowed

Limitations

· Thinking aloud is difficult and can interfere with performance

· Thinking allowed is of dubious value

· User may need promoting which can disrupt them

Post event protocol

· Users view video of their actions and provides a commentary on it

· Good for situations in which the users has to concentrate when using the interface

· Doubts as to the abilities of users to recall their actions

· Users may rationalise their actions with benefit of hindsight

Software logging

· Does not require the researcher to be present

· Analysis of data can be automated to some extent

· Can be synchronised with video and audio recording

· Should make users aware they are being logged

· Expensive

· Large quantities of data

Users opinions

· Users interact with the system and then there subjective opinions are elicited

· Users attitudes are important and will effect the acceptance of a system

Structured interviews

· Predetermined questions

· Asked in a strict order

· No exploration of individual attitudes

· Good for comparing the responses of different subjects

· Good for doing statistical analysis

Flexible interviews

· Have some set topics

· No set sequence

· Interviewer can follow up on interviewees comments

· Relies on the skill of the interviewer to get the interview and make the person feel easy

· Interviewee may be reluctant to criticise the system

Structured verus flexable

· Structured are easier

· Easier to analyse

· Less scope for picking up relevant issues

· Both benefit from pilot studies

Checklist for semi structured interviews

· Why do you do this?-to get the users goal

· How do you do it?

· Why didn’t you do it this way?

Questionnaires and surveys

· Preparation of questions

· Can be used to reach a large number of people

· Closed questions(select from a group of options)

· Open questions(responds with personal answer)

We were also shown some examples of good and bad questionnaires

Analysis

· Mean

· Standard deviation

· More advanced statistics

Experimentation

· Users asked to do tasks in a controlled environment

· User performance is analysed

· Purpose of the experiment

· a hypothesis that can be tested

· what statistical methods will be used? which are best?

Results

· size of the effect

· alternative interpretations

· consistency

· generalisation

interpretive evaluation

· users perform natural tasks in the working environment

· often there is user participation in collecting analysing or interpreting the data

Context

· behaviour is only useful in context

· work context

· time context

· motivational context

· social context

predictive evaluation

· use a team of experts to review the system to predict the system

· present there findings as a report typically with a prescriptive element

limitations

· experts tend to have strong opinions, so having a few makes it more equal

· finding the right experts can be hard

· can require a alot of detailed knowledge

· experts cannot capture the behaviour of all the users

those were the notes i took based on the slides and comments of the lecturer(whos name i always forget.lol)in todays lecture!well thats it from me

be the music, not the scene

x

Evaluation

Today’s lecture was about evaluation. Evaluation is about gathering data, about the usability or design of a product. We are interested in a particular set of users, related to a particular set of tasks. We are not interested in every user. And we need to evaluate which situation it will be used in and in what location?

Why are we evaluating?

· to understand the real world

· to compare designs

· to engineer the product towards a target

· to check conformance to a standard

· to ask different questions at different points of the process.

Real world

· what does the user actually want?

· Remember the designer is not the end user, what we want is not what they want!

· What problems do users experience? What is wrong with the current system?

· What tasks do users perform

· How do users perform tasks

· What problems do they experience

Comparing designs

· Commercial competition? What do the rivals do, what is good? What can we improve?

· Which design is best and what do we mean by best? What makes it the best? Speed, appeal?

· Answer specific design questions

Engineering towards a target

· Aiming for specific performance measures

· Improving areas of an existing design

· Being better than a competitor

Conformance to a standard

· ISO standards

· Accessibility standards

· Health and safety standards

· In-house standards

· Design guidelines

When to evaluate

· Formulative evaluation

o Evaluation as part of the design process

o Predesign proof and viability

o Initial design with user

o Interactive design

o Acceptance testing

· Predict usability of system or aspects thereof

· Check design teams understanding of user requirements

· Test out ideas quickly and informally

· Summative evaluation

o Evaluation of a essentially finished system

o Interactive design

o Acceptance testing

· Identifying user difficulties so products can be finely tuned

· Upgrading product

Factors

· Kind of information required

· Nature of system or specification under evaluation

· Stage in the life cycle being evaluated

· Whether or not statistical validity is required

· Resources available

Qualitative evaluation

· Descriptive

· Typically non-numerical

· May be subjective

· Introspection

· Direct observation

· Query via questionnaires and interviews

· Continuous evaluation through feedback and field studies

Quantitative evaluation

· Precise measurement

· Numerical values

· Bound on how to correct our statements are

· User performance

· Controlled experiments

· Statistical analysis

Measurable quantities

· Time taken to complete task

· Number of tasks completed within a given time

· Ratio between successful interactions to errors

· Time spent recovering from errors

· Number of errors

· Number of commands or features actually used by the user

· Number of features user can remember after the test

This was basically everything said in the lecture. Not long until the Christmas break!w0o0o0o0o0o0o0o0o0o!!!!!Bye for now!

useability

Today’s lecture was about usability. It was our first lecture with Daniel Cunliffe. Who will be lecturing us for the next 4 weeks. In our lecture we were introduced to the basics of usability. We are going to be focused on multimedia applications such as Macromedia suite or Adobe photoshop. Multimedia applications are used in a variety of contexts. Some are only used once such as software at trade shows and others are used multiple times, such as presentations in shops. They can be used for single users(one user using the software alone) or multiple users. You must consider what the application is used for. Eg. Informing/ teaching/entertaining. And who is going to use the software? The program will have to be different for novice users as opposed to more experienced users as they will not know what to with the software straight away and will struggle to adjust. What location are the users going to be using the software? Online test will have supervision for example and the program could be designed for the specific computers. However on a web based application you have little or no control of the location it will be used in. we can use the CUTS model which stands for Context (how will it be used?, in what situation?),User (who will use it? Are they experienced users or do they need a more simple system? What do they want from the application? What do they need for it? How do you motivate the user to use the application?), Task(what is it made to do? Does it do the same task all the time? How many times a day is it used?) and System (does it require standard equipment such as mouse and keyboard or can you use different input devices? What type of specification is required? Processing speed, how long will it take to do a required task? What about connection speed?) to realise what we need to consider when developing an application. Definition of effectiveness: “The effectiveness efficiency and satisfaction in which specified users can achieve specified goals”. Also the system should be pleasant to use and relatively straight forwards to operate. It should not require too much concentration. The system should have good functionality but also be useable and suit the environment.

Copywrite laws

This weeks lecture was on Copy write laws. It covered many of the basics of copy write law to do with software and multimedia. Copy write effects all of the music you listen to or buy, and most of the programs you will own or “aquire” ;).for example if you pay for a CD album, you are only allowed to listen to that CD privately. Although you have paid for the CD you are not allowed to make a back up of the music, copy it to your computer or even convert it to mp3 to use on your mp3 player. Also you are not allowed to lend the CD to a friend or even relative as this is technically a breach of copy write. Software is also covered by a version of copy write law. However these laws are a little more reasonable as you are allowed to make a back up of the software as long as you keep it to yourself and don’t give it to others. Some software gives you an opportunity to “try before you buy”. This is called Shareware. These programs give you limited features, a certain amount of time to try them or often reminders to purchase the product. If you choose to buy you are given a serial number which can be used to activate the full product. This gives people the chance to see if the program is for them before forking out large amounts of money for software. However these sharewares can be “cracked” to allow the user to use the full product for free. This is done by modifying some of the files or obtaining a serial number from a third party, this is of course a breach of copy write and illegal. There is also open source software which is free from the off. You can also modify this software to make it work how you want it to. I think this will be the future of software as the gap between open source and company made software is closing.

We also were shown the music video for Nizlopi’s shit song “the jcb song”. The video was made in flash and was quite impressive. Its just a shame it was for one of the worst written, most commercially driven song I have ever heard. But personal opinions aside, the video was a work of art.

Computing for all

Today’s lecture was about disabled computer users and how you should design projects to incorporate there needs. It is actually law to accommodate less able users. Users have all different needs. Some have vision difficulties which means you need to choose sensible colour schemes. Others may be deaf so any audio files will need to have subtitles or a description of what they contain as the information would otherwise be useless. Others cannot see at all and use software such as dragon dictate or JAWS to speak text to them. For this reason images need to be captioned and videos annotated. These measures as well as others are all steps towards the internet becoming accessible to all users. You can have your website tested to see how user friendly it is, and if you pass you can put a logo that states it is for all users on your page. We also watched a video which shows how users with different disabilities use computers. Some have a brail pad, which translates text, line by line. Other users who do not have use of there arms move there heads onto sensor pads to interact with the computer. It is amazing what can be done to make technology available to all.

Crap sites of the web unite!

Todays lecture was on Human computer Interfaces(HCI's).
Hci's are designed to aide the user and to make the operation of a computer more simple and user friendly. HCI's range from the Graphical user interface of an operating system, such as Windows Vista or Mac OSX. Features of these HCI's are things like the windows, tabs, drop down and pop up menus, or the much dreaded cascade menu(which single handedly pushed me to uninstall windows media player 10!). Icons are also used in these operating systems to help the user identify files, folders or programs more quickly. These HCI's replaced the DOS operating system, which was command based, instead of the array of colours, sounds and graphics that make up more modern operating systems. Today DOS only really exists as an emulated platform in windows ME,XP and XPpro. Although not user friendly it is very useful for more advanced users and often alot faster than going through all the windows,directories and menus of a HCI. HCI's are also used in websites. Web designers utilise lots of methods to add to the user experience and make there sites more appealing and useable. for instance text can be made bold or underlined to make it stand out, flashing text can also be used as well as using text that stands out from the background colour. Text can also be presented in different ways to add to its apperence. You can use CAPITAL LETTERS to make the words stand out from the page. or you can allign the text to the right with a jagged left edge to make it stand out from other infomation. Also the use of colours can be used to make the page more appealing although you should tread carefully in this area. You cannot choose a colour scheme to please everyone but you should hold certain things in mind when you choose the pallette of your design. As a rule you should limit yourself to 4 colours so the page will not become overdone. You should also bare in mind that not all monitors can display as many colours as others.some for example 256bit monocrome monitors can only display 256 colours and due to this many will not display correctly. for this reason it is advised that projects are designed in monocrome first. It is also important to concider that colours represent different things in different cultures, for example in the western world blue is a calming colour and is quite regal. But in japan it is concidered to represent "dirty work" or generally frowned on activity. As you can see user concideration is paramount in web design. Disabled users with disabilitys such as colourblindness cannot see certain colours and others can only see shades of grey. Even the healthy eye has problems with certain colour schemes. it is very difficult for the human eye to destinguish between the primary colours blue and red quickly.
In the end of the lecture we were shown examples of good and bad websites. radioqualia.va.com is a good example of a terrable website. the site had no real navigation and was just barely viewable icon leading to bland illegable text! it was soo bad it was frustrating to look at. Also arngren.net was an example of a totally overcrowded website! it was as bad as milliondollarhomepage.com, but atleast that guy made a cool million for barly any effort! i learned the web is great if sites are done well, however a poorly designed site is either annoying and frustrating or just simply a laughing stock!

peace
x

stick death!

this weeks lecture was about multimedia design techniques. in the lecture we learned about various techniques you can use to create a multimedia project. we were told about the need to consider the user of the project and not to please your own tastes. we were made aware that certain colour schemes would be hard for certain people to see and were shown first hand how frustrating it is if a flash presentation has over the top navigation. if the navigation is too complicated most users will "switch off" or just give up on the page all together. however we were also showen a very creative flash movie based in the flash2004 program. i stick man is drawn on the screen and is then a target of attacks by various features of the program!the stick man defends himself and it turns into an out and out war with the cursor towards the end of the movie. this video was hilarious, even better than wedding crashers. it was soo impressive because it was a simple idea executed soo well.i was glad that this was an interesting lecture because i was fit to drop after the 5 hours of lectures i had in the day!i have found a good use of flash in a commercial myspace profile for the rap group g-unit. it is a menu at the top of the page, made in flash, to redirect you to the G-unit artists profile you are looking for. the page can be found at http://www.myspace.com/gunit

peace
x

the wedding crashers!!lol

i saw the best bit of flash ive ever seen in yesterdays lecture! it was from the film the wedding crasher and you could upload a .jpg of your face and put it into the advert!the example we had was of a picture of some random guys face overlayed onto Vince Vaughns character! the results were hilarious, its definatly one to show your friends! the main lecture was mainly about multimedia file formats. although basic i did learn about a few new file formats and was refreshed on the others. it was interesting to see which file formats were popular and which failed. although personally i think the .avi format is possibly more widespread than quicktime and definatly more useful as it can easily be transfered into mpeg files which are compatable with many media players and codecs or ever into .3gp files which are up and coming and are useful for devices such as mobile phones and the sonys multimedia device the PSP(playstation portable).i also learnt about the different sizes of compression for different files.for instance a .jpg file is far smaller than bitmap images which are huge.this is where quicktime video is good because although quality isnt always to the standard of avi/mpg the file compression is greater and so the reduced file size is preferable for websites.

this blog entry is about done for now as my girlfriend looks extremly bored as i sit here typing! so have a good weekend or whatever!:D

x

lecture 2!

I had my second lecture in multimedia today and it was as usual one of the more interesting modules in my network management and security course. there seems to be some really exciting possibilities with Flash and im looking forward to developing my skills and hopefully gain enough skill and experience to make some amazing flash presentations!todays lecture built on what we had learned last week, and advanced into "tweening". this is a simple animation technique where text and other objects can be manipulated into an animated video. this looks like a very intresting development and will definatly come in useful, however we also learned in the lecture that too many tweened objects on a page can be distracting as it draws the eye away from more bland objects such as text which will be the main body of a page.
Also i checked out the "aqualtis" flash move on the website we were trying to view in the lecture, and i was stunned!this is possibly the best example of flash i have ever seen. the whole page is nothing short of genius!you can find this page at http://www.dariopiana.com/home.html. i would recommend this site to anyone simply because of the skill involved in its production!i will check out the rest of the site when im back in uni tomorrow as the uni connection is far quicker than mine!
Thankfully i discovered that my copy of macromedia suite 8 is compatable with the mx 2004 software we use in glamorgan. this was useful as i have struggled to locate a copy of the now outdated mx edition through my usual channels!you can simple save the project as either a flash 8 file or the more compatable mx '04 file!this is very useful as i can not only get on with my work at home, but i can also use the software on my laptop which i take to uni! all this sounds good but it doesnt leave many excuses as to why i didnt do my project when i should have!!!!(only joking, im a good student really!)

well its bye for now as i prepare for another day at uni and hopefully a good weekend!

stay smooth!
tom
xXx

ENROLEMENT NUMBER!!!!!

ENROLEMENT NUMBER:06030289

thanks :D

MATHS!!!!!!!!!

well i had my first maths lecture today and it was very interesting! i struggled at first but was desperate to keep up, so concentration payed off bigtime!i actually left knowing what the lecturer was on about, first time ive ever really understood a maths lesson! I also had a lecture involving Databases and all that jazz. Most of my peers in this module have done it all before at a-level so i have to look sharp to keep up, but it appears im not alone in this which is a bit more encoraging!
I also had a programming lecture today which was very interesting. I am looking forward to having a go at programming next week. I have no prior experience at programming but i am eager to learn so i am quite optemistic about that module. The computer systems module particulary appealed to me as i am from more of a practically based computing background as opposed to a more theory orientated background. Athought the lecture started off very basically, it began to get a little more complexed. i really enjoyed this lecture and think this will be one of my favorite subjects, as i am very interested in the way computers work and the components that make up computer systems.
As well as this i also had my first internet application development lecture. I thoroughly enjoyed this lecture as it was very interesting to see what can be done with the flash software, and i look forward to learning this. I was also shown some amazing examples of flash in action and this really appealed to me because i like examples to help me learn!
As you can see it was a long day with 6, hour long lectures. But the laptop battery held out(shame the same insnt true of the cars!lol), and i just had enough free time to check out the resteraunt at the students union. So all in all it was a good day and an excellent insight into the modules i will be studying.

bye for now

x


ENROLEMENT NUMBER:06030289

OHHHH a day off a week!(for studies obviously)

My first wednesday off!was fun! just bought a printer! told u i would.lol! A day of xbox live is not a day wasted at all! jus watchin american pie now n blogging while im at it!N the bit where he gets with the girl in the band has made me miss my girlfiend! so on that note im off 2 talk to her!

peace out!

x

Day 2 and ive managed to find the laptop study room(J3)! i have been downloading files with download speeds up 2 40mbps in the mornings!:D i have also discovered the wireless SIAN network, and with a bit of persistance and a few frayed nerves i have sucsessfully connected! nothing like coffee and myspace to pass the 3hour gap between lectures!

they wernt joking when they said parking would be a bugger either, ive found that if ur much later than 9.15 youve had it for parking this side of trefforist!

But that problem aside im really starting to get into the whole uni thing, my only other issue is really the whole printing thing. Its a impressive system but its a pain having to go one end of campus for a token(which borders intrestingly between good value and a rip off), then having to go to the bottom of J-block to get it on the system, then finding a machine to print from, before the final pilgrimage up to J107 to finally get your print out!
I may ajust to the system but at the moment it does seem a little long winded, maybe a printer is a good way of burning some of that student loan!

well im off for now hope to blog everyday if i find the time! check out tomorrows entry for more of one guy, one laptop and a lack of patience!lol. :D

Hello, this is my uni blog, it is my first! dunno what to say?!

so bye for now! +D

http://www.myspace.com/tombright this is me!!:D