Aim: Working in groups of three or less, or
individually, students will conduct a major investigation into some aspect
of socio-technical system design or evaluation. This assignment can build upon
assignment 3 or be entirely separate. You can either:
1. Evaluate fully three STS products: Gather unbiased data from 36 subjects, analyse it and draw valid conclusions.
2. Build and pilot evaluate an STS feature or function: Use your knowledge to justify, design and build some STS feature and then pilot test it on 6 subjects.
Details:
The details vary whether evaluate or build is chosen. However in both cases the report must follow the academic format and use APA referencing, i.e. the following headings are required:
1) Introduction: state in your own words what the report is about. Define your STS type. Show that it is important (give facts). State its business needs, user needs and/or social needs. Identify who this will interest. Give any background or history. 10%
2) Theory/Literature review: state in your own words what others have done, and how your evaluation/design derives or differs from them. Expect about 10 academic references. 20%
a) For the evaluation, define the general system performance construct(s) you will use, and explain why you use them. Follow previous work or theory. Use the literature to suggest factors you expect to influence STS performance. Conclude with the main criteria you intend to evaluate the systems by. Conclude with a diagram or table.
b) For the build and pilot evaluate, define the human or social need or process that you are addressing. Review what others have done or currently do (use screenshots as necessary). Outline reasons why you believe this can be improved. Justify the design decisions of the next section (gaps or potential gains).
3) Method/Design: This section describes what you actually did. The interaction should be via email. 20%
a) For the evaluation, describe your evaluation method (the evaluation tasks, the random design of subjects to tasks, how the variables were measured, the subjects, and the procedure (what you did step by step)). You will need 4 matched typical (not trivial) tasks for 3 systems, i.e. 12 tasks. The evaluation needs 1-3 demographic questions, e.g. user experience, plus 3-6 system performance questions. One question should be open ended to invite comment. As subjects first use class members then associates.
b) For the build and pilot evaluate, describe your system design, including interface design (with screenshots), data structures (table names, key and properties), data relationships (ERD), data flows (DFD), and program structure (subroutine names, calls and parameters passed). The pilot evaluate is done with two systems: A with the function and B without the function. Randomly allocate three subjects first try A (with) and then try B (without), and three to first try B (without) and then try A (with). Give subjects simple tasks, and after each trial ask them to answer a rating question(s). Finally, ask them to rate their preference of A vs B on a 7 point scale. One question should be open ended to invite comment.
4) Results and analysis: Present and analyse your results 20%:
a) Raw
data table (in Annex): Show
measured variables as columns, and cases or subjects as rows.
b) Descriptive statistics from your raw data. Calculate means (variances) for a given N. Show in table format.
c) Inferential statistics from your raw data calculate statistics that show if there is a real difference or just chance. The evaluation will need to use ANOVA. The build is only comparing two choices, so can use a t-test.
d) Qualitative analysis. Analyse responses to the open ended question for themes.
e) Conclusions. Based on your results, present a list of conclusions.
5) Discussion. Based on our conclusions, discuss what it all means, especially with regard to your introduction problem or needs. Suggest for the future. How is this useful to others? 10%
6) Annexes: These must be attached to the main document AS ANNEXES (do not include in the main document): 10%
a) For the evaluation:
i) Task scripts: As presented to subjects.
ii) Email request. Exactly the email as sent to subjects.
iii) Question items. The questions exactly as asked, including response scales.
iv) Subject response format. The format that subjects had to fill in to respond to the evaluation. Must ask subjects to be honest in their responses.
i) Raw data table. See earlier. Also include a full list of all subject comments made.
b) For the build and pilot:
i) Program Code: Copy of all code, program structure
ii) Data Structures: Table structures (field name and definition), ERD, DFD
iii) Screenshots: Of the interface
iv) Email request. Exactly the email as sent to subjects.
v) Question. The question exactly as asked, including response scales, and comparison question.
vi) Raw data table. See earlier. Also include a full list of all subject comments made.
7) Presentation: Summarize and present your project in class, giving time for comments from the rest of the class. Print a copy of your slides, 6/page, for the instructor. 10%
8) Participation. Part of this project is participating as subjects in others research. You will learn about other types of systems. When being a subject the number one criterion is to be honest, not just reply what the experimenter may want (which is bias). So be a good subject and you will learn things.
Deliverables:
1. Submit printed report with Annexes in class on the due date. Late arrival automatically -5% + 10% perday.
2. Also send as an email attachment to bwhitworth@acm.org WITH 729 ASG4 IN THE TITLE with copies to all participants by the due date