Measurement & Evaluation of HCC Systems
HCC 4400/6400
This is the syllabus website for Clemson University Spring 2022 course HCC 4400 and 6400: Measurement & Evaluation of HCC Systems.
Meeting information:
Credit hours: 3
Location: Edwards Hall 304 (online option provided), pending potential temporary modality changes
Day and time: Monday & Wednesday, 2:30 – 3:45 pm
Instructor information:
Prof. Bart Knijnenburg
Email: bartk@clemson.edu
Office hours: Wednesday, 3:45 – 4:45 pm, upon request
Important: The information below may change!
Changes will be announced in class and through email.
Course description
This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. The basic idea of this class is explained in this TEDx-talk.
What are we going to do?
Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 3 parts:
- Part 1 (week 1-3): The practice of experimental evaluation (stuff you may have learned in Research Methods)
- Part 2 (week 4-7): Basic statistical methods: Correlation, Regression, ANOVA, and T-test (stuff you have forgotten from undergraduate statistics)
- Part 3 (week 8-15): Non-linear and multilevel statistics (stats for “messy” HCC variables)
Course modality and masks (due to COVID-19)
While the university has not yet made any announcements about updated regulations regarding the surge of the Omicron COVID-19 variant, the most recent figures make me very worried for my own safety, the safety of my family, and your safety (whom I also consider family). As such, I reserve the right to make temporary changes in modality in response to the COVID-19 pandemic (on top of potential other changes in response to forthcoming COVID-related university guidelines).
First two weeks: Given the strong surge in COVID-19 cases that is currently happening, I will conduct the first two lectures fully online, upon which I will re-evaluate plans to continue in a in-person format.
Subsequent format: Once I deem it safe enough to do so, I will start teaching the course from the designated room on campus (Edwards Hall 304). To give everyone the flexibility to be as safe as possible, I will have an online meeting room open during my class for those who feel under the weather, are in quarantine, or are in any other way concerned about their health and safety. Since some people in class (including myself) have unvaccinated children, I strongly prefer that you are wear a KN95 mask (not just a cloth mask) and are fully vaccinated if you join the class in person. The adoption of basic public health measures is entirely consistent with the exemplary role we embody as academics. This includes:
- Getting the vaccine if you have the opportunity
- Wearing a mask when you are indoors (and outdoors if distancing is impossible)
- Respecting each others’ (extended) personal space
Course materials: This course uses the following resources:
- Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
- Chapters 4 and 5 of MacKenzie, I. S. “Human-Computer Interaction: An Empirical Research Perspective”: available for free via our library proxy.
- Almost all chapters of Field, A. et al. “Discovering Statistics Using R”, 1st ed.: for sale or rent on Amazon.
Students are strongly recommended to buy the Field book, since it is an invaluable reference guide for future research projects.
Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. For sample size calculations we will use G*Power (also free).
Office hours: Office hours will be directly after class on Wednesday (3:45-4:45 pm). If you want to meet outside of office hours, please send me an email with a few times you are available and the topic you want to talk about.
Slides: Presentation slides are linked in the course schedule below (topics listed in orange are clickable and link to the slides).
Assignments: There will be 4 assignments for this class. There will be some “insight” questions and some data analysis questions. Insight questions usually require a short (1-2 sentence) answer. Data analysis questions should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words.
Assignments are each worth 10% of your grade. You are allowed to discuss the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy). If you collaborate with others, please add a collaboration statement to your assignment (a simple statement saying “I collaborated with [name(s)]” is sufficient).
Midterms and final exam: The midterms and the final are each worth 15% of your grade. They will be very similar to the assignments, but they are timed, and you are not allowed to collaborate.
Graduate/undergraduate differentiation: The assignments and midterms as listed in the syllabus are for the graduate course (HCC 6400). Undergraduate students are exempt from certain assignment/exam questions. Exempt questions will be announced throughout the semester.
Prerequisites: This course has no prerequisites, but you will get the most out of it if you have a basic understanding of human-centered computing and quantitative research methods.
Grading
- Assignments: 40% (10% each)
- Midterms and final: 60% (15% each)
In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm.
Graduate course grades (HCC 6400): In my default graduate grading scheme, 85+ is an A, 80+ is an A-, 75+ is a B+, 70+ is a B, 65+ is a B-, 60+ is a C+, 55+ is a C, 50+ is a C-, 45+ is a D, and less than 45 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).
Undergraduate course grades (HCC 4400): In my default undergraduate grading scheme, 80+ is an A, 70+ is a B, 60+ is a C, 40+ is a D, and less than 40 is an F. Undergraduate grades may be curved as well; this is done separately from the graduate grades.
Cheat sheets
For the homeworks, students should learn to apply the methods using the book and the lecture slides. For the midterms and the final, I will provide a cheat sheet for each method. Cheat sheets will become available after each homework.
I have created the following cheat sheets for your convenience:
- Correlation
- Linear regression
- Independent t-test
- Dependent t-test
- ANOVA
- Factorial ANOVA
- Logistic regression (and Poisson and ordered logistic regression)
- Chi-square test
- Repeated and mixed ANOVA
- Linear mixed-effects models
- Generalized linear mixed-effects models
Course schedule
For your convenience, you can add the course schedule to your calendar (ICAL or HTML).
| Week | Dates | Topic and contents | Work |
| 1.2 | Wednesday Jan 12 | Overview and welcome (ONLINE ONLY) (video) | Read BEFORE class: Handbook chapter, Chapter 4 of MacKenzie |
| 2.1 | Monday Jan 17 | MLKjr day (no class) | |
| 2.2 | Wednesday Jan 19 | Introduction to HCI evaluation (ONLINE ONLY) (video) | Read before class: Chapter 5 of MacKenzie |
| 3.1 | Monday Jan 24 | Read before class: Chapters 1-4 of Field | |
| 3.2 | Wednesday Jan 26 | Dealing with data - Part 2 (video) | |
| 4.1 | Monday Jan 31 | Read before class: Chapter 5 of Field | |
| 4.2 | Wednesday Feb 2 | Assumptions, sample size - Part 2 (video) | |
| 5.1 | Monday Feb 7 | Read before class: Chapter 6 of Field | |
| 5.2 | Wednesday Feb 9 | No class | |
| 6.1 | Monday Feb 14 | Read before class: Chapter 7 of Field | |
| 6.2 | Wednesday Feb 16 | Regression - Part 2 (video) | Homework 1 available |
| 7.1 | Monday Feb 21 | Read before class: Chapter 9 of Field | |
| 7.2 | Wednesday Feb 23 | Due before class: Homework 1 Read before class: Chapter 10 of Field | |
| 8.1 | Monday Feb 28 | ANOVA - part 2 (video) | |
| 8.2 | Wednesday Mar 2 | Review session | Midterm released on Thursday morning, due Saturday evening |
| 9.1 | Monday Mar 7 | Read before class: Chapter 12 of Field Homework 2 available | |
| 9.2 | Wednesday Mar 9 | Read before class: Chapter 8 of Field (except 8.9) | |
| 10.1 | Monday Mar 14 | Logistic regression - Part 2 (video) | |
| 10.2 | Wednesday Mar 16 | Due before class: Homework 2 Read before class: Chapter 18 of Field (except 18.7-18.12) Homework 3 available | |
| Mar 21+23 | No class - Spring Break | ||
| 11.1 | Monday Mar 28 | Review session | |
| 11.2 | Wednesday Mar 30 | Midterm 2 - Everything up to factorial ANOVA | |
| 12.1 | Monday Apr 4 | Read before class: Chapter 13 of Field | |
| 12.2 | Wednesday Apr 6 | Read before class: Chapter 14 of Field | |
| 13.1 | Monday Apr 11 | Read before class: Chapter 19 of Field Due before class: Homework 3 | |
| 13.2 | Wednesday Apr 13 | Multilevel linear models - Part 2 (video) | Homework 4 available |
| 14.1 | Monday Apr 18 | Generalized multilevel linear models (video) | |
| 14.2 | Wednesday Apr 20 | Review session | |
| 15.1 | Monday Apr 25 | Discuss midterm 3 answers | Due before class: Homework 4 |
| 15.2 | Wednesday Apr 27 | Review + preview of Advanced Measurement & Evaluation (video) | |
| exam | Thursday May 5 | Final exam, 3-5:30pm |
Attending class, etc.
Things discussed in class are part of the course materials, and although the slides and recordings will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please have R and RStudio installed on your computer. For the midterms and the final you will also work on your laptop, so make sure it is properly working on exam days!
You will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!
Academic integrity
Please refer to the following official statement on academic integrity:
As members of the Clemson University community, we are supposed to have a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we are supposed to recognize that academic dishonesty detracts from the value of a Clemson degree. Therefore, we should not tolerate lying, cheating, or stealing in any form.
Practically speaking: Do not cheat. Plagiarism will not be tolerated, and be dealt with through official university channels, see: http://www.clemson.edu/academics/integrity/plagiarism.html.
Cheat sheets are created for your convenience to help you with the midterms and the exam. Please do not use cheat sheets (e.g., from last year) when making the homework. Using last years‘ cheat sheets on the homeworks is considered cheating, and will be dealth with accordingly.
Disability access
p>Students with disabilities requesting accommodations should contact the Office of Student Disability Services in Suite 239, Academic Success Center building 864-656-6848, to discuss specific needs within the first month of classes. Students should present a Faculty Accommodation Letter from Student Disability Services when they meet with instructors. Accomodations are not retroactive and new Faculty Accommodation Letters must be presented each semester.Title IX (Sexual Harassment) statement
Clemson University is committed to a policy of equal opportunity for all persons and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender, pregnancy, national origin, age, disability, veteran‘s status, genetic information or protected activity (e.g., opposition to prohibited discrimination or participation in any complaint process, etc.) in employment, educational programs and activities, admissions and financial aid. This includes a prohibition against sexual harassment and sexual violence as mandated by Title IX of the Education Amendments of 1972. This policy is located at http://www.clemson.edu/campus-life/campus-services/access/title-ix/. Mr. Jerry Knighton is the Clemson University Title IX Coordinator. He also is the Director of Access and Equity. His office is located at 111 Holzendorff Hall, 864.656.3181 (voice) or 864.565.0899 (TDD).