Advanced Measurement & Evaluation of HCC Systems
HCC 8410
This is the syllabus website for Clemson University Fall 2022 course HCC 8410: Advanced Measurement & Evaluation of HCC Systems.
Meeting information:
Credit hours: 3
Room: Poole A103
Day and time: Monday & Wednesday, 2:30 – 3:45 pm
Instructor information:
Prof. Bart Knijnenburg
Email: bartk@clemson.edu
Office location: McAdams Hall 205
Office hours: by appointment
Important: The information below may change!
Changes will be announced in class and through email.
Course description
This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. This course builds upon Measurement & Evaluation of HCC Systems.
This advanced course will pay special attention to two very important state-of-the-art methods for HCC research: The measurement and evaluation of subjective valuations of users‘ usage experience using multi-item psychometric instruments and exploratory and confirmatory factor analysis (EFA and CFA), and the evaluation of structured models of hypotheses using structural equation modeling (SEM). We will also cover advanced methods such as Rasch modeling and factor mixture analysis.
Most existing HCC research tests hypotheses one by one, and uses behavioral proxies or single-item measurements to test users‘ subjective valuations. Evaluations using CFA and SEM are more accurate, more comprehensive, and easier to report.
What are we going to do?
Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 4 parts:
- Part 1 (weeks 1-4): Path models (how to do multiple mediation analyses at once)
- Part 2 (week 5): Psychometrics (how to measure subjective valuations with questionnaires)
- Part 3 (weeks 6-8): Latent variable models (how to create and evaluate measurement scales of subjective valuations using EFA and CFA)
- Part 4 (weeks 8-10): Structural models (how to evaluate structured hypotheses using SEM and multi-level SEM)
- Part 5 (weeks 11-13): Advanced SEM methods (measurement invariance, interaction effects, cross-lagged panel models, LCA and FMA)
- Part 6 (week 15): Rasch modeling (advanced measurement scale analysis)
Course materials: This course uses the following resources:
- Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
- Chapters 6-14 of Kline, R. B. “Principles and Practice of Structural Equation Modeling”, 4th ed.: for sale on Amazon. In a pinch, you can also use the 3rd edition, but the chapters are different.
- Chapters 1-5 of DeVellis R. F. “Scale Development: Theory and Applications”, 2nd ed.: for sale on Amazon (link 1, link 2). The (more expensive) 3rd edition is not required.
- Chapters 5-6 of Loehlin, J. C. “Latent Variable Models”, 4th ed.: available here.
- Chapter 13 of Tabachnick B. G. and Fidell, L. S. “Using Multivariate Statistics”, 5th ed.: available here.
- Chapters 3-4 of Bond, T. G. and Fox, C. M. “Applying the Rasch Model” 2nd ed.: available for online reading here.
- Selected slides from lectures by Muthen, B. and Muthen, L. Videos and Handouts for Mplus Short Courses.
Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. I may show you some CFA and SEM models in MPlus (which is a bit more powerful than R, but not free).
Office hours: Office hours will be by appointment. If you want to attend office hours, please let me know at the end of the class, or send me an email.
Slides: Presentation slides are already linked in the course schedule below (topics listed in orange are clickable and link to the slides).
Assignments: There will be 4 assignments for this class. They should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words (make sure to always explain your answer!).
Assignments are each worth 10% of your grade. You are allowed to discuss the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy). Please add a collaboration statement to your write-up so that I know with whom your collaborated.
Midterms and final: The midterms and final are each worth 15% of your grade, and will be very similar to an assignment, only you cannot collaborate. The midterms will be take-home: they will be made available on Thursday morning and due on Saturday afternoon. The final will be an in-class assignment, to be completed in the time span of the final exam (2.5 hours).
Prerequisites: You are required to have taken HCC6400: Measurement & Evaluation of HCC Systems or a similar course on statistical analysis. If you are not sure whether you meet the prerequisites, please email me.
Grading
- Assignments: 40% (10% each)
- Midterms and final : 60% (15% each)
In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm. In my default grading scheme, 85+ is an A, 80+ is an A-, 75+ is a B+, 70+ is a B, 65+ is a B-, 60+ is a C+, 55+ is a C, 50+ is a C-, 45+ is a D, and less than 45 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).
Cheat sheets
The following cheat sheets have been created for your convenience, to use in the midterms and final exam. They outline the steps to conduct a typical CFA and SEM in R. Please do not “blindly” follow the cheat sheets; the questions on the final may require more advanced methods than those presented in the cheat sheets!
Course schedule
For your convenience, you can add the course schedule to your calendar (ICAL or HTML).
| Week | Dates | Topic and contents | Work |
| 1.2 | Wednesday Aug 24 | Online (via Zoom): Overview and welcome (video) | (Re-)read before class: Knijnenburg and Willemsen |
| 2.1 | Monday Aug 29 | Kline chapters 2-4 Muthen topic 1 slides 15-25 | |
| 2.2 | Wednesday Aug 31 | ||
| 3.1 | Monday Sep 5 | Regression models (recap - part III, if needed) | |
| 3.2 | Wednesday Sep 7 | Kline chapter 6-7 Muthen topic 1 slides 27-38 | |
| 4.1 | Monday Sep 12 | Kline chapter 11-12 | |
| 4.2 | Wednesday Sep 14 | Homework 1 (Path models) available (use the twp dataset) | |
| 5.1 | Monday Sep 19 | DeVellis chapters 1-4 | |
| 5.2 | Wednesday Sep 21 | DeVellis Chapter 5 Homework 2 (Psychometrics) available Due Friday 6pm: Homework 1 (Path models) | |
| 6.1 | Monday Sep 26 | Kline chapter 9 Muthen topic 1 slides 40-58 | |
| 6.2 | Wednesday Sep 28 | Review of path models | Midterm 1 (Path models) available Thursday morning, due Saturday evening |
| 7.1 | Monday Oct 3 | Kline chapter 13 Muthen topic 1 slides 107-132 and 147-155 | |
| 7.2 | Wednesday Oct 5 | Loehlin chapters 5-6 Muthen topic 1 slides 59-105 | |
| 8.1 | Monday Oct 10 | EFA - Part II (video) | Tabachnick chapter 13 Muthen topic 1 topic 2 slides 113-140 Homework 3 (EFA and CFA) available Due before class: Homework 2 (Psychometrics) |
| 8.2 | Wednesday Oct 12 | Kline chapter 10 Muthen topic 1 slides 173-200 and topic 2 slides 142-164 | |
| 9.1 | Monday Oct 17 | Kline chapter 14 Muthen topic 1 slides 228-242 | |
| 9.2 | Wednesday Oct 19 | Muthen topic 7 slides 15-28 Due before class: Homework 3 | |
| 10.1 | Monday Oct 24 | Muthen topic 7 slides 147-157 | |
| 10.2 | Wednesday Oct 26 | Review of EFA and CFA | Midterm 2 (EFA and CFA) available Thursday morning, due Saturday evening |
| 11.1 | Monday Oct 31 | Kline chapter 16 | |
| 11.2 | Wednesday Nov 2 | Kline chapter 17 Muthen topic 1 slides 201-226 | |
| 12.1 | Monday Nov 7 | No class - Fall break | |
| 12.2 | Wednesday Nov 9 | Guest lecture: cross-lagged panel models (video) | Readings TBA Due before class: Homework 4 |
| 13.1 | Monday Nov 14 | Latent Categorical Analysis (LCA) and Factor Mixture Analysis (FMA) (video) | Muthen topic 5 slides 66-118 and 153-179 |
| 13.2 | Wednesday Nov 16 | Review of SEM and multi-level SEM | Kline chapter 18 Midterm 3 (SEM and Multi-level SEM) available Thursday morning, due Saturday evening |
| 14.1 | Monday Nov 21 | Make-up class slot (if needed) | |
| 14.2 | Wednesday Nov 23 | No class - Thanksgiving | |
| 15.2 | Monday Nov 28 | Bond chapters 3-4 | |
| 15.2 | Wednesday Nov 30 | ||
| 16.1 | Monday Dec 5 | ||
| 16.2 | Wednesday Dec 7 | Make-up class slot (if needed) | |
| exam | Thursday Dec 15 | Final exam, 3-5:30pm | |
Attending class, etc.
Things discussed in class are part of the course materials, and although the slides will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please bring your laptop with R and RStudio installed.
You will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!
Academic integrity
Please refer to the following official statement on academic integrity:
As members of the Clemson University community, we are supposed to have a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we are supposed to recognize that academic dishonesty detracts from the value of a Clemson degree.
Practically speaking: Do not cheat (e.g.: do not collaborate on the midterms and/or final). Plagiarism will not be tolerated, and be dealt with through official university channels, see: http://www.clemson.edu/academics/integrity/plagiarism.html.
Disability access
Students with disabilities requesting accommodations should contact the Office of Student Disability Services in Suite 239, Academic Success Center building and/or call 864-656-6848 to discuss specific needs within the first month of classes. Students should present a Faculty Accommodation Letter from Student Disability Services when they meet with instructors. Accomodations are not retroactive and new Faculty Accommodation Letters must be presented each semester.
Title IX (Sexual Harassment) statement
Clemson University is committed to a policy of equal opportunity for all persons and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender, pregnancy, national origin, age, disability, veteran‘s status, genetic information or protected activity (e.g., opposition to prohibited discrimination or participation in any complaint process, etc.) in employment, educational programs and activities, admissions and financial aid. This includes a prohibition against sexual harassment and sexual violence as mandated by Title IX of the Education Amendments of 1972. This policy is located at http://www.clemson.edu/campus-life/campus-services/access/title-ix/. Alesia Smith is the Clemson University Title IX Coordinator. She is also the the Executive Director of Equity Compliance. Her office is located at 223 Brackett Hall, phone: 864-656-3181, email: alesias@clemson.edu.