Advanced Measurement & Evaluation of HCC Systems

HCC 8410

This is the syllabus website for Clemson University Fall 2022 course HCC 8410: Advanced Measurement & Evaluation of HCC Systems.

Meeting information:

Credit hours: 3

Room: Poole A103

Day and time: Monday & Wednesday, 2:30 – 3:45 pm

Instructor information:

Prof. Bart Knijnenburg


Office location: McAdams Hall 205

Office hours: by appointment

Important: The information below may change!

Changes will be announced in class and through email.

Course description

This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. This course builds upon Measurement & Evaluation of HCC Systems.

This advanced course will pay special attention to two very important state-of-the-art methods for HCC research: The measurement and evaluation of subjective valuations of users‘ usage experience using multi-item psychometric instruments and exploratory and confirmatory factor analysis (EFA and CFA), and the evaluation of structured models of hypotheses using structural equation modeling (SEM). We will also cover advanced methods such as Rasch modeling and factor mixture analysis.

Most existing HCC research tests hypotheses one by one, and uses behavioral proxies or single-item measurements to test users‘ subjective valuations. Evaluations using CFA and SEM are more accurate, more comprehensive, and easier to report.

What are we going to do?

Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 4 parts:

  • Part 1 (weeks 1-4): Path models (how to do multiple mediation analyses at once)
  • Part 2 (week 5): Psychometrics (how to measure subjective valuations with questionnaires)
  • Part 3 (weeks 6-8): Latent variable models (how to create and evaluate measurement scales of subjective valuations using EFA and CFA)
  • Part 4 (weeks 8-10): Structural models (how to evaluate structured hypotheses using SEM and multi-level SEM)
  • Part 5 (weeks 11-13): Advanced SEM methods (measurement invariance, interaction effects, cross-lagged panel models, LCA and FMA)
  • Part 6 (week 15): Rasch modeling (advanced measurement scale analysis)

Course materials: This course uses the following resources:

  • Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
  • Chapters 6-14 of Kline, R. B. “Principles and Practice of Structural Equation Modeling”, 4th ed.: for sale on Amazon. In a pinch, you can also use the 3rd edition, but the chapters are different.
  • Chapters 1-5 of DeVellis R. F. “Scale Development: Theory and Applications”, 2nd ed.: for sale on Amazon (link 1, link 2). The (more expensive) 3rd edition is not required.
  • Chapters 5-6 of Loehlin, J. C. “Latent Variable Models”, 4th ed.: available here.
  • Chapter 13 of Tabachnick B. G. and Fidell, L. S. “Using Multivariate Statistics”, 5th ed.: available here.
  • Chapters 3-4 of Bond, T. G. and Fox, C. M. “Applying the Rasch Model” 2nd ed.: available for online reading here.
  • Selected slides from lectures by Muthen, B. and Muthen, L. Videos and Handouts for Mplus Short Courses.

Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. I may show you some CFA and SEM models in MPlus (which is a bit more powerful than R, but not free).

Office hours: Office hours will be by appointment. If you want to attend office hours, please let me know at the end of the class, or send me an email.

Slides: Presentation slides are already linked in the course schedule below (topics listed in orange are clickable and link to the slides).

Assignments: There will be 4 assignments for this class. They should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words (make sure to always explain your answer!).

Assignments are each worth 10% of your grade. You are allowed to discuss the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy). Please add a collaboration statement to your write-up so that I know with whom your collaborated.

Midterms and final: The midterms and final are each worth 15% of your grade, and will be very similar to an assignment, only you cannot collaborate. The midterms will be take-home: they will be made available on Thursday morning and due on Saturday afternoon. The final will be an in-class assignment, to be completed in the time span of the final exam (2.5 hours).

Prerequisites: You are required to have taken HCC6400: Measurement & Evaluation of HCC Systems or a similar course on statistical analysis. If you are not sure whether you meet the prerequisites, please email me.


  • Assignments: 40% (10% each)
  • Midterms and final : 60% (15% each)

In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm. In my default grading scheme, 85+ is an A, 80+ is an A-, 75+ is a B+, 70+ is a B, 65+ is a B-, 60+ is a C+, 55+ is a C, 50+ is a C-, 45+ is a D, and less than 45 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).

Cheat sheets

The following cheat sheets have been created for your convenience, to use in the midterms and final exam. They outline the steps to conduct a typical CFA and SEM in R. Please do not “blindly” follow the cheat sheets; the questions on the final may require more advanced methods than those presented in the cheat sheets!

Course schedule

For your convenience, you can add the course schedule to your calendar (ICAL or HTML).

WeekDatesTopic and contentsWork
1.2Wednesday Aug 24

Online (via Zoom): Overview and welcome (video)

(Re-)read before class: Knijnenburg and Willemsen

2.1Monday Aug 29

Regression models (recap - part I) (video)

Kline chapters 2-4

Muthen topic 1 slides 15-25

2.2Wednesday Aug 31

Regression models (recap - part II) (video)

3.1Monday Sep 5

Regression models (recap - part III, if needed)

3.2Wednesday Sep 7

Path models - part I (video)

Kline chapter 6-7

Muthen topic 1 slides 27-38

4.1Monday Sep 12

Path models - part II (video)


Kline chapter 11-12

4.2Wednesday Sep 14

Path models - part III (video)

Homework 1 (Path models) available (use the twp dataset)

5.1Monday Sep 19

Psychometrics - part I (video)


DeVellis chapters 1-4

5.2Wednesday Sep 21

Psychometrics - part II (video)

DeVellis Chapter 5

Homework 2 (Psychometrics) available

Due Friday 6pm: Homework 1 (Path models)

6.1Monday Sep 26

Confirmatory Factor Analysis (CFA) - part I (video)

Kline chapter 9

Muthen topic 1 slides 40-58

6.2Wednesday Sep 28

Review of path models

Midterm 1 (Path models) available Thursday morning, due Saturday evening

7.1Monday Oct 3

CFA - Part II (video)

Kline chapter 13

Muthen topic 1 slides 107-132 and 147-155

7.2Wednesday Oct 5

Exploratory Factor Analysis (EFA) - part I (video)


Loehlin chapters 5-6

Muthen topic 1 slides 59-105

8.1Monday Oct 10

EFA - Part II (video)

Tabachnick chapter 13

Muthen topic 1 topic 2 slides 113-140

Homework 3 (EFA and CFA) available


Due before class: Homework 2 (Psychometrics)

8.2Wednesday Oct 12

Structural Equation Modeling (SEM) - Part I (video)

Kline chapter 10

Muthen topic 1 slides 173-200 and topic 2 slides 142-164

9.1Monday Oct 17

SEM - Part II (video)

Kline chapter 14

Muthen topic 1 slides 228-242

9.2Wednesday Oct 19

Multi-level SEM - part I (video)


Muthen topic 7 slides 15-28

Due before class: Homework 3

10.1Monday Oct 24

Multi-level SEM - part II (video)


Muthen topic 7 slides 147-157

Homework 4 (SEM and multi-level SEM) available


10.2Wednesday Oct 26

Review of EFA and CFA

Midterm 2 (EFA and CFA) available Thursday morning, due Saturday evening

11.1Monday Oct 31

Measurement invariance (video)


Kline chapter 16

11.2Wednesday Nov 2

Interaction effects in SEM (video)


Kline chapter 17

Muthen topic 1 slides 201-226

12.1Monday Nov 7

No class - Fall break

12.2Wednesday Nov 9

Guest lecture: cross-lagged panel models (video)

Readings TBA

Due before class: Homework 4

13.1Monday Nov 14

Latent Categorical Analysis (LCA) and Factor Mixture Analysis (FMA) (video)


Muthen topic 5 slides 66-118 and 153-179

13.2Wednesday Nov 16

Review of SEM and multi-level SEM

Kline chapter 18

Midterm 3 (SEM and Multi-level SEM) available Thursday morning, due Saturday evening

14.1Monday Nov 21

Make-up class slot (if needed)

14.2Wednesday Nov 23

No class - Thanksgiving

15.2Monday Nov 28

Rasch modeling - part I (video)

Bond chapters 3-4

15.2Wednesday Nov 30

Rasch modeling - part II (video)


16.1Monday Dec 5

Exam review

16.2Wednesday Dec 7

Make-up class slot (if needed)

examThursday Dec 15

Final exam, 3-5:30pm

Attending class, etc.

Things discussed in class are part of the course materials, and although the slides will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please bring your laptop with R and RStudio installed.

You will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!

Academic integrity

Please refer to the following official statement on academic integrity:

As members of the Clemson University community, we are supposed to have a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we are supposed to recognize that academic dishonesty detracts from the value of a Clemson degree.

Practically speaking: Do not cheat (e.g.: do not collaborate on the midterms and/or final). Plagiarism will not be tolerated, and be dealt with through official university channels, see:

Disability access

Students with disabilities requesting accommodations should contact the Office of Student Disability Services in Suite 239, Academic Success Center building and/or call 864-656-6848 to discuss specific needs within the first month of classes. Students should present a Faculty Accommodation Letter from Student Disability Services when they meet with instructors. Accomodations are not retroactive and new Faculty Accommodation Letters must be presented each semester.

Title IX (Sexual Harassment) statement

Clemson University is committed to a policy of equal opportunity for all persons and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender, pregnancy, national origin, age, disability, veteran‘s status, genetic information or protected activity (e.g., opposition to prohibited discrimination or participation in any complaint process, etc.) in employment, educational programs and activities, admissions and financial aid. This includes a prohibition against sexual harassment and sexual violence as mandated by Title IX of the Education Amendments of 1972. This policy is located at Alesia Smith is the Clemson University Title IX Coordinator. She is also the the Executive Director of Equity Compliance. Her office is located at 223 Brackett Hall, phone: 864-656-3181, email: