Measurement & Evaluation of HCC Systems - Part 2

HCC 8810.001

This is the syllabus website for Clemson University Fall 2018 course HCC 8810.001: Measurement & Evaluation of HCC Systems - Part 2.

Meeting information:

Credit hours: 3

Room: McAdams Hall 232

Day and time: Monday & Wednesday, 2:30 – 3:45 pm

Instructor information:

Prof. Bart Knijnenburg


Office location: McAdams Hall 215

Office hours: Monday & Wednesday, 3:45 – 4:45 pm

Phone: 864-656-7898

Important: The information below may change!

Changes will be announced in class and through email.

Course description

This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. This course builds upon Measurement & Evaluation of HCC Systems.

This advanced course will pay special attention to two very important state-of-the-art methods for HCC research: The measurement and evaluation of subjective valuations of users‘ usage experience using multi-item psychometric instruments and exploratory and confirmatory factor analysis (EFA and CFA), and the evaluation of structured models of hypotheses using structural equation modeling (SEM). We will also cover advanced methods such as Rasch modeling and factor mixture analysis.

Most existing HCC research tests hypotheses one by one, and uses behavioral proxies or single-item measurements to test users‘ subjective valuations. Evaluations using CFA and SEM are more accurate, more comprehensive, and easier to report.

What are we going to do?

Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 4 parts:

  • Part 1 (weeks 1-4): Path models (how to do multiple mediation analyses at once)
  • Part 2 (weeks 4-5): Psychometrics (how to measure subjective valuations with questionnaires)
  • Part 3 (weeks 7-8): Latent variable models (how to create and evaluate measurement scales of subjective valuations using EFA and CFA)
  • Part 4 (weeks 9-11): Structural models (how to evaluate structured hypotheses using SEM and multi-level SEM)
  • Part 5 (weeks 12-15): Advanced SEM methods (measurement invariance, interaction effects, LCA and FMA)
  • Part 6 (weeks 15-16): Rasch modeling (advanced measurement scale analysis)

Course materials: This course uses the following resources:

  • Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
  • Chapters 6-14 of Kline, R. B. “Principles and Practice of Structural Equation Modeling”, 4th ed.: for sale on Amazon. In a pinch, you can also use the 3rd edition, but the chapters are different.
  • Chapters 1-5 of DeVellis R. F. “Scale Development: Theory and Applications”, 2nd ed.: for sale on Amazon (link 1, link 2). The (more expensive) 3rd edition is not required.
  • Chapters 5-6 of Loehlin, J. C. “Latent Variable Models”, 4th ed.: available here.
  • Chapter 13 of Tabachnick B. G. and Fidell, L. S. “Using Multivariate Statistics”, 5th ed.: available here.
  • Chapters 3-4 of Bond, T. G. and Fox, C. M. “Applying the Rasch Model” 2nd ed.: available for online reading here.
  • Selected slides from lectures by Muthen, B. and Muthen, L. Videos and Handouts for Mplus Short Courses.

Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. I may show you some CFA and SEM models in MPlus (which is a bit more powerful than R, but not free).

Office hours: Office hours will be directly after class (Monday and Wednesday 3:45-4:45 pm). If you want to attend office hours, please let me know at the end of the class, and we will walk back to my office together. If you show up later, I might have gone home already :-).

Slides: Presentation slides will be posted before each class. They will be linked in the course schedule below (topics listed in orange are clickable and link to the slides).

Assignments: There will be 4 assignments for this class. There will be some “insight” questions and some data analysis questions. Insight questions usually require a short (1-2 sentence) answer. Data analysis questions should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words.

Assignments are each worth 10% of your grade. You are allowed to discuss the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy).

Midterms and final: The midterms and final are each worth 15% of your grade, and will be very similar to an assignment, only you cannot collaborate. You will do the midterms on you laptop with the Wi-Fi off.

Prerequisites: You are required to have taken HCC8810: Measurement & Evaluation of HCC Systems or a similar course.


  • Assignments: 40% (10% each)
  • Midterms and final : 60% (15% each)

In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm. In my default grading scheme, 80+ is an A, 70+ is a B, 60+ is a C, 50+ is a D, and less than 50 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).

Cheat sheets

The following cheat sheets have been created for your convenience, to use in the final exam. They outline the steps to conduct a typical CFA and SEM in R. Please do not “blindly” follow the cheat sheets; the questions on the final may require more advanced methods than those presented in the cheat sheets!

Course schedule

For your convenience, you can add the course schedule to your calendar (ICAL or HTML).

WeekDatesTopic and contentsWork
1.2Wednesday Aug 22

Overview and welcome

(Re-)read before class: Knijnenburg and Willemsen

2.1Monday Aug 27

Regression models (recap - part I)

Kline chapters 2-4

Muthen topic 1 slides 15-25

2.2Wednesday Aug 29

Regression models (recap - part II)

3.1Monday Sep 3

Path models - part I

Kline chapter 6-7

Muthen topic 1 slides 27-38

3.2Wednesday Sep 5

Path models - part II


Kline chapter 11-12

4.1Monday Sep 10

Path models - part III

Homework 1 (Path models) available (use the twp dataset)

4.2Wednesday Sep 12

Psychometrics - part I


DeVellis chapters 1-4

5.1Monday Sep 17

Psychometrics - part II

DeVellis Chapter 5

Homework 2 (Psychometrics) available

Due before class: Homework 1

5.2Wednesday Sep 19

Review of path models

6.1Monday Sep 24

Midterm 1 (Path models)

6.2Wednesday Sep 26

No class - Instructor out of town

7.1Monday Oct 1

Confirmatory Factor Analysis (CFA) - part I

Kline chapter 9

Muthen topic 1 slides 40-58

7.2Wednesday Oct 3

CFA - Part II

Kline chapter 13

Muthen topic 1 slides 107-132 and 147-155

8.1Monday Oct 8

Online: Exploratory Factor Analysis (EFA) - part I


Loehlin chapters 5-6

Muthen topic 1 slides 59-105

Due before class: Homework 2

8.2Wednesday Oct 10

EFA - Part II

Tabachnick chapter 13

Muthen topic 1 topic 2 slides 113-140

Homework 3 (EFA and CFA) available


9.1Monday Oct 15

Structural Equation Modeling (SEM) - Part I

Kline chapter 10

Muthen topic 1 slides 173-200 and topic 2 slides 142-164

9.2Wednesday Oct 17

SEM - Part II

Kline chapter 14

Muthen topic 1 slides 228-242

Due before class: Homework 3

10.1Monday Oct 22

Review of EFA and CFA

10.2Wednesday Oct 24

Midterm 2 (EFA and CFA)

11.1Monday Oct 29

Multi-level SEM - part I


Muthen topic 7 slides 15-28

11.2Wednesday Oct 31

Multi-level SEM - part II


Muthen topic 7 slides 147-157

Homework 4 (SEM and multi-level SEM) available


12.1Monday Nov 5

No class - Fall break

12.2Wednesday Nov 7

Measurement invariance


Kline chapter 16

Due before class: Homework 4

13.1Monday Nov 12

Interaction effects in SEM


Kline chapter 17

Muthen topic 1 slides 201-226

13.2Wednesday Nov 14

Review of SEM and multi-level SEM

Kline chapter 18

14.1Monday Nov 19

Midterm 3 (SEM and Multi-level SEM)

14.2Wednesday Nov 21

No class - Thanksgiving

15.2Monday Nov 26

Latent Categorical Analysis (LCA) and Factor Mixture Analysis (FMA)


Muthen topic 5 slides 66-118 and 153-179

15.2Wednesday Nov 28

Rasch modeling - part I

Bond chapters 3-4

16.1Monday Dec 3

Rasch modeling - part II


16.2Wednesday Dec 5

Exam review

examThursday Dec 13

Final exam, 3-5:30pm

Attending class, etc.

Things discussed in class are part of the course materials, and although the slides will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please bring your laptop with R and RStudio installed. , but please pay attention to the presentation. You may interrupt the speaker with questions, but leave large discussions for the end.

There is no class on September 26 because the instructor will be out of town. Beyond that, you will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!

Academic integrity

Unfortunately, it recently came to my attention that Clemson University plagiarized the contributions of its own alumni. I hold my students to a higher moral standard. As members of the Clemson University community, we are supposed to have a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we are supposed to recognize that academic dishonesty detracts from the value of a Clemson degree. Therefore, we should not tolerate lying, cheating, or stealing in any form.

As such, plagiarism will not be tolerated, and be dealt with through official university channels, see:

Disability access

Students with disabilities requesting accommodations should contact the Office of Student Disability Services in Suite 239, Academic Success Center building 864-656-6848, to discuss specific needs within the first month of classes. Students should present a Faculty Accommodation Letter from Student Disability Services when they meet with instructors. Accomodations are not retroactive and new Faculty Accommodation Letters must be presented each semester.

Title IX (Sexual Harassment) statement

Clemson University is committed to a policy of equal opportunity for all persons and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender, pregnancy, national origin, age, disability, veteran‘s status, genetic information or protected activity (e.g., opposition to prohibited discrimination or participation in any complaint process, etc.) in employment, educational programs and activities, admissions and financial aid. This includes a prohibition against sexual harassment and sexual violence as mandated by Title IX of the Education Amendments of 1972. This policy is located at Mr. Jerry Knighton is the Clemson University Title IX Coordinator. He also is the Director of Access and Equity. His office is located at 111 Holzendorff Hall, 864.656.3181 (voice) or 864.565.0899 (TDD).