Advanced Measurement & Evaluation of HCC Systems
HCC 8410
This is the syllabus website for Clemson University Fall 2022 course HCC 8410: Advanced Measurement & Evaluation of HCC Systems.
Meeting information:
Credit hours: 3
Room: McAdams Hall 107
Day and time: Monday & Wednesday, 2:30 – 3:45 pm
Prerequisites: You are required to have taken HCC6400: Measurement & Evaluation of HCC Systems or a similar course on statistical analysis. If you are not sure whether you meet the prerequisites, please email me.
Instructor information:
Prof. Bart Knijnenburg
Email: bartk@clemson.edu
Office location: McAdams Hall 205
Office hours
Office hours will be by appointment. If you want to attend office hours, please let me know at the end of the class, or send me an email.
Important: The information below may change!
Changes will be announced in class and through email.
Course description
This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. This course builds upon Measurement & Evaluation of HCC Systems.
This advanced course will pay special attention to two very important state-of-the-art methods for HCC research: The measurement and evaluation of subjective valuations of users‘ usage experience using multi-item psychometric instruments and exploratory and confirmatory factor analysis (EFA and CFA), and the evaluation of structured models of hypotheses using structural equation modeling (SEM). We will also cover advanced methods such as Rasch modeling and factor mixture analysis.
Most existing HCC research tests hypotheses one by one, and uses behavioral proxies or single-item measurements to test users‘ subjective valuations. Evaluations using CFA and SEM are more accurate, more comprehensive, and easier to report.
Course Learning Objectives
Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 4 parts:
- Part 1 (weeks 1-4): Path models (how to do multiple mediation analyses at once)
- Part 2 (week 5): Psychometrics (how to measure subjective valuations with questionnaires)
- Part 3 (weeks 6-8): Latent variable models (how to create and evaluate measurement scales of subjective valuations using EFA and CFA)
- Part 4 (weeks 8-11): Structural models (how to evaluate structured hypotheses using SEM and multi-level SEM)
- Part 5 (weeks 11-13): Advanced SEM methods (measurement invariance, interaction effects, cross-lagged panel models, LCA and FMA)
- Part 6 (week 15-16): Rasch modeling (advanced measurement scale analysis)
Course modality and COVID/flu precautions
Those enrolled in remote sections of this course may join using Zoom. The Zoom link may also be used by anyone who feels under the weather, is in quarantine, or is in any other way concerned about their health and safety. Since some people in class have unvaccinated children, I strongly prefer that you are vaccinated and boosted if you join the class in person. Furthermore, if you have a cough or a runny nose, I strongly prefer that you wear a KN95 mask. The adoption of basic public health measures is entirely consistent with the exemplary role we embody as academics.
Course materials
Readings: This course uses the following resources:
- Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
- Chapters 6-14 of Kline, R. B. “Principles and Practice of Structural Equation Modeling”, 4th ed.: for sale on Amazon. In a pinch, you can also use the 3rd edition, but the chapters are different.
- Chapters 1-5 of DeVellis R. F. “Scale Development: Theory and Applications”, 2nd ed.: for sale on Amazon (link 1, link 2). The (more expensive) 3rd edition is not required.
- Chapters 5-6 of Loehlin, J. C. “Latent Variable Models”, 4th ed.: available here.
- Chapter 13 of Tabachnick B. G. and Fidell, L. S. “Using Multivariate Statistics”, 5th ed.: available here.
- Chapters 3-4 of Bond, T. G. and Fox, C. M. “Applying the Rasch Model” 2nd ed.: available for online reading here.
- Selected slides from lectures by Muthen, B. and Muthen, L. Videos and Handouts for Mplus Short Courses.
Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. I may show you some CFA and SEM models in MPlus (which is a bit more powerful than R, but not free).
Slides: Presentation slides are already linked in the course schedule below (topics listed in orange are clickable and link to the slides).
Course Grading Scheme and Grading Policies
Assignments: There will be 4 assignments for this class. They should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words (make sure to always explain your answer!).
Assignments are each worth 10% of your grade. You are allowed to discuss the assignments or consult the Internet or an AI system, but you have to write your own write-up (i.e. you can discuss, but not copy). If you collaborate with others, please add a collaboration statement to your assignment (a simple statement saying “I collaborated with [name(s)]” is sufficient). If you found something online, add a citation. If you used AI, acknowledge this by mentioning the system and the prompt that you used to get to the answer.
Midterms and final exam: The midterms and the final are take-home and are each worth 15% of your grade. They will be very similar to the assignments, but you have a shorter amount of time for them, and you are not allowed to collaborate, use AI, or look stuff up online.
Grading Scheme:
- Assignments: 40% (10% each)
- Midterms and final : 60% (15% each)
In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm. In my default grading scheme, 85+ is an A, 80+ is an A-, 75+ is a B+, 70+ is a B, 65+ is a B-, 60+ is a C+, 55+ is a C, 50+ is a C-, 45+ is a D, and less than 45 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).
Cheat sheets
For the homeworks, students should learn to apply the methods using the book and the lecture slides. For the midterms and the final, I will provide a cheat sheet for each method. The cheat sheets outline the steps to conduct a typical CFA and SEM in R. Please do not “blindly” follow the cheat sheets; the questions on the final may require more advanced methods than those presented in the cheat sheets!
- (no cheat sheets available yet)
Course schedule
For your convenience, you can add the course schedule to your calendar (ICAL or HTML).
| Week | Dates | Topic and contents | Work |
| 1.2 | Wednesday Aug 19 | (Re-)read before class: Knijnenburg and Willemsen | |
| 2.1 | Monday Aug 24 | Kline chapters 2-4 Muthen topic 1 slides 15-25 | |
| 2.2 | Wednesday Aug 26 | ||
| 3.1 | Monday Aug 31 | Kline chapter 6-7 Muthen topic 1 slides 27-38 | |
| 3.2 | Wednesday Sep 2 | Kline chapter 11-12 | |
| 4.1 | Monday Sep 7 | No class - Labor Day | |
| 4.2 | Wednesday Sep 9 | Homework 1 (Path models) available (use the twp dataset) | |
| 5.1 | Monday Sep 14 | DeVellis chapters 1-4 | |
| 5.2 | Wednesday Sep 16 | DeVellis Chapter 5 Homework 2 (Psychometrics) available Due before class: Homework 1 (Path models) | |
| 6.1 | Monday Sep 21 | Review of path models | Midterm 1 (Path models) available after class, due Wednesday before class |
| 6.2 | Wednesday Sep 23 | Kline chapter 9 Muthen topic 1 slides 40-58 Due before class: Midterm 1 | |
| 7.1 | Monday Sep 28 | Kline chapter 13 Muthen topic 1 slides 107-132 and 147-155 | |
| 7.2 | Wednesday Sep 30 | Loehlin chapters 5-6 Muthen topic 1 slides 59-105 | |
| 8.1 | Monday Oct 5 | EFA - Part II | Tabachnick chapter 13 Muthen topic 1 topic 2 slides 113-140 Homework 3 (EFA and CFA) available Due before class: Homework 2 (Psychometrics) |
| 8.2 | Wednesday Oct 7 | Kline chapter 10 Muthen topic 1 slides 173-200 and topic 2 slides 142-164 | |
| 9.1 | Monday Oct 12 | No class - Fall break | |
| 9.2 | Wednesday Oct 14 | Kline chapter 14 Muthen topic 1 slides 228-242 Due before class: Homework 3 | |
| 10.1 | Monday Oct 19 | Review of EFA and CFA | Midterm 2 (EFA and CFA) available after class, due Wednesday before class |
| 10.2 | Wednesday Oct 21 | Muthen topic 7 slides 15-28 Due before class: Midterm 2 | |
| 11.1 | Monday Oct 26 | Muthen topic 7 slides 147-157 | |
| 11.2 | Wednesday Oct 28 | Kline chapter 16 | |
| 12.1 | Monday Nov 2 | Kline chapter 17 Muthen topic 1 slides 201-226 | |
| 12.2 | Wednesday Nov 4 | Guest lecture: cross-lagged panel models | Readings TBA Due before class: Homework 4 |
| 13.1 | Monday Nov 9 | Review of SEM and multi-level SEM | Kline chapter 18 Midterm 3 (SEM and Multi-level SEM) available after class, due Wednesday before class |
| 13.2 | Wednesday Nov 11 | Latent Categorical Analysis (LCA) and Factor Mixture Analysis (FMA) | Muthen topic 5 slides 66-118 and 153-179 Due before class: Midterm 3 |
| 14.1 | Monday Nov 16 | Make-up class slot (if needed) | |
| 14.2 | Wednesday Nov 18 | Make-up class slot (if needed) | |
| 15.2 | Monday Nov 23 | Bond chapters 3-4 | |
| 15.2 | Wednesday Nov 25 | No class - Thanksgiving | |
| 16.1 | Monday Nov 30 | ||
| 16.2 | Wednesday Dec 2 | ||
| exam | Final exam released Tuesday Dec 8 at 5:30pm | Final exam due Thursday Dec 10 at 5:30pm | |
Attending class, etc.
Things discussed in class are part of the course materials, and although the slides will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please bring your laptop with R and RStudio installed.
You will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!
AI Policy
Assignments: You are allowed to consult the Internet or an AI system for help with the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy). If you use AI, acknowledge this by mentioning the system and the prompt that you used to get to the answer.
Midterms and final exam: You are not allowed to collaborate, use AI, or look stuff up online for the midterms and the final.
Concerns regarding potential violations of this policy will be reviewed fairly through the appropriate university academic integrity process.
University Policies
You can find information about campus wide policies, including student affairs information and accessibility services, by clicking on Clemson‘s University Policies Page. There, you can find information about academic integrity, access and equity (including student accessibility and Title IX info), student financial services, emergency planning, and more. My expectation is that you will review these policies carefully and be responsible for them this semester.
Academic integrity
As members of the Clemson University community, we have inherited Thomas Green Clemson‘s vision of this institution as a “high seminary of learning.” Fundamental to this vision is a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we recognize that academic dishonesty detracts from the value of a Clemson degree. Therefore, we shall not tolerate lying, cheating, or stealing in any form. Using materials generated using artificial intelligence (AI) that are turned in without attribution is considered plagiarism.
Practically speaking: Do not cheat (e.g.: do not collaborate or use Google/AI on the midterms and/or final). Plagiarism will not be tolerated, and be dealt with through official university channels, see: http://www.clemson.edu/academics/integrity/plagiarism.html.
Cheat sheets are created for your convenience to help you with the midterms and the exam. Please do not use cheat sheets (e.g., from last year) when making the homework. Using last years‘ cheat sheets on the homeworks is considered cheating, and will be dealth with accordingly.
All infractions of academic dishonesty by undergraduates must be reported to Undergraduate Studies for resolution through that office. In cases of plagiarism instructors may use the Plagiarism Resolution Form. See the following resources:
Disability access
Clemson University values the diversity of our student body as a strength and a critical component of our dynamic community. Students with disabilities or temporary injuries/conditions may require accommodations due to barriers in the structure of facilities, course design, technology used for curricular purposes, or other campus resources. Students who experience a barrier to full access to this class should let the instructor know and make an appointment to meet with a staff member in Student Accessibility Services as soon as possible. You can make an appointment by calling 864-656-6848, by emailing CUSAS@clemson.edu, or by visiting Suite 239 in the Academic Success Center building. Appointments are strongly encouraged — drop-ins will be seen, if at all possible, but there could be a significant wait due to scheduled appointments. Students who have accommodations are strongly encouraged to request, obtain, and send these to their instructors via SAS as early in the semester as possible so that accommodations can be made in a timely manner. It is the student‘s responsibility to follow this process each semester.
Title IX (Sexual Harassment) statement
The Clemson University Title IX statement: Clemson University is committed to creating and continuously fostering a caring community based on the core values of integrity, honesty and respect. Sexual discrimination, which includes sexual harassment, sexual violence, stalking and domestic and/or relationship violence, is unacceptable and has no place in Clemson’s community. Consistent with its Title IX obligation, the University prohibits discrimination, including sexual and gender-based harassment and violence, in all its programs and activities, including academics, employment, athletics, and other extracurricular activities. This Title IX policy is available online. Katherine Weathers is the Clemson University Title IX Coordinator and VP of Inclusive Excellence. She can be reached at (864) 656-3413 or via email at kweath3@clemson.edu. Remember, email is not a fully secured method of communication and should not be used to discuss Title IX issues.