Measurement & Evaluation of HCC Systems - Part 2
This is the syllabus website for Clemson University Fall 2018 course HCC 8810.001: Measurement & Evaluation of HCC Systems - Part 2.
Credit hours: 3
Room: McAdams Hall 232
Day and time: Monday & Wednesday, 2:30 – 3:45 pm
Prof. Bart Knijnenburg
Office location: McAdams Hall 215
Office hours: Monday & Wednesday, 3:45 – 4:45 pm
Important: The information below may change!
Changes will be announced in class and through email.
This course will teach you how to scientifically evaluate computing systems using a quantitative, user-centric approach. By the end of this course you will be able to statistically evaluate data obtained from a user experiment, a survey, or system usage log files. This course builds upon Measurement & Evaluation of HCC Systems.
This advanced course will pay special attention to two very important state-of-the-art methods for HCC research: The measurement and evaluation of subjective valuations of users‘ usage experience using multi-item psychometric instruments and exploratory and confirmatory factor analysis (EFA and CFA), and the evaluation of structured models of hypotheses using structural equation modeling (SEM). We will also cover advanced methods such as Rasch modeling and factor mixture analysis.
Most existing HCC research tests hypotheses one by one, and uses behavioral proxies or single-item measurements to test users‘ subjective valuations. Evaluations using CFA and SEM are more accurate, more comprehensive, and easier to report.
What are we going to do?
Course content and structure: This class will be a lot of work, but the advanced methods will give you a competitive advantage over other HCC students at other institutions. This course roughly consists of 4 parts:
- Part 1 (weeks 1-4): Path models (how to do multiple mediation analyses at once)
- Part 2 (weeks 4-5): Psychometrics (how to measure subjective valuations with questionnaires)
- Part 3 (weeks 7-8): Latent variable models (how to create and evaluate measurement scales of subjective valuations using EFA and CFA)
- Part 4 (weeks 9-11): Structural models (how to evaluate structured hypotheses using SEM and multi-level SEM)
- Part 5 (weeks 12-15): Advanced SEM methods (measurement invariance, interaction effects, LCA and FMA)
- Part 6 (weeks 15-16): Rasch modeling (advanced measurement scale analysis)
Course materials: This course uses the following resources:
- Knijnenburg B. P. and Willemsen, M. C. “Evaluating Recommender Systems with User Experiments”: author copy available for free here.
- Chapters 6-14 of Kline, R. B. “Principles and Practice of Structural Equation Modeling”, 4th ed.: for sale on Amazon. In a pinch, you can also use the 3rd edition, but the chapters are different.
- Chapters 1-5 of DeVellis R. F. “Scale Development: Theory and Applications”, 2nd ed.: for sale on Amazon (link 1, link 2). The (more expensive) 3rd edition is not required.
- Chapters 5-6 of Loehlin, J. C. “Latent Variable Models”, 4th ed.: available here.
- Chapter 13 of Tabachnick B. G. and Fidell, L. S. “Using Multivariate Statistics”, 5th ed.: available here.
- Chapters 3-4 of Bond, T. G. and Fox, C. M. “Applying the Rasch Model” 2nd ed.: available for online reading here.
- Selected slides from lectures by Muthen, B. and Muthen, L. Videos and Handouts for Mplus Short Courses.
Software: For the most part, we will use R and RStudio. R is like a programming language, and RStudio is an IDE for R (like how Eclipse is an IDE for Java). R and RStudio are both free. I may show you some CFA and SEM models in MPlus (which is a bit more powerful than R, but not free).
Office hours: Office hours will be directly after class (Monday and Wednesday 3:45-4:45 pm). If you want to attend office hours, please let me know at the end of the class, and we will walk back to my office together. If you show up later, I might have gone home already :-).
Slides: Presentation slides will be posted before each class. They will be linked in the course schedule below (topics listed in orange are clickable and link to the slides).
Assignments: There will be 4 assignments for this class. There will be some “insight” questions and some data analysis questions. Insight questions usually require a short (1-2 sentence) answer. Data analysis questions should be done in R (unless suggested otherwise), and the requisite dataset will be provided. The answers to data analysis questions should contain the executed R commands, a summary of the output (only the parts that answer the question), and an explanation/description of the results in your own words.
Assignments are each worth 10% of your grade. You are allowed to discuss the assignments, but you have to write your own write-up (i.e. you can discuss, but not copy).
Midterms and final: The midterms and final are each worth 15% of your grade, and will be very similar to an assignment, only you cannot collaborate. You will do the midterms on you laptop with the Wi-Fi off.
Prerequisites: You are required to have taken HCC8810: Measurement & Evaluation of HCC Systems or a similar course.
- Assignments: 40% (10% each)
- Midterms and final : 60% (15% each)
In unusual circumstances these percentages could change, but I do not expect that to happen. Your final grade will be calculated by multiplying the percentages with the points you achieve on each assignment and midterm. In my default grading scheme, 80+ is an A, 70+ is a B, 60+ is a C, 50+ is a D, and less than 50 is an F. I sometimes apply a curve to lower some of these thresholds (this has historically happened mostly for the threshold between B and C).
The following cheat sheets have been created for your convenience, to use in the final exam. They outline the steps to conduct a typical CFA and SEM in R. Please do not “blindly” follow the cheat sheets; the questions on the final may require more advanced methods than those presented in the cheat sheets!
|Week||Dates||Topic and contents||Work|
|1.2||Wednesday Aug 22|
(Re-)read before class: Knijnenburg and Willemsen
|2.1||Monday Aug 27|
Kline chapters 2-4
Muthen topic 1 slides 15-25
|2.2||Wednesday Aug 29|
|3.1||Monday Sep 3|
Kline chapter 6-7
Muthen topic 1 slides 27-38
|3.2||Wednesday Sep 5|
Kline chapter 11-12
|4.1||Monday Sep 10|
Homework 1 (Path models) available (use the twp dataset)
|4.2||Wednesday Sep 12|
DeVellis chapters 1-4
|5.1||Monday Sep 17|
DeVellis Chapter 5
Due before class: Homework 1
|5.2||Wednesday Sep 19|
Review of path models
|6.1||Monday Sep 24|
Midterm 1 (Path models)
|6.2||Wednesday Sep 26|
No class - Instructor out of town
|7.1||Monday Oct 1|
Kline chapter 9
Muthen topic 1 slides 40-58
|7.2||Wednesday Oct 3|
Kline chapter 13
Muthen topic 1 slides 107-132 and 147-155
|8.1||Monday Oct 8|
Loehlin chapters 5-6
Muthen topic 1 slides 59-105
Due before class: Homework 2
|8.2||Wednesday Oct 10|
EFA - Part II
Tabachnick chapter 13
Muthen topic 1 topic 2 slides 113-140
|9.1||Monday Oct 15|
Kline chapter 10
Muthen topic 1 slides 173-200 and topic 2 slides 142-164
|9.2||Wednesday Oct 17|
Kline chapter 14
Muthen topic 1 slides 228-242
Due before class: Homework 3
|10.1||Monday Oct 22|
Review of EFA and CFA
|10.2||Wednesday Oct 24|
Midterm 2 (EFA and CFA)
|11.1||Monday Oct 29|
Muthen topic 7 slides 15-28
|11.2||Wednesday Oct 31|
Muthen topic 7 slides 147-157
|12.1||Monday Nov 5|
No class - Fall break
|12.2||Wednesday Nov 7|
Kline chapter 16
Due before class: Homework 4
|13.1||Monday Nov 12|
Kline chapter 17
Muthen topic 1 slides 201-226
|13.2||Wednesday Nov 14|
Review of SEM and multi-level SEM
Kline chapter 18
|14.1||Monday Nov 19|
Midterm 3 (SEM and Multi-level SEM)
|14.2||Wednesday Nov 21|
No class - Thanksgiving
|15.2||Monday Nov 26|
Muthen topic 5 slides 66-118 and 153-179
|15.2||Wednesday Nov 28|
Bond chapters 3-4
|16.1||Monday Dec 3|
|16.2||Wednesday Dec 5|
|exam||Thursday Dec 13|
Final exam, 3-5:30pm
Attending class, etc.
Things discussed in class are part of the course materials, and although the slides will be put on this website, I cannot guarantee that no additional material are discussed in class. Classes will include “follow along” examples, so please bring your laptop with R and RStudio installed. , but please pay attention to the presentation. You may interrupt the speaker with questions, but leave large discussions for the end.
There is no class on September 26 because the instructor will be out of town. Beyond that, you will get an email notification in the event that class is cancelled. If the instructor is more than 15 minutes late, you can assume a last-minute cancellation. Hopefully this will not happen!
Unfortunately, it recently came to my attention that Clemson University plagiarized the contributions of its own alumni. I hold my students to a higher moral standard. As members of the Clemson University community, we are supposed to have a mutual commitment to truthfulness, honor, and responsibility, without which we cannot earn trust and respect of others. Futhermore, we are supposed to recognize that academic dishonesty detracts from the value of a Clemson degree. Therefore, we should not tolerate lying, cheating, or stealing in any form.
As such, plagiarism will not be tolerated, and be dealt with through official university channels, see: http://www.clemson.edu/academics/integrity/plagiarism.html.
Students with disabilities requesting accommodations should contact the Office of Student Disability Services in Suite 239, Academic Success Center building 864-656-6848, to discuss specific needs within the first month of classes. Students should present a Faculty Accommodation Letter from Student Disability Services when they meet with instructors. Accomodations are not retroactive and new Faculty Accommodation Letters must be presented each semester.
Title IX (Sexual Harassment) statement
Clemson University is committed to a policy of equal opportunity for all persons and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender, pregnancy, national origin, age, disability, veteran‘s status, genetic information or protected activity (e.g., opposition to prohibited discrimination or participation in any complaint process, etc.) in employment, educational programs and activities, admissions and financial aid. This includes a prohibition against sexual harassment and sexual violence as mandated by Title IX of the Education Amendments of 1972. This policy is located at http://www.clemson.edu/campus-life/campus-services/access/title-ix/. Mr. Jerry Knighton is the Clemson University Title IX Coordinator. He also is the Director of Access and Equity. His office is located at 111 Holzendorff Hall, 864.656.3181 (voice) or 864.565.0899 (TDD).