Syllabus

Status of Syllabus

• This is the Spring 2027 draft. The syllabus is subject to change, provided the class is informed.

Updates to Syllabus

Professor

Francisco R. Ortega, Ph.D. (NUI Lab). Associate Professor (tenured).

• Office: CSB‑360. Available for Zoom/Teams.
• Office Hours: TBA, and other times upon request.
• Phone: 1‑970‑491‑7445 (I don't use this number much – use Teams).
• Email: fortega@colostate.edu · Class email: compsci_CS465@colostate.edu
• Note: Office hours begin the second week of classes and may be cancelled or moved, so please always contact me before coming. My preferred method of communication for quick questions is Teams. If office hours change, I will announce it via Teams.

Teaching Assistants

• TBA
• Online office hours: anyone can request to meet with a TA during online office hours, but priority goes to online students. If several students are in the queue, the TA will answer one question and move to the next student so everyone gets a turn.
• TA office hours run from the second week of classes through the last week of classes, before Finals Week.

Overview

CS 465 is an undergraduate, research-based capstone course on multimodal interaction for 3D user interfaces. A user interface (UI) is the medium of communication between a user and a computer system; a 3DUI is one where the input, the output, or both are in 3D space. This course is about the input side: how people communicate with a system through controllers, hands and gestures, gaze, speech, touch, body, and combinations of these, and how we design, build, and evaluate the interaction techniques that use them.

The course centers on three things.

1. Multimodal input and interaction in AR/VR (the core)

The theory and practice of input devices, interaction techniques (e.g., world-in-miniature, ray casting, gaze + pinch), and multimodal interaction, and how to evaluate them (e.g., 3D Fitts' law, gesture elicitation, controlled studies, qualitative methods). AR/VR is required in this course: every project must run in a headset, and every project must make input and interaction its central contribution. Most head-mounted displays will be Meta Quest (with video-see-through AR); some optical-see-through AR (e.g., HoloLens 2) and additional input devices may be available. Some output work is needed to give feedback on the interaction, but the key part of the course is how the user interacts with the system, not what they see.

2. Agentic AI for XR development and research

A core skill of this course, and one useful well beyond it, is working with agentic AI. We use agentic coding and research agents (Claude Code, OpenAI Codex, opencode, and others, including agentic AI inside VS Code) for coding, system setup, and research, combining local AI models with frontier models. You stay the decision-maker and remain fully accountable for what the agents produce, while learning where they help, where they fail, and how to keep a human in control.

3. LLMs as an instrument

Language models are not only a coding aid; in this course they are also an instrument. In the interface, an LLM can be part of the interaction itself: speech as a modality, interpreting a user's intent from speech plus gesture or gaze, or an assistant that acts in the XR scene. In the research, a model can be a study instrument: helping structure the analysis of qualitative data, generating stimuli, or serving as a consistent, controllable component of an experiment. We treat model choice as a human-centered design question: from small on-device models to large frontier LLMs, capability grows along with cost, latency, privacy exposure, and the chance of confident-but-wrong answers, so what a person should delegate, verify, or keep under direct control shifts along that spectrum. Every use of a model as an instrument must be disclosed and described in the methods of your paper.

Who This Course Is For

CS 465 is an undergraduate course. It is research-based and includes a substantial implementation, which also makes it a challenging opportunity for graduate students, who may count up to two 400-level courses toward their degree. Requirements, deliverables, and grading are the same for everyone.

Working With Your Advisor or Another Faculty Member

You are welcome to work on a problem with me, with your own faculty advisor, or with another faculty member, as long as the project fits this course: it must run in AR/VR and make input and interaction its central contribution. Talk to me early so we can scope it to the course requirements (proposal, checkpoints, paper, and implementation). Your work and authorship remain your own, and you stay accountable for what you submit.

Course Objectives

By the end of CS 465, you should be able to:

• Understand the theory and concepts behind input devices, interaction techniques, and multimodal interaction for 3D user interfaces.
• Design and build multimodal interaction techniques in AR/VR.
• Run experiments for input devices and interaction techniques (a briefer introduction than CS 464), including quantitative and qualitative methods.
• Use agentic AI tools and local + frontier models effectively and responsibly for coding, system setup, and research, and use LLMs as an instrument in the interface and in the study, keeping a human in control and staying accountable for the results.
• Carry out a capstone project that includes coding and research in a team, from proposal through implementation and a research paper.
• Communicate research progress clearly through presentations and writing.

How This Class Is Using LLMs This Semester

Disclosure matters. As part of his research on the intersection of agentic AI with XR and HCI, the instructor uses a range of models, both local and frontier, to edit text, generate content, and assist with course material, while still making every decision himself. The buck stops with the professor. Agentic AI is a tool, nothing more. We will talk about this at length in class.

Use of AI in CS 465

AI is allowed in this course. If you use it, you remain fully responsible and accountable for everything you submit. AI is a tool, not a substitute for understanding.

• Coding: Use AI to explain concepts, scaffold, and debug. You may not submit work you cannot explain. If I ask you about your code or results and you can't account for them, you may receive a zero. The work is yours; "the AI did it" is not a defense.
• Writing: You may use AI to help you write and edit (including tools like Grammarly). The ideas, claims, and citations remain your responsibility, including verifying that every reference exists and is relevant.
• Disclosure: This class requires you to disclose AI used for writing assistance, even though ACM no longer does (IEEE does, and you may not know the venue in advance). AI used in the research itself (data analysis, statistics, generating data or stimuli, models inside your system, code or figures that affect your conclusions) must be described in detail in the methods section, following the current ACM Policy on Authorship.

The AI Dilemma

How computer scientists should use AI to aid coding and writing (and these are different) is an open question I take seriously, alongside the ethical considerations (energy, copyright, confidentiality). I will share my current thinking and we will discuss it in class. If you let AI do everything, you will not learn, and you will not be able to maintain what you "wrote." Use it to support your learning, not to replace it.

Email, Canvas, and Teams

Email, Canvas, Teams, and this syllabus (plus the Final Project page) are critical. Check them regularly. Classes will be recorded, but I can't guarantee the system's functionality; online students will get a special recording if Echo360 fails.

How to communicate

• Use Teams for a faster response. Use email for private or essential matters where timing is less critical. When emailing, use the class email or CC the TAs, and begin the subject line with "CS-465". If I don't reply within 48 hours, email again.
• Do not use the Canvas messaging system to contact the TAs or the instructor.
• All official announcements are sent via Microsoft Teams. You will be added in the first week. Use the correct channel and be respectful to your peers, TAs, and instructor. Hate speech or harassment will not be tolerated.
• Please address me as Professor, Professor Ortega, Dr. Ortega, or Francisco — your preference.
• My pronouns are He/His/Him. If I get your pronouns or the pronunciation of your name wrong, please correct me, in private or in class. I have an accent and some hearing/speech differences, so feel free to correct me at any time.
• I don't know everything about HCI, VR, or AI. I will always come back with an answer when I can find one.

Style of HCI Courses

HCI courses, including this one, focus on concepts and theory as well as building, which is not common in all CS undergraduate courses. Expect to read research papers, write throughout the semester, and build a working AR/VR system.

Learning Breaks

Research in the science of learning shows that sustained attention during long lectures decreases learning. I will use structured mini-breaks (3–5 minutes) during some class sessions; use them for peer discussion and to process the material. I recommend Rhodes, M. G., Cleary, A. M., & Bjork, E. L. (2024). A Guide to Effective Studying and Learning.

Project Teams

Projects are done in groups of 3–4. Hardware is the limiting factor for smaller groups. Students may form their own groups, and I can assist if needed. You will evaluate your team members about three times during the semester. A larger group is expected to deliver more than a smaller one, in both the implementation and the paper. Details, milestones, and the rubric are on the Final Project page and on Canvas by the end of the second week.

Project Requirements

• Every project must run in AR or VR (headset).
• Every project must make input and interaction its central contribution (unimodal or multimodal).
• Every project must be human-centric and include an evaluation.
• You are not required to have a novel approach (though it is encouraged); evaluating existing techniques or asking new questions about them is welcome.
• LaTeX is required for the research paper and the checkpoints that lead to it.

Possible project categories (more ideas will be presented in class):

• A multimodal input technique in VR (e.g., gaze + pinch, speech + gesture).
• Interaction in passthrough MR or optical see-through AR.
• Evaluating an input method (e.g., 3D Fitts' law, gesture elicitation).
• Multimodal input with an LLM as part of the interaction (e.g., a speech or LLM agent in XR).

Presentations

There will be presentations this semester. The format and schedule will be announced on the Schedule and Final Project pages.

Prerequisites

See your advisor or the CS website. Programming maturity is assumed: you need to be self-sufficient in the project and assignments.

Grading Scale

• A+: given only to top students who have gone above and beyond (and already have an A). For your GPA, an A+ is the same as an A.
• I reserve the right to curve in your favor per deliverable, exam, or for the class. This is uncommon, but if it happens I will announce it.

Grading Breakdown

This course consists of 1,000 points; 1,000 points = 100%.

• Quizzes: each quiz says where its answers can be found. Answers drawn from outside those resources may be marked incorrect.
• Hardware: if you borrow hardware, you are expected to return it in the same condition (including boxes and accessories). Hardware not returned, or returned damaged or dirty, may cost points.

Exam Dates

Exams 1 and 2 are given during class time via the Canvas proctored application. If Canvas is unavailable, a hard copy is provided for face-to-face students.

The final exam date and time are set by the Registrar and cannot be changed. Exams are proctored with a CSU-approved system.

Equipment for Exams

You need a computer or laptop in a quiet location. You will take a practice quiz to test your system by a specific date; it opens by the second or third week of classes. If you have trouble getting a laptop, talk to me and the TAs.

Additional Information about Exams

Coding is not the main topic of this class (we will still code), so you will not be asked coding questions. You may be asked about the concepts behind the assignments (e.g., Fitts' law).

Online Students

Online students follow the same requirements, deliverables, and grading as face-to-face students, with a few small differences because they cannot attend in person:

• Hardware: online students are mailed a headset (unless they can pick it up in person) and return it by mail; CSU Online pays for shipping. Face-to-face students share headsets within their group.
• Computer: online students need a laptop or desktop with a GPU compatible with Quest development (mobile deployment at minimum; PCVR needs a stronger GPU).
• Exams: online students get a reasonable time window to take each exam, with the same duration as face-to-face students.
• Participation: attendance is tracked for face-to-face students; online students' participation is graded as described on Canvas.
• Presentations and demos: online students present by video upload.
• Office hours: online students get priority in the TAs' online office hours.
• Recordings: online students get a special recording if Echo360 fails.
• Extra credit: online students will have extra-credit opportunities in place of in-class pop-up quizzes.

Attendance (face-to-face students)

If you are a face-to-face student, I expect you to attend every class and be an active participant. You may be absent five times during the semester, no questions asked (travel, illness, or any reason). Beyond that, extreme cases with documentation from case management will be considered. Attendance is tracked to comply with student financial aid. There may be extra-credit pop-up quizzes in class for face-to-face students.

Electronics

Electronics should be used only to take notes or follow the lecture slides. If you need to text, take a call, or browse, please step outside. If you use a laptop, try to sit in the back.

Late Work and Make-ups

Late work is not accepted unless I make an exception for a serious case. The course is built with drops (lowest exam, lowest assignment, lowest quiz) so a single missed item will not sink you. If a submission carries a one-day late policy, I will announce it, with a 10% penalty on that submission.

Travel

I may have to travel a few times during the semester; the class will be covered by a TA or a guest speaker. I also understand if you need to travel for research, hackathons, or work, but you are responsible for meeting deadlines. Don't travel on exam days.

About the Class Schedule

Students' needs differ from one semester to the next, so the schedule is subject to change. This class covers state-of-the-art topics, so not all lectures are known in advance. Exam dates are fixed, and other dates will be posted with advance notice on Canvas.

Time Outside of Class

Reserve the time this class needs. A rule of thumb is 3–4 hours of study per credit per week; for a 4-credit course that is 12–16 hours outside class, including at least 10 hours per week on the project, more toward the end. Outside work includes lecture materials, readings, and coding tutorials or practice with our resources and support.

Unity Game Engine and Other Engines

You may use any game engine or language for your final project, but the assignments are in Unity. If you choose another engine (e.g., Godot or Unreal), that is acceptable, but we cannot provide support for it. The lab computers run only Unity.

Labs and Computer Room

Labs are on your own; there is no lab section to attend. Lab computers are available in CSB 225 (hours TBA). Other classes also use CSB 225, so be respectful of their time blocks. Back up your work to a remote drive and use Git; local lab drives are wiped periodically.

Textbook

We use resources provided in class (PDFs and slides in Canvas). One free book we draw from: Steven M. LaValle, Virtual Reality, Cambridge University Press, 2023 (lavalle.pl/vr). See the Resources page.

LaTeX

LaTeX is required for the research paper and for the checkpoints that lead to it. Templates and details are on the Final Project page by the second week of classes.

Performance Measures

Some questions and deliverables are not multiple choice; the instructor uses judgment and experience to evaluate them, based on the expected answer and the level of this class. Because this is a research course, the quality of the work is part of the grade.

Respect for Others

Be respectful of everyone's right to be who they are, including members of the LGBTQ+ community, minorities (race, ethnicity, etc.), and people of differing gender, culture, personality, and political views. This class deals with humans, so opinions will sometimes differ; we will use evidence and science to work through the issues. Order in class is required.

Important Information Regarding Accommodations for Students

Colorado State University's Student Disability Center (SDC) coordinates accommodations. If you have not registered with the SDC, start at New SDC Student: Get Started.

• Contact me as soon as you receive your memo from the SDC to discuss your needs. Please read this section before our conversation.
• Our conversation should address the specific requests in the SDC memo, their applicability, and reasonable arrangements for the course.
• All students are expected to submit all deliverables as defined in the syllabus. Time extensions may be limited by the team-based, progress-based structure of the project; an extension as a reasonable accommodation will be considered.
• Accommodations cannot fundamentally alter the nature of a course.
• Retroactive accommodations are generally not permitted.

Teaching Philosophy and Other Matters

• My objective is for you to learn the core principles of the course: the theory and practice of multimodal interaction in 3D user interfaces.
• I aim for a science-of-learning approach (repetition, spacing, retrieval).
• I start from trust. I will not try to trick you on exams or deliverables. This does not mean cheating is acceptable.
• AI detectors are unreliable and can disproportionately harm historically marginalized groups; I do not use them as proof.
• There is, by design, a separation between students and me — I can affect your grade. TAs do not date students, and professors don't date students or TAs.
• When in doubt, ask me, preferably in writing.
• I am the arbiter of this class, but you may appeal a grade or contact my superiors. Try to resolve issues with me first.
• Your grade evaluates where you were during this class in a given period. It says nothing about you as a person.
• I will not tolerate sexual harassment, abuse, physical violence, or intimidation, and I will report it.
• I want you to have as much fun as you can while there is grading!

About Cheating (Academic Integrity)

Don't cheat. Make sure you understand the CSU Academic Honor Principle. On exams and quizzes, all work must be your own. For the project, collaboration within your team is expected, but work shared across teams must be cited, and you must attribute any code or text you did not write. If you are talking in plain English about ideas across teams, you are probably fine; if you are dictating code line by line to another team, you have crossed the line. Submitting plagiarized work may result in an F for the class and additional consequences.

© 2027 Dr. Francisco R. Ortega. All rights reserved.