Day 1: Overview of ECAs and XR fundamentals, including hardware, development tools, and a hands-on "Hello World" exercise.
Day 2: Theoretical frameworks for ECAs, covering embodiment, social presence, UX, ethics, and key components like NLP and computer vision, with hands-on component breakdown.
Day 3: Workshop on developing a basic ECA, focusing on scene understanding, decision-making, and multimodal interactions.
Day 4: Advanced ECA customization, covering emotion modeling, user adaptation, evaluation methods, and hands-on work with GenAI prompting pipelines.
Project refinement, performance optimization, and final presentations of team-developed ECAs.
A minimum of 75% attendance is required.
Teaching methods
Lectures: Each day introduces core concepts and theories.
Hands-on Workshops: Practical exercises that guide students through developing ECA components and integrating them into XR.
Project Report: Students develop and present a complete ECA in XR as a culmination of the course.
Assessment methods
Daily Assignments: Reinforce daily topics through short exercises.
Project Report: Teams document their developed ECAs, which are assessed on effectiveness, engagement, and adherence to course principles.
Bibliography
Bellucci et al, Immersive Tailoring of Embodied Agents Using Large Language Models, Submitted.
Duong et al., Effective Prompting Strategies to Build LLM-based Embodied Conversational. Submitted.
Lugrin JL, Jacucci G., et al.Present and Future of Health and Safety in Extended Reality. Submitted
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.
S. Aghel Manesh, T. Zhang, Y. Onishi, K. Hara, S. Bateman, J. Li, and A. Tang. How people prompt generative ai to create interactive vr scenes. In Proceedings of the 2024 ACM Designing Interactive Systems Conference, pp. 2319–2340, 2024. 5
