Deep Generative Models for Visual Data: Undergraduate Course Syllabus

Course Description

This course focuses on modern deep generative models trained on large-scale, unstructured, multimodal data, such as text and images. These models can produce high-quality outputs that closely resemble real data and can be controlled or conditioned on desired attributes. Many important real-world problems can be naturally framed as generative tasks, in which structured outputs are produced from given inputs, such as generating textual descriptions from images. Beyond their direct generative capabilities, a central focus of the course is their transformative role in creating realistic, diverse, and labeled synthetic datasets, enabling the training and evaluation of AI systems in domains where annotated data is limited, expensive, or unavailable.

Course Content

This course provides a comprehensive introduction to the architectures and foundation models that underpin modern generative AI, as well as the generative modelling approaches built on top of them. It begins with key challenges in computer vision. The course then introduces core architectural foundations, including Transformers and Vision Transformers, and examines how these models support both discriminative and generative systems. Building on this foundation, students study major generative approaches, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and, with primary emphasis, diffusion models and the Stable Diffusion family. The course highlights modern techniques for controllable, fine-grained generation, as well as the use of generative models to create realistic synthetic training and evaluation data. Adopting a code-first approach, it presents core concepts through hands-on examples using contemporary libraries and tools.

Prerequisites

All necessary background material, including machine learning and deep learning fundamentals and the PyTorch library, is introduced during the course. Students are expected to be proficient in Python programming and comfortable with basic linear algebra and probability. Prior exposure to machine learning or computer vision is helpful but not required.

Expected Outcomes

Students who complete the course will acquire a broad, in-depth understanding of deep generative models and their applications. By the end of the course, students will have gained hands-on experience with state-of-the-art AI models and software libraries, developing practical skills through structured, guided projects. These projects will provide opportunities to design and implement AI solutions by combining, adapting, and extending existing components. The course emphasizes both technical proficiency and creative problem-solving, preparing students to innovate using modern AI technologies in real-world development scenarios. Students will develop a tangible AI project that can be showcased in a professional portfolio. At the same time, in-class presentations will enhance their ability to present and articulate complex technical work with confidence.

Course Format

The course runs for 13 weeks with one session per week. Ten sessions are lectures on the week's topic, taught with a code-first approach through hands-on examples using contemporary libraries and tools. The remaining three sessions (Weeks 5, 9, and 13) are dedicated entirely to student project presentations: proposal, interim, and final. In all three, teams present and receive in-class feedback from the instructor and peers.

Course Project

During the course, students will propose, implement, and present an innovative project based on the concepts studied in class. Each project will address a novel vision-related problem chosen by the students, with particular emphasis on using generative models to create synthetic training and evaluation data. The project will involve comparing different strategies for synthetic data generation and evaluating multiple deep learning models, including at least one model based on the Transformer architecture. For inspiration, a sample of past student projects supervised by the instructor is available at apartsin.com/teaching/student-projects. The milestones, all presented in class, are:

Grading

Delivery	Grade
In-class project proposal presentation	No grade, for feedback and approval only
In-class project interim presentation	20%
In-class final project presentation	40%
Final project submission (GitHub repository)	40%

Weekly Schedule

Below is an approximate weekly schedule outlining the subjects that will be covered. The actual order or content may vary depending on the class background, recent advancements in the field, or the specific focus of student projects. Chapter numbers refer to the course text [1]; each entry links to the corresponding chapter. Presentation weeks are highlighted.

Week	Theme and readings
1	Introduction Typical vision tasks, project requirements, PyTorch and deep-learning tutorials. Textbook [1]: Ch. 18, Neural Networks & PyTorch for Vision; Ch. 30, Foundations of Generative Modeling. Sections: 30.1 Generative vs Discriminative; 30.2 A Map of Generative Families.
2	Introduction to Language Foundation Models HuggingFace libraries, LLMs for text generation and representation, and an introduction to fine-tuning. Textbook [1]: Ch. 25, Self-Supervised Learning & Vision Foundation Models, §25.4 CLIP: Language as Supervision; Ch. 34, Text-to-Image, §34.1 Connecting Text & Pixels: CLIP & Text Encoders.
3	Vision Foundation Models Vision models for image representation and classification, and image feature encoding. Textbook [1]: Ch. 20, CNN Architectures: From LeNet to ConvNeXt; Ch. 25, Self-Supervised Learning & Vision Foundation Models.
4	Synthetic Image Data Generation for Model Training Introduction to Stable Diffusion pipelines; using generative models to create synthetic data for image classification and object detection. Textbook [1]: Ch. 33, Diffusion Models; Ch. 34, Text-to-Image, §34.2 Inside Stable Diffusion: VAE, U-Net, DiT & Conditioning.
5	Student Presentations: Project Proposal
6	Transformers Deep Dive Tokenization and word embedding, attention, transformer blocks, pretraining, and Parameter-Efficient Fine-Tuning (PEFT). Textbook [1]: Ch. 22, Vision Transformers (attention, transformer blocks, pretraining); Ch. 35, Controllable Generation & Image Editing, §35.2 Personalization: LoRA, DreamBooth & Textual Inversion.
7	Video Transformers Vision Transformers, DeiT/DINO/SWIN models, DETR, and multimodal models: CLIP/BLIP. Textbook [1]: Ch. 22, Vision Transformers (DeiT, DINO, Swin); Ch. 23, Object Detection (DETR); Ch. 25, Vision Foundation Models, §25.4 CLIP.
8	Variational Autoencoders Autoencoders, autoencoder variants, the VAE, and VAE variants. Textbook [1]: Ch. 31, Autoencoders & Variational Autoencoders. Sections: 31.1 Autoencoders; 31.3 The VAE: ELBO, Reparameterization & Amortized Inference; 31.4 Disentanglement & beta-VAE; 31.6 Discrete Latents: VQ-VAE.
9	Student Presentations: Interim Report
10	Stable Diffusion Mathematical background, diffusion models, latent diffusion models, guidance, long prompts, and SDXL. Textbook [1]: Ch. 33, Diffusion Models. Sections: 33.1 Forward & Reverse Processes; 33.2 DDPM; 33.4 Fast Sampling: DDIM & Solvers; 33.7 Latent Diffusion; and Ch. 34, Text-to-Image, §34.2 Inside Stable Diffusion (guidance, SDXL).
11	Stable Diffusion Applications Inpainting, DepthNet, Text2Video, Prompt2Prompt, instruction editing, image inversion, ControlNet, DreamBooth, and Textual Inversion. Textbook [1]: Ch. 35, Controllable Generation & Image Editing. Sections: 35.1 Spatial Control: ControlNet & Conditioning Adapters; 35.2 Personalization: LoRA, DreamBooth & Textual Inversion.
12	Generative Adversarial Networks GAN introduction, DCGAN, WGAN, Pix2Pix/SRGAN, CycleGAN, StyleGAN, and BigGAN. Textbook [1]: Ch. 32, Generative Adversarial Networks (DCGAN, WGAN, Pix2Pix, CycleGAN, StyleGAN, BigGAN).
13	Student Presentations: Project Finals

Policies

Use of AI tools

The use of AI tools for generating project code and presentation slides is highly encouraged, provided that two essential requirements are met: the project must demonstrate novelty by addressing a new and valuable task, and the team must maintain full ownership and responsibility for all submitted code and presentation materials.

Collaboration and integrity

Discussion across teams is encouraged; the code, experiments, and writing submitted by each team must be its own. All results reported in presentations and documentation must be backed by runnable artifacts in the team's repository.

References

[1] A. Apartsin and Y. Aperstein, Building Vision AI: From Pixels to Generative Models, 2nd ed., 2026. Open access online at https://visionbook.icsgen-ai.org; also available as an ebook at Amazon: https://www.amazon.com/dp/B0H5BT8Y75. Part IV (Generative Vision Models, Chapters 30 to 38) is the primary text for the course; Part III (Chapters 18 to 29) supports the deep-learning-for-vision material.

Supplementary books

[2] D. Foster, Generative Deep Learning, 2nd ed. O'Reilly Media, 2022.

[3] J. M. Tomczak, Deep Generative Modeling, 2nd ed. Springer, 2024.

[4] O. Sanseviero, P. Cuenca, A. Passos, and J. Whitaker, Hands-On Generative AI with Transformers and Diffusion Models. O'Reilly Media, 2024.

[5] M. Liu, Learn Generative AI with PyTorch. Manning Publications, 2024. Available at manning.com.