2017 Schedule

Lecture Date Title Details Related Reading Presenter
1 09/25/2017 Introduction
[Silvio's slides] [Amir's slides]
  • Motivation
  • Overview of the topics and lectures
  • What is a representation (X2VEC)? Why does it matter?
  • Logistics and policies
Silvio Savarese and Amir Zamir
2 10/02/2017 Representation 101
Background (pre 2012)
Fully supervised representations I

Fully supervised representations II [slides]
  • Basic desirable properties of representations
    • Ill-posedness, nonlinearity, complexity, dimensionality, vertical vs horizontal domain, etc.
  • Handcrafted representations
    • 2D Matching features (e.g. SIFT, HOG, DAISY, Kernels)
    • Temporal (video) Features (e.g., 3DSIFT, Dense Trajectory Features - DTF, STIP, ICCV15 Storyline)
  • Fully supervised representations:
    • 2D matching features (e.g. matchnet)
    • 2D object detection features (e.g. Imagenet)
  • Brief executive summary of representation understanding methods

Amir Zamir
3 10/9/2017 Recurrence/Feedback based representations
Structured representations

Losses and GAN's [slides]
  • Curriculum learning
  • Feedback
  • structure
    • CRF+CNN
    • Structural-RNN
    • Quaternions
    • Exponential maps

  • Emergence of attributes and parts in Object-based representations
  • Emergence of parts in Scene representations
    • Minimal Image.
    • Empirical Receptive Field
Amir Zamir
4 10/16/2017 Losses

Student Paper Presentation
  • Lnorm
  • Metric Learning
  • Perceptual loss
  • Intro to GAN
    • Pitfalls (e.g. mode collapse and training). Remedies.
    • Wasserstein GAN. Energy based GAN.
    • Conditional GAN
    • Generative Latent Optimization
    • Cycle consistency and transport loss.
    • GAN Applications
  • Metric Learning

  • Architectures
  • Losses
Amir Zamir

10/18/2017 Project Proposal Due
(11:59 PM)
5 10/23/2017 2D and 3D Object and scene representations [Silvio's slides]

Student Paper Presentations
  • Basic principles for designing a good representation for object recognition
  • Why are 3D representations useful for object understanding?
  • Overview of classic and more recent methods for 2D and 3D object detection and classification
  • 2D & 3D Scene Understanding
  • Why is a 3D representation useful for scene understanding?
  • Relating objects and space in the 3D physical world
  • From objects to activity understanding in the 3D physical world
  • Datasets

  • Visualizing Representations:
    • Representation inversion methods. (E.g. Fergus’s 16)
    • Disentangling representations (Reed’14)
    • Elements probing methods. (Bengio 17)
    • interpretability methods.

Neuron domain:
Image Domain: CS231N slides
Silvio Savarese

6 10/30/2017 Domain Adaptation

Student Paper Presentations
  • Classic domain adaptation
  • Domain confusion loss
  • End-to-end learned domain adaptation
  • Nonparametric domain adaptation
  • GAN based domain adaptation

  • Domain Adaptation by learning joint or invariant features

    Domain adaptation by aligning domains

    Also a good paper
    Ozan Sener

    7 11/06/2017 Broad perception representations
    Self supervised learning

    Student Paper Presentations
  • Self supervision
  • Supersized self-supervision.
  • Broad perception
  • Multitask. Ubernet.
  • Curiosity
  • (unpublic material)

  • Learning to learn

    Memory Mechanisms and Lifelong Learning

    Self-supervised learning:
    Amir Zamir

    8 11/13/2017 Representations in the brain
    Memory-based representation and inference

    Student Paper Presentation [Slides]
  • Details TBD

  • Hebb’s postulate revisited

    Meta-discussion and things inspired by the brain

    Using Models to Model the Brain

    Representations in the brain
    Daniel Yamins

    11/17/2017 Project Progress Report Due
    (11:59 PM)
    11/20/2017 No Class.
    9 11/27/2017 From representation to actuation: active perception

    Student Paper Presentation
  • Active vision.
  • Reinforcement learning
  • Action Prediction based frameworks.
  • Curiosity
  • Prediction and probability based reps.
  • Intuitive physics
  • Video frame prediction

  • Learning by Being Active

    Learning by Being Active

    Amir Zamir

    10 12/4/2017 Unsupervised learning of representations

    Student Paper Presentation
  • Clustering
  • Dimensionality reduction
  • Sparse coding
  • Generative unsupervised learning
    • VAE, BiGAN, RBM
  • Discriminative unsupervised learning
    • NAT
    • Dosovitskiy 2016

  • Generative Models


    Amir Zamir

    11 12/8/2017 Final Presentations Students
    12 12/11/2017 Final Presentations Students
    12/14/2016 Final Project Report Due
    (11:59 PM)