[ Notebook (Soln.) ]

Publication: Liu et al. KAN: Kolmogorov-Arnold Networks. ArXiv, 2024
Task: Building an understanding of Kolmogorov-Arnold Networks (KAN) using B-splines as activation functions through practical coding examples and theoretical insights.
Libraries: PyTorch, NumPy, Matplotlib, Jupyter
Series Overview: This tutorial series guides you through the complexities of KAN by dissecting its core components and demonstrating their application through practical examples. Each notebook in the series focuses on a different aspect of KANs:

• B-Splines for KAN - Introduces B-splines and their role in KANs.
• Deeper KANs - Explores network construction and backpropagation in deeper KAN architectures.
• Grid Manipulation in KANs - Discusses model capacity expansion and continual learning without catastrophic forgetting.
• Symbolic Regression using KANs - Teaches integration of symbolic regression within KAN networks.

  [ Notebook (Soln.) ]

Publication: Acerbi et al. Large Language Models Show Human-like Content Biases in Transmission Chain Experiments. PNAS, 2023
Task: Reproducing and understanding the results regarding content biases in LLMs as observed in transmission chain experiments
Libraries: openai, rpy2, python-docx, matplotlib, pandas
Series Overview: This series includes multiple detailed notebooks, guiding you through the process of setting up, executing, and analyzing transmission chain experiments with LLMs to explore human-like content biases. Each notebook corresponds to one of the six plots in the main paper and includes three sections:

• Authors’ Results - Reproduces the paper’s findings using Python (originally conducted in R).
• Authors’ Summaries / GPT-4 Evaluation - Evaluates summaries from the original study using GPT-4.
• New Summaries / GPT-4 Evaluation - Uses new summaries from updated experiments and evaluates them using GPT-4.

  [ Notebook (Soln.) ]

Publication: Park et al. Generative Agents: Interactice Simulacra of Human Behavior. UIST, 2023
Task: Creating and understanding generative agents proposed in Simulacra
Libraries: openai
Series Overview: This tutorial series spans four comprehensive notebooks, each designed to progressively build your understanding and skills in simulating sociological phenomena with generative agents. Check out the blog post where I break down the big ideas behind creating sociological simulations in an easy-to-get way.

• Notebook 1.1 - Basic Simulation Structure: An introduction to the essential components of simulation, including environment interaction and the construction of a basic simulation loop.
• Notebook 1.2 - Building Generative Agents: A deep dive into the creation of generative agents, with emphasis on memory structures derived from the Simulacra paper.
• Notebook 1.3 - Agent Schedule Planning: Techniques for developing agents’ abilities to autonomously plan their schedules.
• Notebook 1.4 - Agent Cognition and Communication: Exploration of agents’ cognitive functions, such as perception and memory retrieval, and how they interact and communicate.

  [ Notebook (Soln.) ]

Publication: Müller et al. Transformers can do Bayesian Inference. ICLR, 2022
Task: Using Transformers to estimate Posterior Predictive Distributions
Libraries: PyTorch
Learning objectives:

• Understand the principles of Prior-Data Fitted Networks (PFNs) and their application in integrating prior knowledge to predict the Posterior Predictive Distribution (PPD) in machine learning models.
• Acquire practical skills in defining priors, creating dataset loaders for synthetic data generation, developing transformer models for PPD approximation, formulating loss functions for specific regression tasks, and evaluating model output quality.

  [ Notebook (Soln.) ]

Publication: Müller et al. Transformers can do Bayesian Inference. ICLR, 2022
Hollmann et al. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. ICLR, 2023
Task: Generating data according to different priors for PFNs
Libraries: PyTorch
Learning objectives: Learn how to generate synthetic data based on specified priors and utilize it for training neural networks to approximate Bayesian inference.

  [ Notebook (DIY) ] [ Notebook (Soln.) ]

Publication: Gruver et al. Large Language Models are Zero Shot Time Series Forecasters. NeurIPS, 2023
Task: Weather forecasting using LLMs
Libraries: openai, tiktoken, jax

Learning objectives: Explore zero-shot prompting with Large Language Models (LLMs) for time series forecasting. In this tutorial, we aim to:

• Acquaint you with the application of machine learning techniques using Large Language Models (LLMs).
• Enhance your understanding of LLMs and the parameters influencing their behavior.
• Guide you through the essentials for successful time series prediction with LLMs.
• Translate knowledge from transformers to the realm of LLMs.

  [ Notebook (DIY) ] [ Notebook (Soln.) ]

Learning objectives: Delve into the inner workings of transformer models beyond basic applications
Key Areas:

• Adversarial Inputs: Crafting inputs to challenge language models
• Attention Visualization: Understanding focus mechanisms in transformers
• Fine-Tuning with LoRA: Implementing Low-Rank Adaptation for model refinement
• Bias Detection: Investigating biases in model responses

Note: This tutorial is an introductory exploration of transformers. It’s a starting point for more advanced study.

  [ Notebook (Soln.) ]

Publication: Maziarka, et al. "Molecule Attention Transformer"
Task: Classification task to predict Blood-brain barrier permeability (BBBP)
Dataset: BBBP
Libraries: PyTorch, DeepChem, RDKit

Learning objectives:

• Learn key concepts required to work with molecules
• Perform critical data preprocessing tasks, such as feature extraction, graph formation, and scaffold splitting
• Explore challenges of drug discovery, particularly designing drugs that can cross the blood-brain barrier and enter the central nervous system
• Implement Molecule Attention Transformer (MAT) proposed by Maziarka et al. (2020) using a deep learning pipeline
• Train and evaluate the model on molecular datasets

  [ Notebook (DIY) ]

DeepNote: (Jupyter notebook hosting service) DIY Notebooks.
Github: Repository to work on your local machine.
Programming Language: Python
Libraries: Pandas, feedparser, newscatcherapi, psaw, requests, twarc (Twitter API), requests-html
Learning objectives:

• Use Python to collect research data from the social web
• Give due consideration to the ethics of data collection
• Following topics are covered:
• Accessing RSS feeds
• Accessing Reddit and Wikipedia through APIs
• Accessing Twitter using Twitter API
• Web crawling

  [ Notebook (DIY) ] [ Notebook (Soln.) ]

Publication: Vaswani, Ashish, et al. "Attention is all you need." NeurIPS 2017.
Task: Neural Machine Translation (e.g, German-English)
Dataset: Multi30k
Libraries: PyTorch, NLTK, Spacy, torchtext

Learning objectives:

• Build a transformer model for neural machine translation
• Train the model using proposed label smoothing loss and learning rate scheduler
• Use the trained model to infer likely translations using
• Greedy Decoding
• Beam Search

Topics

• Attention: Applications (2014-2017) [Notebook]
• Transformers (2017-2022) [Notebook]
• LLMs (2020-2023) [Notebook]

Practical

Tutorial on Attention is all you need (Vaswani et al. 2017).

Topics [ Slides/Notebook ]

• History of vision architectures (1998-2012)
• Network in Network (2014)
• InceptionNet (2014)
• ResNet (2015)
• Pre-activations: Improved ResNet (2016)
• DenseNet (2016)
• WaveNet (2016)
• Depthwise Separable Convolutions (2017)
• Squeeze-and-Excite: ResNet (2017)
• Neural Architecture Search (2018 - Present)

Practical [ Notebook (DIY) ] [ Notebook (Soln.) ]

Task: Build an image classifier
Dataset: CIFAR-10
Libraries: PyTorch, torchvision

Learning objectives:

• Build and train InceptionNet on CIFAR-10
• Build and train DenseNet on CIFAR-10

Topics [ Slides/Notebook ]

• Motivation
• Network Pruning: Pipeline
• Network Pruning: Unstructured vs Structured
• Network Pruning: Criterion
• Network Pruning: Prune Rate
• Network Pruning: Lottery Ticket Hypothesis
• Knowledge Distillation
• Quantization

Practical [ Notebook (DIY) ] [ Notebook (Soln.) ]

Task: Build a land use classifier
Dataset: EuroSat (RGB)
Libraries: PyTorch, torchvision

Learning objectives:

• Build and train a simple classifier
• Use different techniques of pruning available in PyTorch to reduce the size of the network
• Observe the effect of pruning on model accuracy

Topics [ Slides/Notebook ]

• Autoencoders: Motivation & History
• Autoencoders: Loss function
• Undercomplete Autoencoders
• Overcomplete Autoencoders
• Stacked / Deep Autoencoders
• Variational Autoencoders
• Applications of Autoencoders

Practical [ Notebook (DIY) ] [ Notebook (Soln.) ]

Task: Explore the use of autoencoders for

• Data compression
• Data generation
• Data interpolation

Dataset: MNIST
Libraries: PyTorch, torchvision

Learning objectives:

• Build and train the following types of autoencoders
• Sparse autoencoder with L1 penalty
• Sparse autoencoder with KL penalty
• Contractive Autoenoder
• Variational Autoenoder
• Explore the quality of their learned latent space

Topics [ Slides/Notebook ]

• Sources of noise
• Impact of noise
• Examples of noise
• Noise reduction techniques
• Rolling Window
• Convolution
• Digital Filters
• Machine Learning Fitlers
• Denoising Autoencoders

Practical [ Notebook (DIY) ] [ Notebook (Soln.) ]

Task: Denoise corrupted images
Dataset: Fashion MNIST
Libraries: PyTorch, torchvision

Learning objectives:

• Build a simple Denoising Autoencoder (DAE) for images
• Train such a DAE on images with a given corrupted noise
• Visualize the quality of reconstruction

  [ Notebook (DIY) ] [ Notebook (Soln.) ]

Based on: Zoabi et al. "Machine learning-based prediction of COVID-19 diagnosis based on symptoms." npj digital medicine 4.1 (2021): 1-5.
Task: Predict COVID-19 infection from reported symptoms
Dataset: English translation of COVID infections reported by Israeli Ministry of Health
Learning objectives:

• Explore a realistic dataset and prepare it for building a practical machine learning system
• Think through various practical issues related to deploying such a system (e.g., class imbalance, data collection bias)
• Build and train such an ML system addressing the issues discovered above