AML 21.0104

Advanced Machine Learning

Models and Methods

PhD level coursework in Advanced ML

Instructor

Dr Ram Prasad K, PhD

Director, VisionCog R&D

Course Highlight

Emphasis on both classical/statistical and deep learning models and methods
Training to learn advanced concepts directly from research papers and articles
Implement and understand state of the art ML models and methods using popular tools/libraries

Content Highlight

SVM, Random Forests, Boosting, Clustering, Projections, Manifolds, Bayesian Learning
CNN, RNN, GNN, GAN, CapsNet, Autoencoders, Attentions, Transformers, BERT & GPT variants
Hyperparameter tuning, ML optimizations, CV, NLP, Time series and Graph structure applications

Tools Highlight

Numpy, Matplotlib, Pandas, Scikit-Learn
TensorFlow, PyTorch, OpenCV, Scikit-Image
Spacy, Gensim, HuggingFace, Google Colaboratory environment

Duration

4 months; 40 classes
Timings: 6:30 PM - 8:30 PM

Course Schedule

AML 21.0104

Day	Date	Description	Materials and Reading Assignments
Table will be updated frequently until Day 40+
Day 01	16-01-2021 Saturday	Course Logistics Preparing for research works Introduction to LaTeX Scribe: Ram Prasad K	Research preparations LaTeX typesetting Scribe notes preparation GitHub repository Other resources to follow for latest research works in ML Scientific Practice LaTeX typesetting tutorial Lecture materials will be shared with participants via private link to download
Day 02	19-01-2021 Tuesday	Scientific Computing Environment Google Colab Scientific Python Scribe: Ram Prasad K Summary: Ram Prasad K	Scientific Python Python fundamentals Google Colab tutorial Reading Assignments Chap-2 of Python Data Science Handbook Chap-4 of Python Data Science Handbook Stanford tutorial on Python, Numpy and Matplotlib (CS231n)
Day 03	21-01-2021 Thursday	Scientific Python libaries Scribe: Ram Prasad K Summary: Ram Prasad K	Scientific Python Numpy Matplotlib Reading Assignments Chap-2 of Python Data Science Handbook (Numpy) Chap-4 of Python Data Science Handbook (Matplotlib) Stanford tutorial on Python, Numpy and Matplotlib (CS231n)
Day 04	23-01-2021 Saturday	Data Analytics library Scribe: Ram Prasad K Summary: Ram Prasad K	Scientific Python Pandas Reading Assignments Chap-3 of Python Data Science Handbook (Pandas)
Day 05	02-02-2021 Tuesday	Machine Learning - Insights Scribe: Ram Prasad K Summary: Ram Prasad K	ML Insights Types of Machine Learning methods Parameter-transformative and Geometric insights
Day 06	04-02-2021 Thursday	Machine Learning practices Multivariate regression analysis example implemntation Scribe: Ram Prasad K Summary: Ram Prasad K	Implementation Scikit-Learn Reading Assignments Section 5.1 of Deep Learning Textbook (pg: 96-108) Section 2.1 of Introduction to Statistical Learning (pg: 15-29) Section 3.1 of Dive into Deep Learning
Day 07	06-02-2021 Saturday	Multivariate regression analysis example implementation Scribe: Ram Prasad K Summary: Ram Prasad K	Implementations TensorFlow PyTorch Reading Assignments Section 5.1 of Deep Learning Textbook (pg: 96-108) Section 2.1 of Introduction to Statistical Learning (pg: 15-29) Section 3.1 of Dive into Deep Learning
Day 08	08-02-2021 Monday	Linear model for multivariate regression analysis Deep Neural Networks model for multivariate regression analysis Scribe: Ram Prasad K Summary: Ram Prasad K	Methods Vector representation Error-function Analytical vs numerical solutions R^2 ANOVA test
Day 09	10-02-2021 Wednesday	Mathematical background for Gradient Descent Scribe: All Summary: Ram Prasad K	Methods First-order differential calculus Derivatives, Gradients Jacobian and Hessian matrix Reading Assignments Section 4.3: Gradient based optimization; Deep Learning Textbook (pg: 80-91) Section 11.1: Optimization and Deep Learning; Dive into Deep Learning Section 11.3. Gradient Descent; Dive into Deep Learning
Day 10	13-02-2021 Saturday	Types of Gradient Descent Scribe: All Summary: Ram Prasad K	Methods Full-batch Gradient Descent Stochastic Gradient Descent Mini-batch Gradient Descent Implementation Using Numpy Using Sklearn SGD-class Reading Assignments Section 5.9: Stochastic Gradient Descent; Deep Learning Textbook (pg: 149-150) Section 11.4. Stochastic Gradient Descent; Dive into Deep Learning Section 11.5. Minibatch Stochastic Gradient Descent; Dive into Deep Learning Module 1.5. Stochastic Gradient Descent; Scikit-Learn Documentation
Day 11	16-02-2021 Tuesday	DeepNets and Backpropagation-Part1 Scribe: All Summary: Ram Prasad K	Methods DeepNets Classification Backpropagation - Automatic Differentiation Manual Differentiation Newton's difference quotient Reverse-mode Autodiff Reading Assignments Chapter 2 - How the backpropagation algorithm works; Neural Networks and Deep Learning by Michael Nielsen
Day 12	20-02-2021 Saturday	Binary and multi-class Classificaiton Summary: Ram Prasad K	Classification Logistic and Softmax Classification DNN based classification Tabular data classification Reading Assignments Section 6.2.2.2: Sigmoid Units for Bernoulli Output Distributions; Deep Learning Textbook (pg: 178-180) Section 6.2.2.3: Softmax Units for Multinoulli Output Distributions; Deep Learning Textbook (pg: 180-184)
Day 13	23-02-2021 Tuesday	Multi-class Classificaiton Model evaluation - Confusion Matrix Summary: Ram Prasad K	Methods Precision Recall F1-Score Reading Assignments Logistic and Softmax Regression; Scikit-Learn documentation
Day 14	25-02-2021 Thursday	TensorFlow and Scikit-Learn implementation for multi-class classificaiton Summary: Ram Prasad K	Reading Assignments Confusion Matrix; Scikit-Learn documentation Precision, Recall and F-measure for multi-class ; Scikit-Learn documentation
Day 15	27-02-2021 Saturday	PyTorch implementation for multi-class classificaiton Summary: Ram Prasad K	Reading Assignments Cross-Entropy Loss; PyTorch documentation Section 3.4.6.1: Negative Log-likelihood; Dive into Deep Learning
Day 16	02-03-2021 Tuesday	Introduction to Natural Language Processing (NLP) Summary: Ram Prasad K	Methods Count Vectorizer Text tokenization and TFIDF features Spam classification - example Libraries Scikit-Learn Reading Assignments Bag-of-Words representation; Scikit-Learn documentation TF-IDF representation; Scikit-Learn documentation
Day 17	04-03-2021 Thursday	spaCy library for text preprocessing Summary: Ram Prasad K	Libraries Scikit-Learn spaCy Reading Assignments spaCy 101: Everything you need to know;
Day 18	09-03-2021 Tuesday	Deep Learning optimizations Vanishing/Exploding gradient issues Summary: Ms Shilpa Abraham (Gradient Descent methods) Dr Sunitha (Backpropagation)	Methods Activations and initializations Batch-Normalization Dropouts Reading Assignments Understanding the difficulty of training deep feedforward neural networks; Xavier Glorot, Yoshua Bengio
Day 19	13-03-2021 Saturday	Recurrent Neural Networks Backpropagation Through Time (BPTT) Summary: Ram Prasad K	Methods Simple RNN Long Short-Term Memory (LSTM) Gated Recurrent Unit (GRU) Reading Assignments Recurrent Neural Networks; Dive into Deep Learning
Day 20	16-03-2021 Tuesday	Time Series Prediction using various RNN Summary: Ram Prasad K	Methods Temperature prediction SimpleRNN/LSTM/GRU Reading Assignments https://www.tensorflow.org/tutorials/structured_data/time_series; TensorFlow tutorial on Time Series Forecasting
Day 21	20-03-2021 Saturday	Regularization techniques Challenges with activation functions (Sigmod vs ReLU) Scribe: TBA Summary: TBA	Regularization Methods L1 (LASSO) L2 (Ridge) Elastic-Net Dropout Reading Assignments Section 4.4. Model Selection, Underfitting, and Overfitting; Dive into Deep Learning Modeling one neuron; Commonly used activation functions; CS231n Convolutional Neural Networks for Visual Recognition
Day 22	24-03-2021 Tuesday	SimpleRNN Manual implementation for forward pass Summary: Mr Akhil (Regularization and activation functions)	Implementation Compare the outputs of recurrent hidden layer from Keras model with manually coded forward pass Reading Assignments The Unreasonable Effectiveness of Recurrent Neural Networks; Andrej Karpathy
Day 23	25-03-2021 Thursday	Word embeddings Sentiment Analysis Summary: Ms Reshma (Stanford CS231n; MLP case study)	Methods Visualize word embeddings Reading Assignments Section 9.4.2. Bidirectional Model; Dive into Deep Learning Bidirectional LSTM; Methods from Papers with Code
Day 24	30-03-2021 Tuesday	Bidirectional RNN Sequence to Sequence model Summary: Ram Prasad K	Methods Bidirectional versions of SimpleRNN, LSTM, GRU Encoder-Decoder architecture for Neural Machine Translation (NMT) Reading Materials Text classification using Bidrectional RNN; TensorFlow tutorial Encoder-Decoder for NMT; PyTorch tutorial
Day 25	01-04-2021 Thursday	Language translation using Seq2Seq model Summary: Ram Prasad K	Methods Encoder and Decoder with GRU Reading Assignments NLP From Scratch: Translation with a Sequence to Sequence Network and Attention; PyTorch tutorial
Day 26	07-04-2021 Wednesday	Preparing test data for language translation RNN Language model (English to French) Summary: Ram Prasad K	Methods Training and evaluating language models Teacher-forcing and autoregressive mode Reading Assignments NLP From Scratch: Translation with a Sequence to Sequence Network and Attention; PyTorch tutorial
Day 27	10-04-2021 Saturday	Attention Mechanism in Encoder-Decoder RNN Summary: Ram Prasad K	Methods Bahdanau attention model PyTorch implementation Reading Assignments Neural Machine Translation by Jointly Learning to Align and Translate; Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
Day 28	14-04-2021 Wednesday	Transformers (Theory and intuitions) Summary: Ram Prasad K	Reading Assignments Attention Is All You Need; Ashish Vaswani, Noam Shazeer et al
Day 29	17-04-2021 Saturday	Transformers - Implementation Summary: Ram Prasad K	Reading Assignments PyTorch implementation of Transformer for Seq2Seq Language translation. Encoder and Decoder self-attention
Day 30	21-04-2021 Wednesday	Transformer variants GPT; BERT; Vision Transformer Summary: Ram Prasad K	Reading Assignments Refer to reading materials suggested in the shared slides;
Day 31	24-04-2021 Saturday	Convolutional Neural Networks (CNN) Summary: Ram Prasad K	Reading Assignments Convolutional Neural Networks (CNNs / ConvNets); Stanford CS231n: Convolutional Neural Networks for Visual Recognition
Day 32	28-04-2021 Wednesday	Data Augmentation Skip Connections Visualize pretrained ConvNets Summary: Ram Prasad K	Reading Assignments Data augmentation; TensorFlow Tutorials Keras Pretrained ConvNets; Keras Applications
Day 33	01-05-2021 Saturday	ResNet Transfer Learning Fine Tuning Summary: Ram Prasad K	Reading Assignments Transfer Learning and Fine Tuning; CS231n: Convolutional Neural Networks for Visual Recognition
Day 34	05-05-2021 Wednesday	Support Vector Machines - Linear SVM Soft/Hard Max Margin Classifier Summary: Ram Prasad K	Reading Assignments Section 1.4; Support Vector Machines; Scikit-Learn Tutorial Chapter 9; Support Vector Machines; Section 9.1 Max Margin Classifier; Introduction to Statistical Learning
Day 35	08-05-2021 Saturday	Kernel SVM Radial Basis Function (RBF) Kernel Summary: Ram Prasad K	Reading Assignments Section 1.4.6; Support Vector Machines - Kernel Functions; Scikit-Learn Tutorial Chapter 9; Support Vector Machines; Section 9.3 (pages 349-354); Introduction to Statistical Learning
Day 36	12-05-2021 Wednesday	Decision Trees Ensemble Learning (Bagging - Random Forest) Summary: Ram Prasad K	Reading Assignments Section 1.10; Decision Trees; Scikit-Learn Tutorial Chapter 8; Tree-Based Methods; Section 8.1 (pages 303-316); Introduction to Statistical Learning
Day 37	20-05-2021 Thursday	Boosting - AdaBoost; Gradient Boost Summary: Ram Prasad K	Reading Assignments Section 1.11. Ensemble methods; Scikit-Learn Tutorial Chapter 8; Tree-Based Methods; Section 8.2 Bagging, Random Forests, Boosting; Introduction to Statistical Learning
Day 38	21-05-2021 Friday	Hyperparameter Tuning Grid Search Random Search Dimensionality Reduction - PCA Summary: Ram Prasad K	Methods Sklearn - GridsearchCV and RandomizedSearchCV Principal Component Analysis Reading Assignments Section 3.2. Tuning the hyper-parameters of an estimator ; Scikit-Learn Tutorial
Day 39	25-05-2021 Tuesday	Autoencoders Summary: Ram Prasad K	Methods Simple Autoencoders Stacked Autoencoders Convolutional Autoencoders
Day 40	27-05-2021 Thursday	t-SNE Running Github projects in Google Colab with GPU/TPU Summary: Ram Prasad K	Methods PCA vs tSNE using Scikit Learn Reading Assignments Neural Style Transfer; Github code implementation
Day 41	29-05-2021 Saturday	Hyperparamter tuning for Deep Learning models Huggingface - Transfer Learning for NLP Summary: Ram Prasad K	Methods Keras-Tuner / Ray-Tune Hyperband Bayesian Optimization Huggingface DistillBERT Reading Assignments Huggingface Transformers; Official Documentation Huggingface Transfer-Learning and Fine-tuning; Official Documentation
Day 42	05-06-2021 Saturday	Clustering Prototype and Density based clustering Summary: Ram Prasad K	Methods K-Means / K-Means++ DBSCAN
Table will be updated frequently until Day 40+
Notes Lecture materials will be shared with participants via private link to download Feel free to suggest topics or research papers you want to include in the lectures

Reference Materials

Classical Machine Learning

Python Data Science Handbook

Jake VanderPlas

Introduction to Statistical Learning

Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani

Probabilistic Machine Learning: An Introduction

Kevin Patrick Murphy

Deep Learning

Neural Networks and Deep Learning

Michael Nielsen

Deep Learning

Ian Goodfellow and Yoshua Bengio and Aaron Courville

Dive into Deep Learning

Aston Zhang, Zack C. Lipton et al

CS231n: Convolutional Neural Networks for Visual Recognition

Fei-Fei Li, Ranjay Krishna, Danfei Xu

Deep Learning with PyTorch: Build, Train, and Tune Neural Networks Using Python Tools

Eli Stevens, Luca Antiga, and Thomas Viehmann

Library Tutorials

Scikit-Learn Tutorial

TensorFlow Tutorial

Keras Tutorial

PyTorch Tutorial

Paper Discussions

AML 21.0104

Tentative list of papers to be discussed in the course (more papers to be added)

Visual Data

LambdaNetworks: Modeling long-range Interactions without Attention
International Conference on Learning Representations, ICLR (2021)
Transformers in Vision: A Survey
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, Mubarak Shah;
arXiv preprint arXiv:2101.01169 (2021)
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou;
arXiv preprint arXiv:2012.14214 (2020)

Sequential Data

CTRLsum: Towards Generic Controllable Text Summarization
Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, Caiming Xiong;
International Conference on Learning Representations, ICLR (2021)
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts et al
arXiv preprint arXiv:1910.10683 (2019)
Attention Is All You Need
Vaswani, Ashish, et al.
Conference on Neural Information Processing Systems (2017)

Graph Data

A Generalization of Transformer Networks to Graphs
Vijay Prakash Dwivedi, Xavier Bresson;
AAAI 2021 Workshop on Deep Learning on Graphs: Methods and Applications (DLG-AAAI 2021)

General

Underspecification Presents Challenges for Credibility in Modern Machine Learning
Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi et al;
arXiv preprint arXiv:2011.03395 (2020)
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot, Yoshua Bengio;
JMLR Workshop and Conference Proceedings 9:249-256 (2010)

Participants

AML 21.0104

Dr Sunitha E V

Research interest: Machine Learning methods and algorithms

Ms Christy Jacqueline

Research interest: Graph structured data, Optimization algorithms

Ms Shilpa Abraham

Research interest: Computer Vision, Deep Learning

Ms Diana Mathew

Research interest: Computer Vision, Bio-informatics

Ms Meharuniza Nazeem

Research interest: Video analytics, Deep Learning

Ms Rani Koshy

Research interest: Graph structured data, Text analysis

Ms Shwetha Jayant

Research interest: Image Processing, Deep Learning

Ms Sandhya Ramakrishnan

Research interest: Text analysis and summarization

Ms Reshma Sheik

Research interest: Natural Language Processing

Mr Akhil P V

Research interest: Machine Learning, Recommender Systems, Sentiment Analysis

About Instructor

Dr Ram Prasad K, PhD

Ram Prasad received his Ph.D degree in Computer Science Engineering from Universidad Autónoma de Madrid, Spain, and pursued his postdoctoral research work at Dublin City University, Ireland. His doctoral and postdoctoral research works were funded by the prestigious European Commission's Marie Curie Fellowship. He was also a visiting researcher at University of Twente, Netherlands and University of Halmstad, Sweden. He graduated in Computer Science from Chennai Mathematical Institute, India. His expertise is in the areas of classical machine learning, deep learning, computer vision, biometrics, software developments and algorithm design. He is the founder and director of VisionCog Research and Development Private Limited, India and manages commercial R&D works in the domain of computer vision and biometrics.

Resume

Download: RamKrish.pdf

Contact

email: ram.krish@visioncog.com