| layout | title | subtitle |
|---|---|---|
page |
Intro to ML Resources |
(Feel free to add more links here)
The following are materials from a Deep Learning Hackathon at the University of Arizona organized by Ansh Gupta (anshrg[at]utexas[dot]edu)
Dr. CK Chan gave a short presentation on the importance of deep learning in science. Jackson then gave a presentation introducing fundamentals of machine learning. Basic concepts covered include the linear algebra foundation of ML, gradient descent, and optimization problems. Ansh ran a hands-on activity about handwritten digit recognition using a basic neutral network and CNN model trained on the MNIST dataset.
- Presentation on intro to ML by Jackson: Math_Lecture_Astro_Workshop.pdf
- Hands-on activity on ML basics by Ansh: UArizona Deep Learning Hackathon Intro.ipynb
- Zoom recording: link
- Momentum gradient descent: Stochastic Gradient Descent with Momentum in Python
- Neural networks (playlist): But what is a neural network? Chapter 1, Deep learning
- Explanation of convolution: But what is a convolution?
Jackson finished up their presentation on the fundamentals of machine learning, discussing common terminology and explaining the differences between models. Ansh gave a presentation on how neural networks learn by visually showing the backpropagation procedure on a neural network. Ansh then ran a hands-on session in which participants implemented autodifferentiation, a tool which is used to carry out backpropagation, using dual numbers. Dr. CK Chan wrote the code for that hands-on activity.
- Presentation on intro to ML by Jackson (continued): Math_Lecture_Astro_Workshop.pdf
- Presentation on backpropagation fundamentals by Ansh: ML Hackathon Backpropagation
- Hands-on activity on auto differentiation by Ansh (primarily written by CK): ad-gd.ipynb
- Solutions to hands-on activity: ad-gd-solutions.ipynb
- Zoom recording: link
- Paper on dataset splitting: https://onlinelibrary.wiley.com/doi/full/10.1002/sam.11583
- Weight dropping/ML speedup: https://github.com/google-research/jaxpruner
- ML video series: A Gentle Introduction to Machine Learning
- Hands-on bayesian neural networks: https://arxiv.org/pdf/2007.06823.pdf
- Tinker with neural network in browser: https://playground.tensorflow.org/
- Tinker with neural network in browser: neurovis.mitchcrowe.com/
Dr. CK Chan started by answering questions from last week. Ansh then ran a hands-on session in which participants trained a CNN to classify images of galaxies from the galaxy10 dataset. Participants were able to code based on their level of expertise, with hints available in the notebook ranging from small suggestions to nearly full solutions.
- Hands-on activity on galaxy classification using a CNN by Ansh: CNN Galaxy Classifier.ipynb
- Zoom recording: [link needed]
- UA HPC workshops: https://uarizona.atlassian.net/wiki/spaces/UAHPC/pages/75989137/Training
- NVIDIA deep learning training workshops: https://www.nvidia.com/en-us/training/
Dr. CK Chan led a discussion session to help participants talk through any questions from last week. Topics include Deep Double Descent (1, 2, 3). Participants also started thinking about ideas for the coding competition which will occur later in the hackathon.
The group started by sharing project ideas. Dr. CK Chan then led an introduction to recurrent neural network and transformer architecture. Participants then tune a pre-trained transformer to perform semantic segmentation for galaxies.
- Hands-on activity on semantic segmentation of galaxy classification using a pre-trained transformer https://github.com/rndsrc/dl/blob/main/segformer.ipynb
- Zoom recording: link
- RNN: https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-1/
- Original Transformer Paper: https://arxiv.org/abs/1706.03762
- Vision Transformer: https://arxiv.org/abs/2010.11929
- A Survey of Large Language Models: https://arxiv.org/abs/2303.18223
- Transformers example: https://github.com/huggingface/transformers/tree/main/examples/
CK showed some open source and pre-trained models on hugging face. Ansh gave an introduction to diffusion models and presented his work in another hackathon. Participants were encouraged to team up and try out their project ideas.
- Presentation introducing diffusion models fundamentals: Introduction to Diffusion Models
- Applying diffusion models to denoise galaxy images: Deblurring Galaxies w/ Diffusion Models
- Zoom recording: link
- Hugging Face (Datasets): https://huggingface.co/
- Kaggle (More Datasets): https://www.kaggle.com/
- Example diffusion model problem with lots of useful papers, resources, and more: https://github.com/Ciela-Institute/Astromatic/blob/main/Astromatic2023/p5_deconvolution.ipynb
- Python package with diffusion model functions: https://pypi.org/project/score-models/
- GitHub repository for above: https://github.com/AlexandreAdam/score_models/tree/master
- Excellent video on stats/math behind diffusion models: What are Diffusion Models?
CyVerse researcher trained participants on using CACAO https://cacao.jetstream-cloud.org/
- Zoom recording: link
- CACAO documentation: https://docs.jetstream-cloud.org/ui/cacao/intro/
- List of ACCESS resources: https://allocations.access-ci.org/resources
In small groups, we refined project ideas and did troubleshooting with CACAO https://cacao.jetstream-cloud.org/. We decided on a final project presentation date of Friday, April 26. Participants will focus on defining their projects in more details, refining their understanding of the necessary ML concepts, starting some preliminary tests, and presenting their process by the end of the semester. Attendees will pursue full implementation of their models and work on obtaining final results in the upcoming semester(s).
- Zoom recording: link