Schedulers

README.md

@@ -1,16 +1,3 @@
-<!-- START doctoc generated TOC please keep comment here to allow auto update -->
-<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-**Table of Contents**
-
-- [Pytorch Image Models (timm)](#pytorch-image-models-timm)
-  - [Install](#install)
-  - [How to use](#how-to-use)
-    - [Create a model](#create-a-model)
-    - [List Models with Pretrained Weights](#list-models-with-pretrained-weights)
-    - [Search for model architectures by Wildcard](#search-for-model-architectures-by-wildcard)
-
-<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-
 # Pytorch Image Models (timm)
 > `timm` is a deep-learning library created by <a href='https://twitter.com/wightmanr'>Ross Wightman</a> and is a collection of SOTA computer vision models, layers, utilities, optimizers, schedulers, data-loaders, augmentations and also training/validating scripts with ability to reproduce ImageNet training results. 
 
@@ -32,7 +19,7 @@ cd pytorch-image-models && pip install -e .
 
 ### Create a model
 
-```python
+```
 import timm 
 import torch
 
@@ -45,7 +32,7 @@ It is that simple to create a model using `timm`. The `create_model` function is
 
 To create a pretrained model, simply pass in `pretrained=True`.
 
-```python
+```
 pretrained_resnet_34 = timm.create_model('resnet34', pretrained=True)
 ```
 
@@ -54,7 +41,7 @@ pretrained_resnet_34 = timm.create_model('resnet34', pretrained=True)
 
 To create a model with a custom number of classes, simply pass in `num_classes=<number_of_classes>`.
 
-```python
+```
 import timm 
 import torch
 
@@ -75,7 +62,7 @@ model(x).shape
 
 `timm.list_models()` returns a complete list of available models in `timm`. To have a look at a complete list of pretrained models, pass in `pretrained=True` in `list_models`.
 
-```python
+```
 avail_pretrained_models = timm.list_models(pretrained=True)
 len(avail_pretrained_models), avail_pretrained_models[:5]
 ```
@@ -98,7 +85,7 @@ There are a total of **271** models with pretrained weights currently available
 
 It is also possible to search for model architectures using Wildcard as below:
 
-```python
+```
 all_densenet_models = timm.list_models('*densenet*')
 all_densenet_models
 ```

docs/RandAugment.html

@@ -47,7 +47,7 @@ <h2 id="Training-models-with-RandAugment-using-timm's-training-script">Training
 <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
 <div class="text_cell_render border-box-sizing rendered_html">
 <p>To train your models using <code>randaugment</code>, simply pass the <code>--aa</code> argument to the training script with a value. Something like:</p>
-<div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">train</span><span class="o">.</span><span class="n">py</span> <span class="o">../</span><span class="n">imagenette2</span><span class="o">-</span><span class="mi">320</span> <span class="o">--</span><span class="n">aa</span> <span class="n">rand</span><span class="o">-</span><span class="n">m9</span><span class="o">-</span><span class="n">mstd0</span><span class="o">.</span><span class="mi">5</span>
+<div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">train</span><span class="o">.</span><span class="n">py</span> <span class="o">../</span><span class="n">imagenette2</span><span class="o">-</span><span class="mi">320</span> <span class="o">--</span><span class="n">aa</span> <span class="n">rand</span><span class="o">-</span><span class="n">m9</span><span class="o">-</span><span class="n">mstd0</span><span class="mf">.5</span>
 </pre></div>
 
 </div>

docs/SGDR.html

@@ -79,12 +79,30 @@
 
 
 <div class="output_markdown rendered_html output_subarea ">
-<h2 id="CosineLRScheduler" class="doc_header"><code>class</code> <code>CosineLRScheduler</code><a href="https://github.com/rwightman/pytorch-image-models/tree/master/timm/scheduler/cosine_lr.py#L18" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>CosineLRScheduler</code>(<strong><code>optimizer</code></strong>:<code>Optimizer</code>, <strong><code>t_initial</code></strong>:<code>int</code>, <strong><code>t_mul</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>lr_min</code></strong>:<code>float</code>=<em><code>0.0</code></em>, <strong><code>decay_rate</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>warmup_t</code></strong>=<em><code>0</code></em>, <strong><code>warmup_lr_init</code></strong>=<em><code>0</code></em>, <strong><code>warmup_prefix</code></strong>=<em><code>False</code></em>, <strong><code>cycle_limit</code></strong>=<em><code>0</code></em>, <strong><code>t_in_epochs</code></strong>=<em><code>True</code></em>, <strong><code>noise_range_t</code></strong>=<em><code>None</code></em>, <strong><code>noise_pct</code></strong>=<em><code>0.67</code></em>, <strong><code>noise_std</code></strong>=<em><code>1.0</code></em>, <strong><code>noise_seed</code></strong>=<em><code>42</code></em>, <strong><code>initialize</code></strong>=<em><code>True</code></em>) :: <code>Scheduler</code></p>
+<h2 id="CosineLRScheduler" class="doc_header"><code>class</code> <code>CosineLRScheduler</code><a href="timm/scheduler/cosine_lr.py#L16" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>CosineLRScheduler</code>(<strong><code>optimizer</code></strong>:<code>Optimizer</code>, <strong><code>t_initial</code></strong>:<code>int</code>, <strong><code>lr_min</code></strong>:<code>float</code>=<em><code>0.0</code></em>, <strong><code>cycle_mul</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>cycle_decay</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>cycle_limit</code></strong>:<code>int</code>=<em><code>1</code></em>, <strong><code>warmup_t</code></strong>:<code>int</code>=<em><code>0</code></em>, <strong><code>warmup_lr_init</code></strong>:<code>float</code>=<em><code>0</code></em>, <strong><code>warmup_prefix</code></strong>:<code>bool</code>=<em><code>False</code></em>, <strong><code>t_in_epochs</code></strong>:<code>bool</code>=<em><code>True</code></em>, <strong><code>noise_range_t</code></strong>:<code>Union</code>[<code>int</code>, <code>float</code>, <code>List</code>[<code>Union</code>[<code>int</code>, <code>float</code>]]]=<em><code>None</code></em>, <strong><code>noise_pct</code></strong>:<code>float</code>=<em><code>0.67</code></em>, <strong><code>noise_std</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>noise_seed</code></strong>:<code>int</code>=<em><code>42</code></em>, <strong><code>k_decay</code></strong>:<code>float</code>=<em><code>1.0</code></em>, <strong><code>initialize</code></strong>:<code>bool</code>=<em><code>True</code></em>) :: <code>Scheduler</code></p>
 </blockquote>
 <p>Cosine decay with restarts.
 This is described in the paper <a href="https://arxiv.org/abs/1608.03983">https://arxiv.org/abs/1608.03983</a>.</p>
 <p>Inspiration from
 <a href="https://github.com/allenai/allennlp/blob/master/allennlp/training/learning_rate_schedulers/cosine.py">https://github.com/allenai/allennlp/blob/master/allennlp/training/learning_rate_schedulers/cosine.py</a></p>
+<p>k-decay option based on <code>k-decay: A New Method For Learning Rate Schedule</code> - <a href="https://arxiv.org/abs/2004.05909">https://arxiv.org/abs/2004.05909</a></p>
+<p>Args:<br>
+    optimizer (torch.optim.Optimizer): torch optimizer to schedule<br>
+    t_initial (int): Number of epochs it initial (first) cycle.<br>
+    lr_min (float, optional):  Minimum learning rate to use during the scheduling. Defaults to 0..<br>
+    cycle_mul (float, optional): Multiplyer for cycle length. Defaults to 1..<br>
+    cycle_decay (float, optional): Factor to decay lr at next cycle. Defaults to 1..<br>
+    cycle_limit (int, optional): Number of cycles. Defaults to 1.<br>
+    warmup_t (int, optional): Number of epochs to warmup. Defaults to 0.<br>
+    warmup_lr_init (float, optional): Initial learning rate during warmup . Defaults to 0.<br>
+    warmup_prefix (bool, optional): If True, after warmup annealing starts from initial LR. Defaults to False.<br>
+    t_in_epochs (bool, optional): If set to False, returned lr are None. Defaults to True.<br>
+    noise_range_t (Union[int, float, List[int, float]], optional): Epoch when noise starts.            If list or tuple - epoch range, when noise applied. Defaults to None.<br>
+    noise_pct (float, optional): Percentage of noise to add. Defaults to 0.67.<br>
+    noise_std (float, optional): Noise standard deviation. Defaults to 1.0.<br>
+    noise_seed (int, optional): Seed to use to add random noise. Defaults to 42.<br>
+    k_decay (float, optional): Power for k<em>decay. Defaults to 1.0.<br>
+    initialize (bool, optional): Add initial</em>{field_name} to optimizer param group. Defaults to True.</p>
 
 </div>
 

docs/_data/sidebars/home_sidebar.yml

@@ -96,6 +96,12 @@ entries:
         - output: web,pdf
           title: Plateau
           url: /plateau
+        - output: web,pdf
+          title: Multistep
+          url: /multistepLR
+        - output: web,pdf
+          title: PolyLR
+          url: /PolyLR
         title: Schedulers
       - output: web
         subfolderitems:

docs/dataloader.html

@@ -322,14 +322,13 @@ <h3 id="Prefetch-loader">Prefetch loader<a class="anchor-link" href="#Prefetch-l
 <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
 <div class="text_cell_render border-box-sizing rendered_html">
 <p>From the <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams">documentation</a> available on PyTorch:</p>
-
-<pre><code>markdown
-A CUDA stream is a linear sequence of execution that belongs to a specific device. You normally do not need to create one explicitly: by default, each device uses its own “default” stream.
+<div class="highlight"><pre><span></span>A CUDA stream is a linear sequence of execution that belongs to a specific device. You normally do not need to create one explicitly: by default, each device uses its own “default” stream.
 
 Operations inside each stream are serialized in the order they are created, but operations from different streams can execute concurrently in any relative order, unless explicit synchronization functions (such as synchronize() or wait_stream()) are used.
 
 
-When the “current stream” is the default stream, PyTorch automatically performs necessary synchronization when data is moved around. However, when using non-default streams, it is the user’s responsibility to ensure proper synchronization.</code></pre>
+When the “current stream” is the default stream, PyTorch automatically performs necessary synchronization when data is moved around. However, when using non-default streams, it is the user’s responsibility to ensure proper synchronization.
+</pre></div>
 
 </div>
 </div>

docs/dataset.html

@@ -66,15 +66,14 @@ <h3 id="Parser">Parser<a class="anchor-link" href="#Parser"> </a></h3>
 <div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
 <div class="text_cell_render border-box-sizing rendered_html">
 <p>The <code>parser</code> is set automatically using a <code>create_parser</code> factory method. The <code>parser</code> finds all images and targets in <code>root</code> where the <code>root</code> folder is structured like so:</p>
-
-<pre><code>markdown
-root/dog/xxx.png
+<div class="highlight"><pre><span></span>root/dog/xxx.png
 root/dog/xxy.png
 root/dog/xxz.png
 
 root/cat/123.png
 root/cat/nsdf3.png
-root/cat/asd932_.png</code></pre>
+root/cat/asd932_.png
+</pre></div>
 
 </div>
 </div>
@@ -297,7 +296,7 @@ <h3 id="Usage">Usage<a class="anchor-link" href="#Usage"> </a></h3>
 <span class="o">&lt;</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">14</span><span class="o">-</span><span class="mi">9085</span><span class="n">b17eda0c</span><span class="o">&gt;</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span>
 <span class="o">----&gt;</span> <span class="mi">1</span> <span class="n">iterable_dataset</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
 
-<span class="o">~/</span><span class="n">opt</span><span class="o">/</span><span class="n">anaconda3</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">python3</span><span class="o">.</span><span class="mi">8</span><span class="o">/</span><span class="n">site</span><span class="o">-</span><span class="n">packages</span><span class="o">/</span><span class="n">torch</span><span class="o">/</span><span class="n">utils</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">dataset</span><span class="o">.</span><span class="n">py</span> <span class="ow">in</span> <span class="fm">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span>
+<span class="o">~/</span><span class="n">opt</span><span class="o">/</span><span class="n">anaconda3</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">python3</span><span class="mf">.8</span><span class="o">/</span><span class="n">site</span><span class="o">-</span><span class="n">packages</span><span class="o">/</span><span class="n">torch</span><span class="o">/</span><span class="n">utils</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">dataset</span><span class="o">.</span><span class="n">py</span> <span class="ow">in</span> <span class="fm">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span>
      <span class="mi">30</span> 
      <span class="mi">31</span>     <span class="k">def</span> <span class="fm">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">T_co</span><span class="p">:</span><span class="o">---&gt;</span> <span class="mi">32</span>         <span class="k">raise</span> <span class="ne">NotImplementedError</span>     <span class="mi">33</span> 
      <span class="mi">34</span>     <span class="k">def</span> <span class="fm">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">:</span> <span class="s1">&#39;Dataset[T_co]&#39;</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="s1">&#39;ConcatDataset[T_co]&#39;</span><span class="p">:</span>

docs/jsd_cross_entropy.html

@@ -352,7 +352,7 @@
 
 <div class="inner_cell">
     <div class="input_area">
-<div class=" highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span> <span class="o">.</span><span class="mi">1</span><span class="o">*</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">losses</span><span class="p">))],[</span><span class="n">loss</span> <span class="k">for</span> <span class="n">loss</span> <span class="ow">in</span> <span class="n">losses</span><span class="p">])</span>
+<div class=" highlight hl-ipython3"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">([</span> <span class="mf">.1</span><span class="o">*</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">losses</span><span class="p">))],[</span><span class="n">loss</span> <span class="k">for</span> <span class="n">loss</span> <span class="ow">in</span> <span class="n">losses</span><span class="p">])</span>
 <span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;JS Divergence&#39;</span><span class="p">)</span>
 <span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;Change in output&#39;</span><span class="p">)</span>
 <span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>