Merge branch 'master' into transformer

Author: ArnoldLIULJ

CHANGELOG.md

@@ -79,6 +79,7 @@ To release a new version, please update the changelog as followed:
 ### Deprecated
 
 ### Fixed
+- RNN updates: remove warnings, fix if seq_len=0, unitest (#PR 1033)
 
 ### Removed
 
@@ -112,7 +113,8 @@ To release a new version, please update the changelog as followed:
 - Remove `private_method` decorator (#PR 1025)
 - Copy original model's `trainable_weights` and `nontrainable_weights` when initializing `ModelLayer` (#PR 1026)
 - Copy original model's `trainable_weights` and `nontrainable_weights` when initializing `LayerList` (#PR 1029)
-- remove redundant parts in `model.all_layers` (#PR 1029)
+- Remove redundant parts in `model.all_layers` (#PR 1029)
+- Replace `tf.image.resize_image_with_crop_or_pad` with `tf.image.resize_with_crop_or_pad` (#PR 1032)
 
 ### Removed
 
@@ -122,7 +124,7 @@ To release a new version, please update the changelog as followed:
 
 - @zsdonghao
 - @ChrisWu1997: #1010 #1015 #1025 #1030
-- @warshallrho: #1017 #1021 #1026 #1029
+- @warshallrho: #1017 #1021 #1026 #1029 #1032
 - @ArnoldLIULJ: #1023
 - @JingqingZ: #1023
 

README.md

@@ -36,9 +36,9 @@
 
 TensorLayer is a novel TensorFlow-based deep learning and reinforcement learning library designed for researchers and engineers. It provides a large collection of customizable neural layers / functions that are key to build real-world AI applications. TensorLayer is awarded the 2017 Best Open Source Software by the [ACM Multimedia Society](https://twitter.com/ImperialDSI/status/923928895325442049).
 
-🔥📰🔥 [Deep Reinforcement Learning Model ZOO](https://github.com/tensorlayer/tensorlayer/tree/master/examples/reinforcement_learning) Release !!!
+🔥📰🔥 Reinforcement Learning Model Zoos: [Low-level APIs for Research](https://github.com/tensorlayer/tensorlayer/tree/master/examples/reinforcement_learning) and [High-level APIs for Production](https://github.com/tensorlayer/RLzoo)
 
-🔥📰🔥 [Sipeed](https://github.com/sipeed/Maix-EMC): Run TensorLayer models on the **low-cost AI chip** (e.g., K210) (Alpha Version)
+🔥📰🔥 [Sipeed Maxi-EMC](https://github.com/sipeed/Maix-EMC): Run TensorLayer models on the **low-cost AI chip** (e.g., K210) (Alpha Version)
 
 🔥📰🔥 [NNoM](https://github.com/majianjia/nnom): Run TensorLayer quantized models on the **MCU** (e.g., STM32) (Coming Soon)
 
@@ -201,4 +201,4 @@ If you use TensorLayer for any projects, please cite this paper：
 
 # License
 
-TensorLayer is released under the Apache 2.0 license.
+TensorLayer is released under the Apache 2.0 license. We also host TensorLayer on [iHub](https://code.ihub.org.cn/projects/328) and [Gitee](https://gitee.com/organizations/TensorLayer).

docs/user/contributing.rst

@@ -4,14 +4,10 @@
 Contributing
 ===============
 
-TensorLayer is a major ongoing research project in Data Science Institute, Imperial College London. 
-The goal of the project is to develop a compositional language while complex learning systems
+TensorLayer 2.0 is a major ongoing research project in CFCS, Peking University, the first version was established at Imperial College London in 2016. The goal of the project is to develop a compositional language while complex learning systems
 can be built through composition of neural network modules.
 
-Numerous contributors come from various horizons such as: Tsinghua University, Carnegie Mellon University, University of Technology of Compiegne, 
-Google, Microsoft, Bloomberg and etc.
-
-There are many functions need to be contributed such as Maxout, Neural Turing Machine, Attention, TensorLayer Mobile and etc.
+Numerous contributors come from various horizons such as: Imperial College London, Tsinghua University, Carnegie Mellon University, Stanford, University of Technology of Compiegne, Google, Microsoft, Bloomberg and etc.
 
 You can easily open a Pull Request (PR) on `GitHub`_, every little step counts and will be credited.
 As an open-source project, we highly welcome and value contributions!
@@ -27,10 +23,9 @@ As an open-source project, we highly welcome and value contributions!
 Project Maintainers
 --------------------------
 
-
 The TensorLayer project was started by `Hao Dong <https://zsdonghao.github.io>`_ at Imperial College London in June 2016. 
 
-For TensorLayer 2.x, it is now actively developing and maintaining by the following people who has more than 50 contributions*:
+For TensorLayer 2.x, it is now actively developing and maintaining by the following people who has more than 50 contributions:
 
 - **Hao Dong** (`@zsdonghao <https://github.com/zsdonghao>`_) - `<https://zsdonghao.github.io>`_
 - **Jingqing Zhang** (`@JingqingZ <https://github.com/JingqingZ>`_) - `<https://jingqingz.github.io>`_
@@ -56,10 +51,9 @@ What to contribute
 Your method and example
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you have a new method or example in terms of Deep learning and Reinforcement learning,
-you are welcome to contribute.
+If you have a new method or example in terms of Deep learning or Reinforcement learning, you are welcome to contribute.
 
-* Provide your layer or example, so everyone can use it.
+* Provide your layers or examples, so everyone can use it.
 * Explain how it would work, and link to a scientific paper if applicable.
 * Keep the scope as narrow as possible, to make it easier to implement.
 

docs/user/get_involved.rst

@@ -4,8 +4,20 @@
 Get Involved in Research
 =========================
 
+Ph.D. Postition @ PKU
+=============================================================
+
+
+Hi, I am Hao Dong, a new faculty member in EECS, Peking University. I now have a few Ph.D. positions per year open for international students who would like to study AI. If you or your friends are interested in it, please contact me ASAP.
+PKU is a top 30 university in the global ranking. The application is competitive, apply early is recommended. For the application of next year, please note that the DDL of Chinese Government Scholarship is in the end of each year, please check this `link <http://www.isd.pku.edu.cn/info/1503/5676.htm>`__ for more details.
+
+My homepage: https://zsdonghao.github.io 
+
+Contact: hao.dong11[AT]imperial.ac.uk
+
+
 
-Center on Frontiers of Computing Studies, Peking University
+Faculty Postition @ PKU
 =============================================================
 
 The Center on Frontiers of Computing Studies (CFCS), Peking University (PKU), China, is a university new initiative co-founded by Professors John Hopcroft (Turing Awardee) and Wen Gao (CAE, ACM/IEEE Fellow). The center aims at developing the excellence on two fronts: research and education. On the research front, the center will provide a world-class research environment, where innovation and impactful research is the central aim, measured by professional reputation among world scholars, not by counting the number of publications and research funding. On the education front, the center deeply involves in the Turing Class, an elite undergraduate program that draws the cream of the crop from the PKU undergraduate talent pool. New curriculum and pedagogy are designed and practiced in this program, with the aim to cultivate a new generation of computer scientist/engineers that are solid in both theories and practices. 
@@ -27,12 +39,12 @@ Application for a postdoctoral position should include a curriculum vita, brief
 We conduct review of applications monthly, immediately upon the recipient of all application materials at the beginning of each month. However, it is highly recommended that applicants submit complete applications sooner than later, as the positions are to be filled quickly. 
 
 
-Data Science Institute, Imperial College London
+Postdoc Postition @ ICL
 ==================================================
 
 Data science is therefore by nature at the core of all modern transdisciplinary scientific activities, as it involves the whole life cycle of data, from acquisition and exploration to analysis and communication of the results. Data science is not only concerned with the tools and methods to obtain, manage and analyse data: it is also about extracting value from data and translating it from asset to product.
 
-Launched on 1st April 2014, the Data Science Institute at Imperial College London aims to enhance Imperial's excellence in data-driven research across its faculties by fulfilling the following objectives.
+Launched on 1st April 2014, the Data Science Institute (DSI) at Imperial College London aims to enhance Imperial's excellence in data-driven research across its faculties by fulfilling the following objectives.
 
 The Data Science Institute is housed in purpose built facilities in the heart of the Imperial College campus in South Kensington. Such a central location provides excellent access to collabroators across the College and across London.
 
@@ -48,3 +60,17 @@ and other ways to
 `get involved <https://www.imperial.ac.uk/data-science/get-involved/>`__
 , or feel free to
 `contact us <https://www.imperial.ac.uk/data-science/get-involved/contact-us/>`__.
+
+Software Engineering@SurgicalAI.cn
+=============================================================
+SurgicalAI is a startup founded by the data scientists and surgical robot experts from Imperial College. Our objective is AI democratise Surgery. By combining 5G, AI and Cloud Computing, SurgicalAI is building a platform enable junor surgeons to perfom complex procedures. As one of the most impactful startup, SurgicalAI is supported by Nvidia, AWS and top surgeons around the world.
+
+Currently based in Hangzhou, China, we are building digital solution for cardiac surgery like TAVR, LAA and Orthopedidcs like TKA and UNA. A demo can be found at here <http://demo5g.surgicalai.cn>
+
+We are activly looking for experts in robotic navigation, computer graphics and  medical image analysis experts to join us, building digitalized surgical service platform for the aging world.
+
+Home Page: http://www.surgicalai.cn
+
+Demo Page: http://demo5g.surgicalai.cn
+
+Contact: liufangde@surgicalai.cn

docs/user/get_start_model.rst

@@ -19,12 +19,12 @@ Static model
   def get_model(inputs_shape):
       ni = Input(inputs_shape)
       nn = Dropout(keep=0.8)(ni)
-      nn = Dense(n_units=800, act=tf.nn.relu, name="dense1")(nn)
+      nn = Dense(n_units=800, act=tf.nn.relu, name="dense1")(nn) # “name" is optional
       nn = Dropout(keep=0.8)(nn)
       nn = Dense(n_units=800, act=tf.nn.relu)(nn)
       nn = Dropout(keep=0.8)(nn)
-      nn = Dense(n_units=10, act=tf.nn.relu)(nn)
-      M = Model(inputs=ni, outputs=nn, name="mlp")
+      nn = Dense(n_units=10, act=None)(nn)
+      M = Model(inputs=ni, outputs=nn, name="mlp") # “name" is optional
       return M
 
   MLP = get_model([None, 784])
@@ -46,10 +46,10 @@ In this case, you need to manually input the output shape of the previous layer
 
           self.dropout1 = Dropout(keep=0.8)
           self.dense1 = Dense(n_units=800, act=tf.nn.relu, in_channels=784)
-          self.dropout2 = Dropout(keep=0.8)#(self.dense1)
+          self.dropout2 = Dropout(keep=0.8)
           self.dense2 = Dense(n_units=800, act=tf.nn.relu, in_channels=800)
-          self.dropout3 = Dropout(keep=0.8)#(self.dense2)
-          self.dense3 = Dense(n_units=10, act=tf.nn.relu, in_channels=800)
+          self.dropout3 = Dropout(keep=0.8)
+          self.dense3 = Dense(n_units=10, act=None, in_channels=800)
 
       def forward(self, x, foo=False):
           z = self.dropout1(x)
@@ -59,7 +59,7 @@ In this case, you need to manually input the output shape of the previous layer
           z = self.dropout3(z)
           out = self.dense3(z)
           if foo:
-              out = tf.nn.relu(out)
+              out = tf.nn.softmax(out)
           return out
 
   MLP = CustomModel()
@@ -156,7 +156,7 @@ Print model information
   #   (dropout_1): Dropout(keep=0.8, name='dropout_1')
   #   (dense_1): Dense(n_units=800, relu, in_channels='800', name='dense_1')
   #   (dropout_2): Dropout(keep=0.8, name='dropout_2')
-  #   (dense_2): Dense(n_units=10, relu, in_channels='800', name='dense_2')
+  #   (dense_2): Dense(n_units=10, None, in_channels='800', name='dense_2')
   # )
   
   import pprint
@@ -195,7 +195,7 @@ Print model information
   #                                   'name': 'dropout_3'},
   #                          'class': 'Dropout',
   #                          'prev_layer': ['dense_2_node_0']},
-  #                         {'args': {'act': 'relu',
+  #                         {'args': {'act': None,
   #                                   'layer_type': 'normal',
   #                                   'n_units': 10,
   #                                   'name': 'dense_3'},

examples/basic_tutorials/tutorial_cifar10_cnn_static.py

@@ -1,11 +1,9 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 
-import multiprocessing
 import time
-
 import numpy as np
-
+import multiprocessing
 import tensorflow as tf
 import tensorlayer as tl
 from tensorlayer.layers import (BatchNorm, Conv2d, Dense, Flatten, Input, LocalResponseNorm, MaxPool2d)
@@ -80,14 +78,11 @@ def get_model_batchnorm(inputs_shape):
 print_freq = 5
 n_step_epoch = int(len(y_train) / batch_size)
 n_step = n_epoch * n_step_epoch
-shuffle_buffer_size = 128  # 100
-# init_learning_rate = 0.1
-# learning_rate_decay_factor = 0.1
-# num_epoch_decay = 350
+shuffle_buffer_size = 128
 
 train_weights = net.trainable_weights
-# learning_rate = tf.Variable(init_learning_rate)
 optimizer = tf.optimizers.Adam(learning_rate)
+# looking for decay learning rate? see https://github.com/tensorlayer/srgan/blob/master/train.py
 
 
 def generator_train():
@@ -127,7 +122,7 @@ def _map_fn_train(img, target):
 
 def _map_fn_test(img, target):
     # 1. Crop the central [height, width] of the image.
-    img = tf.image.resize_image_with_crop_or_pad(img, 24, 24)
+    img = tf.image.resize_with_crop_or_pad(img, 24, 24)
     # 2. Subtract off the mean and divide by the variance of the pixels.
     img = tf.image.per_image_standardization(img)
     img = tf.reshape(img, (24, 24, 3))
@@ -182,14 +177,10 @@ def _map_fn_test(img, target):
 
     # use training and evaluation sets to evaluate the model every print_freq epoch
     if epoch + 1 == 1 or (epoch + 1) % print_freq == 0:
-
         print("Epoch {} of {} took {}".format(epoch + 1, n_epoch, time.time() - start_time))
-
         print("   train loss: {}".format(train_loss / n_iter))
         print("   train acc:  {}".format(train_acc / n_iter))
-
         net.eval()
-
         val_loss, val_acc, n_iter = 0, 0, 0
         for X_batch, y_batch in test_ds:
             _logits = net(X_batch)  # is_train=False, disable dropout
@@ -199,10 +190,6 @@ def _map_fn_test(img, target):
         print("   val loss: {}".format(val_loss / n_iter))
         print("   val acc:  {}".format(val_acc / n_iter))
 
-    # FIXME : how to apply lr decay in eager mode?
-    # learning_rate.assign(tf.train.exponential_decay(init_learning_rate, epoch, num_epoch_decay,
-    #                                                 learning_rate_decay_factor))
-
 # use testing data to evaluate the model
 net.eval()
 test_loss, test_acc, n_iter = 0, 0, 0

examples/basic_tutorials/tutorial_mnist_mlp_dynamic.py

@@ -1,7 +1,5 @@
 import time
-
 import numpy as np
-
 import tensorflow as tf
 import tensorlayer as tl
 from tensorlayer.layers import Dense, Dropout, Input
@@ -19,7 +17,6 @@ class CustomModel(Model):
 
     def __init__(self):
         super(CustomModel, self).__init__()
-
         self.dropout1 = Dropout(keep=0.8)  #(self.innet)
         self.dense1 = Dense(n_units=800, act=tf.nn.relu, in_channels=784)  #(self.dropout1)
         self.dropout2 = Dropout(keep=0.8)  #(self.dense1)
@@ -52,27 +49,20 @@ def forward(self, x, foo=None):
 for epoch in range(n_epoch):  ## iterate the dataset n_epoch times
     start_time = time.time()
     ## iterate over the entire training set once (shuffle the data via training)
-
     for X_batch, y_batch in tl.iterate.minibatches(X_train, y_train, batch_size, shuffle=True):
-
         MLP.train()  # enable dropout
-
         with tf.GradientTape() as tape:
             ## compute outputs
             _logits = MLP(X_batch, foo=1)
             ## compute loss and update model
             _loss = tl.cost.cross_entropy(_logits, y_batch, name='train_loss')
-
         grad = tape.gradient(_loss, train_weights)
         optimizer.apply_gradients(zip(grad, train_weights))
 
     ## use training and evaluation sets to evaluate the model every print_freq epoch
     if epoch + 1 == 1 or (epoch + 1) % print_freq == 0:
-
         MLP.eval()  # disable dropout
-
         print("Epoch {} of {} took {}".format(epoch + 1, n_epoch, time.time() - start_time))
-
         train_loss, train_acc, n_iter = 0, 0, 0
         for X_batch, y_batch in tl.iterate.minibatches(X_train, y_train, batch_size, shuffle=False):
             _logits = MLP(X_batch, foo=1)
@@ -81,7 +71,6 @@ def forward(self, x, foo=None):
             n_iter += 1
         print("   train foo=1 loss: {}".format(train_loss / n_iter))
         print("   train foo=1 acc:  {}".format(train_acc / n_iter))
-
         val_loss, val_acc, n_iter = 0, 0, 0
         for X_batch, y_batch in tl.iterate.minibatches(X_val, y_val, batch_size, shuffle=False):
             _logits = MLP(X_batch, foo=1)  # is_train=False, disable dropout
@@ -90,7 +79,6 @@ def forward(self, x, foo=None):
             n_iter += 1
         print("   val foo=1 loss: {}".format(val_loss / n_iter))
         print("   val foo=1 acc:  {}".format(val_acc / n_iter))
-
         val_loss, val_acc, n_iter = 0, 0, 0
         for X_batch, y_batch in tl.iterate.minibatches(X_val, y_val, batch_size, shuffle=False):
             _logits = MLP(X_batch)  # is_train=False, disable dropout

examples/basic_tutorials/tutorial_mnist_mlp_dynamic_2.py

@@ -1,7 +1,5 @@
 import time
-
 import numpy as np
-
 import tensorflow as tf
 import tensorlayer as tl
 from tensorlayer.layers import Dense, Dropout, Input, LayerList
@@ -19,17 +17,14 @@ class CustomModelHidden(Model):
 
     def __init__(self):
         super(CustomModelHidden, self).__init__()
-
         self.dropout1 = Dropout(keep=0.8)  #(self.innet)
-
         self.seq = LayerList(
             [
                 Dense(n_units=800, act=tf.nn.relu, in_channels=784),
                 Dropout(keep=0.8),
                 Dense(n_units=800, act=tf.nn.relu, in_channels=800),
             ]
         )
-
         self.dropout3 = Dropout(keep=0.8)  #(self.seq)
 
     def forward(self, x):
@@ -43,7 +38,6 @@ class CustomModelOut(Model):
 
     def __init__(self):
         super(CustomModelOut, self).__init__()
-
         self.dense3 = Dense(n_units=10, act=tf.nn.relu, in_channels=800)
 
     def forward(self, x, foo=None):
@@ -74,30 +68,23 @@ def forward(self, x, foo=None):
 for epoch in range(n_epoch):  ## iterate the dataset n_epoch times
     start_time = time.time()
     ## iterate over the entire training set once (shuffle the data via training)
-
     for X_batch, y_batch in tl.iterate.minibatches(X_train, y_train, batch_size, shuffle=True):
-
         MLP1.train()  # enable dropout
         MLP2.train()
-
         with tf.GradientTape() as tape:
             ## compute outputs
             _hidden = MLP1(X_batch)
             _logits = MLP2(_hidden, foo=1)
             ## compute loss and update model
             _loss = tl.cost.cross_entropy(_logits, y_batch, name='train_loss')
-
         grad = tape.gradient(_loss, train_weights)
         optimizer.apply_gradients(zip(grad, train_weights))
 
     ## use training and evaluation sets to evaluate the model every print_freq epoch
     if epoch + 1 == 1 or (epoch + 1) % print_freq == 0:
-
         MLP1.eval()  # disable dropout
         MLP2.eval()
-
         print("Epoch {} of {} took {}".format(epoch + 1, n_epoch, time.time() - start_time))
-
         train_loss, train_acc, n_iter = 0, 0, 0
         for X_batch, y_batch in tl.iterate.minibatches(X_train, y_train, batch_size, shuffle=False):
             _hidden = MLP1(X_batch)