Gated Residual Network block

GRN is one of the elements the TFT model is composed of. The expected benefit from applying such value is a better ability of switching between linear and non-linear processing.

Usage

layer_grn(
  object,
  hidden_units,
  output_size = hidden_units,
  dropout_rate = NULL,
  use_context = FALSE,
  return_gate = FALSE,
  ...
)

Arguments

object

What to compose the new Layer instance with. Typically a Sequential model or a Tensor (e.g., as returned by layer_input()). The return value depends on object. If object is:

• missing or NULL, the Layer instance is returned.

• a Sequential model, the model with an additional layer is returned.

• a Tensor, the output tensor from layer_instance(object) is returned.

hidden_units

Size of the hidden layer.

output_size

Dimensionality of the output feature space.

use_context

Use additional (static) context. If TRUE, an additional layer is created to handle context input.

return_gate

Logical - return gate values. Default: FALSE

Details

Its output is computed as: $$GRN(a,c) = LayerNorm(a + GLU({\eta}_1))$$ $${\eta}_1 = W_1\eta_2 + b_1$$ $$\eta_2 = ELU(W_2a + W_3c + b_2)$$

Input and Output Shapes

Input shape: nD tensor with shape: (batch_size, ..., input_dim). The most common situation would be a 2D input with shape (batch_size, input_dim).

Output shape: nD tensor with shape: (batch_size, ..., units). For instance, for a 2D input with shape (batch_size, input_dim), the output would have shape (batch_size, unit).

References

B. Lim, S.O. Arik, N. Loeff, T. Pfiste, Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting(2020)

Examples

library(keras)
# ================================================================
#             SEQUENTIAL MODEL, NO GATE VALUES RETURNED
# ================================================================
model <-
  keras_model_sequential() %>%
  layer_grn(10, input_shape = 30)
model
#> Model: "sequential_1"
#> ________________________________________________________________________________
#>  Layer (type)                       Output Shape                    Param #     
#> ================================================================================
#>  grn (GRN)                          (None, 10)                      970         
#> ================================================================================
#> Total params: 970
#> Trainable params: 970
#> Non-trainable params: 0
#> ________________________________________________________________________________
output <- model(matrix(1, 32, 30))
#> Error in py_call_impl(callable, dots$args, dots$keywords): RuntimeError: Exception encountered when calling layer "grn" "                 f"(type GRN).
#> 
#> Evaluation error: tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer "dense" "                 f"(type Dense).
#> 
#> {{function_node __wrapped__MatMul_device_/job:localhost/replica:0/task:0/device:GPU:0}} Attempting to perform BLAS operation using StreamExecutor without BLAS support [Op:MatMul]
#> 
#> Call arguments received by layer "dense" "                 f"(type Dense):
#>   • inputs=tf.Tensor(shape=(32, 30), dtype=float32)
#> .
#> 
#> Call arguments received by layer "grn" "                 f"(type GRN):
#>   • inputs=tf.Tensor(shape=(32, 30), dtype=float32)
#>   • context=None
dim(output)
#> Error in eval(expr, envir, enclos): object 'output' not found
output[1,]
#> Error in eval(expr, envir, enclos): object 'output' not found
#'================================================================
#            WITH GATE VALUES AND ADDITIONAL CONTEXT
# ================================================================
inp  <- layer_input(c(28, 5))
ctx  <- layer_input(10)
out  <- layer_grn(
            hidden_units = 10,
            return_gate = TRUE,
            use_context = TRUE
         )(inp, context = ctx)
model <- keras_model(list(inp, ctx), out)
model
#> Model: "model_1"
#> ________________________________________________________________________________
#>  Layer (type)             Output Shape      Param #  Connected to               
#> ================================================================================
#>  input_2 (InputLayer)     [(None, 28, 5)]   0        []                         
#>  input_3 (InputLayer)     [(None, 10)]      0        []                         
#>  grn_1 (GRN)              [(None, 28, 10),  570      ['input_2[0][0]',          
#>                            (None, 28, 10)]            'input_3[0][0]']          
#> ================================================================================
#> Total params: 570
#> Trainable params: 570
#> Non-trainable params: 0
#> ________________________________________________________________________________
arr_1 <- array(1, dim = c(1, 28, 5))
arr_2 <- array(1, dim = c(1, 10))
c(values, gate) %<-% model(list(arr_1, arr_2))
#> Error in py_call_impl(callable, dots$args, dots$keywords): RuntimeError: Exception encountered when calling layer "grn_1" "                 f"(type GRN).
#> 
#> Evaluation error: tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer "dense" "                 f"(type Dense).
#> 
#> {{function_node __wrapped__MatMul_device_/job:localhost/replica:0/task:0/device:GPU:0}} Attempting to perform BLAS operation using StreamExecutor without BLAS support [Op:MatMul]
#> 
#> Call arguments received by layer "dense" "                 f"(type Dense):
#>   • inputs=tf.Tensor(shape=(1, 28, 5), dtype=float32)
#> .
#> 
#> Call arguments received by layer "grn_1" "                 f"(type GRN):
#>   • inputs=tf.Tensor(shape=(1, 28, 5), dtype=float32)
#>   • context=tf.Tensor(shape=(1, 10), dtype=float32)
dim(values)
#> Error in eval(expr, envir, enclos): object 'values' not found
dim(gate)
#> Error in eval(expr, envir, enclos): object 'gate' not found
values[1, all_dims()]
#> Error in eval(expr, envir, enclos): object 'values' not found
gate[1, all_dims()]
#> Error in eval(expr, envir, enclos): object 'gate' not found