Hyperparameter Tuning with the HParams Dashboard

Experiment setup and the HParams experiment summary

Experiment with three hyperparameters in the model:

• Number of units in the first dense layer
• Dropout rate in the dropout layer
• Optimizer

List the values to try, and log an experiment configuration to TensorBoard. This step is optional: you can provide domain information to enable more precise filtering of hyperparameters in the UI, and you can specify which metrics should be displayed.

If you choose to skip this step, you can use a string literal wherever you would otherwise use an HParam value: e.g., hparams["dropout"] or hparams$dropout instead of hparams[HP_DROPOUT].

HP_NUM_UNITS = hp$HParam('num_units', hp$Discrete(list(16, 32)))
HP_DROPOUT = hp$HParam('dropout', hp$RealInterval(0.1, 0.2))
HP_OPTIMIZER = hp$HParam('optimizer', hp$Discrete(list('adam', 'sgd')))

METRIC_ACCURACY = 'accuracy'

with(tf$summary$create_file_writer("logs/hparam_tuning")$as_default(), {
  hp$hparams_config(
    hparams = list(HP_NUM_UNITS, HP_DROPOUT, HP_OPTIMIZER),
    metrics = list(hp$Metric(METRIC_ACCURACY, display_name = "Accuracy"))
  )
})

Adapt TensorFlow runs to log hyperparameters and metrics

The model will be quite simple: two dense layers with a dropout layer between them. The training code will look familiar, although the hyperparameters are no longer hardcoded. Instead, the hyperparameters are provided in an hparams dictionary and used throughout the training function:

train_test_model <- function(hparams) {
  
  model <- keras_model_sequential() %>% 
    layer_flatten() %>% 
    layer_dense(py_to_r(hparams[HP_NUM_UNITS]), activation = "relu") %>% 
    layer_dropout(py_to_r(hparams[HP_DROPOUT])) %>% 
    layer_dense(units = 10, activation = "softmax")
  
  model %>% 
    compile(
      optimizer = py_to_r(hparams[HP_OPTIMIZER]),
      loss = "sparse_categorical_crossentropy",
      metrics = "accuracy"
    )
  
  model %>% fit(
    x = fashion_mnist$train$x, 
    y = fashion_mnist$train$y, 
    epochs = 1
  ) # Run with 1 epoch to speed things up for demo purposes
  
  results <- model %>% evaluate(
    x = fashion_mnist$test$x, 
    y = fashion_mnist$test$y,
    verbose = 0
  )
  
  results$accuracy
}

For each run, log an hparams summary with the hyperparameters and final accuracy:

run <- function(run_dir, hparams) {
  with(tf$summary$create_file_writer(run_dir)$as_default(), {
    hp$hparams(hparams) # record the values used in this trial
    accuracy <- train_test_model(hparams)
    tf$summary$scalar(METRIC_ACCURACY, accuracy, step = as.integer(1))
  })
}

When training Keras models, you can use callbacks instead of writing these directly:

model %>%  fit(
    ...,
    callbacks=list(
        callback_tensorboard(logdir),  # log metrics
        hp$KerasCallback(logdir, hparams),  # log hparams
    ),
)

Start runs and log them all under one parent directory

You can now try multiple experiments, training each one with a different set of hyperparameters.

For simplicity, use a grid search: try all combinations of the discrete parameters and just the lower and upper bounds of the real-valued parameter. For more complex scenarios, it might be more effective to choose each hyperparameter value randomly (this is called a random search). There are more advanced methods that can be used.

Run a few experiments, which will take a few minutes:

session_num <- 0

for(num_units in HP_NUM_UNITS$domain$values) {
  for (dropout_rate in c(HP_DROPOUT$domain$min_value, HP_DROPOUT$domain$max_value)) {
    for (optimizer in HP_OPTIMIZER$domain$values) {
      hparams <- dict(
        HP_NUM_UNITS = num_units,
        HP_DROPOUT = dropout_rate,
        HP_OPTIMIZER = optimizer
      )
      
      run_name <- sprintf("run-%04d", session_num)
      print(sprintf('--- Starting trial: %s',  run_name))
      #purrr::iwalk(hparams, ~print(paste(.y, .x, sep = ": ")))
      run(paste0('logs/hparam_tuning/', run_name), hparams)
      session_num <- session_num + 1
    }
  }
}

Visualize the results in TensorBoard’s HParams plugin

tensorboard("logs/hparam_tuning/")

The left pane of the dashboard provides filtering capabilities that are active across all the views in the HParams dashboard:

• Filter which hyperparameters/metrics are shown in the dashboard
• Filter which hyperparameter/metrics values are shown in the dashboard
• Filter on run status (running, success, …)
• Sort by hyperparameter/metric in the table view
• Number of session groups to show (useful for performance when there are many experiments)

The HParams dashboard has three different views, with various useful information:

• The Table View lists the runs, their hyperparameters, and their metrics.
• The Parallel Coordinates View shows each run as a line going through an axis for each hyperparemeter and metric. Click and drag the mouse on any axis to mark a region which will highlight only the runs that pass through it. This can be useful for identifying which groups of hyperparameters are most important. The axes themselves can be re-ordered by dragging them.
• The Scatter Plot View shows plots comparing each hyperparameter/metric with each metric. This can help identify correlations. Click and drag to select a region in a specific plot and highlight those sessions across the other plots.

A table row, a parallel coordinates line, and a scatter plot market can be clicked to see a plot of the metrics as a function of training steps for that session (although in this tutorial only one step is used for each run).