A1-Q1_HyperparameterSearch.log

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.542, Test Loss:0.405, Test Accuracy: 90.28%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(0, 90.28, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.534, Test Loss:0.433, Test Accuracy: 91.74%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.769, Test Loss:0.668, Test Accuracy: 89.06%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.487, Test Loss:0.659, Test Accuracy: 85.15%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.525, Test Loss:0.559, Test Accuracy: 89.87%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:148: RuntimeWarning: divide by zero encountered in log
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:145: RuntimeWarning: overflow encountered in square
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:112: RuntimeWarning: invalid value encountered in subtract
/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:26: RuntimeWarning: invalid value encountered in reduce
  return umr_maximum(a, axis, None, out, keepdims)
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:94: RuntimeWarning: invalid value encountered in less_equal
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:95: RuntimeWarning: invalid value encountered in greater

Epoch: 1, Learning Rate: 1.00e-02, Training Loss:nan, Test Loss:nan, Test Accuracy: 9.80%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:inf, Test Loss:2.04, Test Accuracy: 20.97%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:nan, Test Loss:nan, Test Accuracy: 9.80%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.719, Test Loss:0.818, Test Accuracy: 87.70%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(1, 91.74, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.12, Test Accuracy: 96.36%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0825, Test Loss:0.101, Test Accuracy: 97.09%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0527, Test Loss:0.0988, Test Accuracy: 97.16%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0431, Test Loss:0.0905, Test Accuracy: 97.53%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0326, Test Loss:0.095, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(9, 97.66, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.128, Test Accuracy: 96.39%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0952, Test Loss:0.12, Test Accuracy: 96.87%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0674, Test Loss:0.112, Test Accuracy: 97.31%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.057, Test Loss:0.12, Test Accuracy: 97.24%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0531, Test Loss:0.108, Test Accuracy: 97.68%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(10, 97.68, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.254, Test Loss:0.195, Test Accuracy: 96.07%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.163, Test Loss:0.187, Test Accuracy: 96.59%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.145, Test Loss:0.191, Test Accuracy: 96.47%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.136, Test Loss:0.188, Test Accuracy: 96.80%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.133, Test Loss:0.209, Test Accuracy: 96.36%
---------------------------------------------------------------------------------------------
(10, 97.68, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.195, Test Loss:0.131, Test Accuracy: 95.86%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.079, Test Loss:0.1, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0444, Test Loss:0.0798, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0287, Test Loss:0.0906, Test Accuracy: 97.66%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.019, Test Loss:0.079, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(12, 98.05, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.203, Test Loss:0.144, Test Accuracy: 95.92%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0875, Test Loss:0.118, Test Accuracy: 96.80%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0557, Test Loss:0.0989, Test Accuracy: 97.50%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0415, Test Loss:0.101, Test Accuracy: 97.61%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0322, Test Loss:0.1, Test Accuracy: 97.74%
---------------------------------------------------------------------------------------------
(12, 98.05, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.255, Test Loss:0.188, Test Accuracy: 96.25%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.156, Test Loss:0.171, Test Accuracy: 96.77%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.128, Test Loss:0.163, Test Accuracy: 97.25%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.109, Test Loss:0.154, Test Accuracy: 97.33%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0975, Test Loss:0.14, Test Accuracy: 97.81%
---------------------------------------------------------------------------------------------
(12, 98.05, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.121, Test Accuracy: 96.17%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0722, Test Loss:0.109, Test Accuracy: 96.71%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0382, Test Loss:0.0758, Test Accuracy: 97.73%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0221, Test Loss:0.07, Test Accuracy: 97.93%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0126, Test Loss:0.066, Test Accuracy: 98.08%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(15, 98.08, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.144, Test Accuracy: 95.81%



Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.147, Test Accuracy: 95.45%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0713, Test Loss:0.0876, Test Accuracy: 97.43%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0392, Test Loss:0.0773, Test Accuracy: 97.81%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0209, Test Loss:0.0711, Test Accuracy: 98.11%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0119, Test Loss:0.0663, Test Accuracy: 98.36%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.129, Test Accuracy: 96.30%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0805, Test Loss:0.101, Test Accuracy: 97.30%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0508, Test Loss:0.0911, Test Accuracy: 97.73%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0335, Test Loss:0.0934, Test Accuracy: 97.80%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0241, Test Loss:0.0787, Test Accuracy: 98.09%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.257, Test Loss:0.189, Test Accuracy: 96.34%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.147, Test Loss:0.166, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.113, Test Loss:0.162, Test Accuracy: 97.08%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0941, Test Loss:0.133, Test Accuracy: 97.87%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0806, Test Loss:0.125, Test Accuracy: 97.93%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.203, Test Loss:0.119, Test Accuracy: 96.43%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0792, Test Loss:0.1, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0488, Test Loss:0.0768, Test Accuracy: 97.62%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0316, Test Loss:0.0737, Test Accuracy: 97.73%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0209, Test Loss:0.0717, Test Accuracy: 98.03%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.208, Test Loss:0.123, Test Accuracy: 96.36%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0845, Test Loss:0.102, Test Accuracy: 97.25%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0546, Test Loss:0.0982, Test Accuracy: 97.30%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0372, Test Loss:0.0894, Test Accuracy: 97.55%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0287, Test Loss:0.0831, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.243, Test Loss:0.176, Test Accuracy: 95.79%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.127, Test Loss:0.145, Test Accuracy: 97.07%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.13, Test Accuracy: 97.53%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0865, Test Loss:0.127, Test Accuracy: 97.62%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0771, Test Loss:0.143, Test Accuracy: 97.34%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.204, Test Loss:0.118, Test Accuracy: 96.36%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0761, Test Loss:0.0944, Test Accuracy: 97.14%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0458, Test Loss:0.0791, Test Accuracy: 97.67%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.028, Test Loss:0.0672, Test Accuracy: 97.93%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0177, Test Loss:0.0649, Test Accuracy: 98.07%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.207, Test Loss:0.123, Test Accuracy: 96.23%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0813, Test Loss:0.0976, Test Accuracy: 97.08%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0515, Test Loss:0.0807, Test Accuracy: 97.69%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0346, Test Loss:0.0744, Test Accuracy: 97.90%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0244, Test Loss:0.0714, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.244, Test Loss:0.168, Test Accuracy: 96.18%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.124, Test Loss:0.137, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0968, Test Loss:0.126, Test Accuracy: 97.67%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.082, Test Loss:0.119, Test Accuracy: 97.79%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0722, Test Loss:0.114, Test Accuracy: 97.91%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.201, Test Loss:0.123, Test Accuracy: 96.28%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.073, Test Loss:0.0878, Test Accuracy: 97.35%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0424, Test Loss:0.0743, Test Accuracy: 97.81%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0269, Test Loss:0.069, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0181, Test Loss:0.064, Test Accuracy: 98.10%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.207, Test Loss:0.127, Test Accuracy: 96.27%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.079, Test Loss:0.0933, Test Accuracy: 97.30%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0492, Test Loss:0.0788, Test Accuracy: 97.73%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0336, Test Loss:0.0721, Test Accuracy: 97.92%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0248, Test Loss:0.0673, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.242, Test Loss:0.168, Test Accuracy: 96.16%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.121, Test Loss:0.138, Test Accuracy: 97.18%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0939, Test Loss:0.122, Test Accuracy: 97.59%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0797, Test Loss:0.113, Test Accuracy: 97.84%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0713, Test Loss:0.109, Test Accuracy: 97.89%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.283, Test Loss:0.192, Test Accuracy: 94.32%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.129, Test Loss:0.158, Test Accuracy: 96.13%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0957, Test Loss:0.135, Test Accuracy: 96.54%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0834, Test Loss:0.167, Test Accuracy: 96.28%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0835, Test Loss:0.158, Test Accuracy: 96.31%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.271, Test Loss:0.258, Test Accuracy: 92.55%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.344, Test Loss:0.265, Test Accuracy: 95.41%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.245, Test Loss:0.248, Test Accuracy: 96.03%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.227, Test Loss:0.247, Test Accuracy: 96.21%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.223, Test Loss:0.243, Test Accuracy: 96.82%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.22, Test Loss:0.256, Test Accuracy: 96.47%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.255, Test Loss:0.18, Test Accuracy: 94.50%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.105, Test Loss:0.12, Test Accuracy: 96.63%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0704, Test Loss:0.129, Test Accuracy: 96.50%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0463, Test Loss:0.0954, Test Accuracy: 97.58%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0297, Test Loss:0.109, Test Accuracy: 97.32%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.276, Test Loss:0.166, Test Accuracy: 95.57%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.125, Test Loss:0.136, Test Accuracy: 96.91%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0928, Test Loss:0.147, Test Accuracy: 96.88%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0738, Test Loss:0.136, Test Accuracy: 97.29%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0612, Test Loss:0.119, Test Accuracy: 97.76%
---------------------------------------------------------------------------------------------
(15, 98.36, {'h1': 700, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.344, Test Loss:0.264, Test Accuracy: 95.29%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.224, Test Loss:0.292, Test Accuracy: 94.77%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.195, Test Loss:0.207, Test Accuracy: 97.12%




Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.347, Test Loss:0.279, Test Accuracy: 94.54%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.228, Test Loss:0.24, Test Accuracy: 96.16%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.19, Test Loss:0.209, Test Accuracy: 96.70%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.167, Test Loss:0.226, Test Accuracy: 96.08%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.146, Test Loss:0.183, Test Accuracy: 97.19%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(32, 97.19, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0.01}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.256, Test Loss:0.18, Test Accuracy: 94.79%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0966, Test Loss:0.119, Test Accuracy: 96.55%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0526, Test Loss:0.0923, Test Accuracy: 97.45%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0295, Test Loss:0.0849, Test Accuracy: 97.80%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0153, Test Loss:0.0821, Test Accuracy: 98.07%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.273, Test Loss:0.167, Test Accuracy: 95.62%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.117, Test Loss:0.131, Test Accuracy: 96.84%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0759, Test Loss:0.118, Test Accuracy: 97.26%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0538, Test Loss:0.112, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0404, Test Loss:0.0943, Test Accuracy: 98.07%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.346, Test Loss:0.304, Test Accuracy: 93.89%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.118, Test Accuracy: 96.44%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0769, Test Loss:0.0969, Test Accuracy: 97.12%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.047, Test Loss:0.085, Test Accuracy: 97.65%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0322, Test Loss:0.0972, Test Accuracy: 97.23%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0229, Test Loss:0.0812, Test Accuracy: 97.87%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.201, Test Loss:0.124, Test Accuracy: 96.14%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.083, Test Loss:0.102, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0555, Test Loss:0.0878, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0403, Test Loss:0.0855, Test Accuracy: 97.82%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0352, Test Loss:0.0907, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.167, Test Accuracy: 96.22%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.132, Test Loss:0.153, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.11, Test Loss:0.145, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0954, Test Loss:0.14, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0861, Test Loss:0.135, Test Accuracy: 97.71%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.195, Test Loss:0.134, Test Accuracy: 95.78%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0729, Test Loss:0.0989, Test Accuracy: 97.12%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0423, Test Loss:0.0732, Test Accuracy: 97.83%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0251, Test Loss:0.079, Test Accuracy: 97.84%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0156, Test Loss:0.0678, Test Accuracy: 98.06%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.201, Test Loss:0.128, Test Accuracy: 96.16%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0799, Test Loss:0.101, Test Accuracy: 97.16%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0491, Test Loss:0.0838, Test Accuracy: 97.74%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0336, Test Loss:0.0793, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0233, Test Loss:0.0744, Test Accuracy: 98.04%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.169, Test Accuracy: 96.39%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.127, Test Loss:0.149, Test Accuracy: 96.98%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.101, Test Loss:0.138, Test Accuracy: 97.47%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0849, Test Loss:0.129, Test Accuracy: 97.62%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0748, Test Loss:0.124, Test Accuracy: 97.71%
---------------------------------------------------------------------------------------------
(33, 98.07000000000001, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.195, Test Loss:0.124, Test Accuracy: 96.02%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0691, Test Loss:0.0836, Test Accuracy: 97.57%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0387, Test Loss:0.0699, Test Accuracy: 97.89%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0222, Test Loss:0.068, Test Accuracy: 98.11%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0136, Test Loss:0.0647, Test Accuracy: 98.10%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(42, 98.1, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.2, Test Loss:0.122, Test Accuracy: 96.27%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0759, Test Loss:0.0951, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0461, Test Loss:0.0852, Test Accuracy: 97.71%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0302, Test Loss:0.077, Test Accuracy: 98.02%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0219, Test Loss:0.0682, Test Accuracy: 98.14%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.166, Test Accuracy: 96.44%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.123, Test Loss:0.144, Test Accuracy: 97.06%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0951, Test Loss:0.125, Test Accuracy: 97.76%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0801, Test Loss:0.116, Test Accuracy: 97.93%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0711, Test Loss:0.112, Test Accuracy: 98.01%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.248, Test Loss:0.137, Test Accuracy: 95.81%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0971, Test Loss:0.0982, Test Accuracy: 96.94%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0632, Test Loss:0.0809, Test Accuracy: 97.56%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0439, Test Loss:0.0703, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0308, Test Loss:0.0662, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.253, Test Loss:0.14, Test Accuracy: 95.69%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.104, Test Accuracy: 96.92%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0682, Test Loss:0.09, Test Accuracy: 97.31%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0491, Test Loss:0.0803, Test Accuracy: 97.68%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0366, Test Loss:0.0783, Test Accuracy: 97.71%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.284, Test Loss:0.177, Test Accuracy: 95.77%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.14, Test Loss:0.144, Test Accuracy: 96.78%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.109, Test Loss:0.131, Test Accuracy: 97.24%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0925, Test Loss:0.124, Test Accuracy: 97.49%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0814, Test Loss:0.118, Test Accuracy: 97.69%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.247, Test Loss:0.133, Test Accuracy: 95.88%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0953, Test Loss:0.0984, Test Accuracy: 97.07%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0619, Test Loss:0.082, Test Accuracy: 97.43%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0436, Test Loss:0.071, Test Accuracy: 97.70%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0316, Test Loss:0.0669, Test Accuracy: 97.79%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.253, Test Loss:0.142, Test Accuracy: 95.71%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.101, Test Loss:0.101, Test Accuracy: 97.11%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0674, Test Loss:0.0861, Test Accuracy: 97.51%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0493, Test Loss:0.0786, Test Accuracy: 97.65%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0377, Test Loss:0.0726, Test Accuracy: 97.87%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.283, Test Loss:0.175, Test Accuracy: 95.68%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.138, Test Loss:0.139, Test Accuracy: 96.97%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.108, Test Loss:0.124, Test Accuracy: 97.51%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0914, Test Loss:0.116, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0812, Test Loss:0.112, Test Accuracy: 97.78%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.249, Test Loss:0.133, Test Accuracy: 95.82%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0941, Test Loss:0.0932, Test Accuracy: 97.21%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0621, Test Loss:0.0767, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0455, Test Loss:0.0681, Test Accuracy: 97.81%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0354, Test Loss:0.0635, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
(43, 98.14, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.254, Test Loss:0.14, Test Accuracy: 95.93%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0999, Test Loss:0.1, Test Accuracy: 97.15%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0669, Test Loss:0.0862, Test Accuracy: 97.49%




Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.252, Test Loss:0.142, Test Accuracy: 95.81%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.1, Test Loss:0.0999, Test Accuracy: 97.02%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0676, Test Loss:0.0849, Test Accuracy: 97.42%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0509, Test Loss:0.0761, Test Accuracy: 97.81%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0408, Test Loss:0.0716, Test Accuracy: 97.93%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.286, Test Loss:0.177, Test Accuracy: 95.77%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.137, Test Loss:0.137, Test Accuracy: 97.04%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.107, Test Loss:0.122, Test Accuracy: 97.49%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.092, Test Loss:0.114, Test Accuracy: 97.73%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0827, Test Loss:0.11, Test Accuracy: 97.82%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.292, Test Loss:0.132, Test Accuracy: 96.07%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.0987, Test Loss:0.1, Test Accuracy: 97.01%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0656, Test Loss:0.102, Test Accuracy: 97.20%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0505, Test Loss:0.0972, Test Accuracy: 97.47%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0411, Test Loss:0.387, Test Accuracy: 93.04%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.273, Test Loss:0.144, Test Accuracy: 95.94%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.105, Test Loss:0.12, Test Accuracy: 96.76%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0723, Test Loss:0.109, Test Accuracy: 97.33%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0579, Test Loss:0.131, Test Accuracy: 96.86%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0551, Test Loss:0.112, Test Accuracy: 97.40%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.302, Test Loss:0.192, Test Accuracy: 96.20%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.165, Test Loss:0.176, Test Accuracy: 96.66%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.141, Test Loss:0.177, Test Accuracy: 96.90%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.13, Test Loss:0.168, Test Accuracy: 97.25%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.122, Test Loss:0.198, Test Accuracy: 96.73%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.227, Test Loss:0.132, Test Accuracy: 96.01%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0821, Test Loss:0.0948, Test Accuracy: 97.34%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0489, Test Loss:0.0865, Test Accuracy: 97.54%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0301, Test Loss:0.0827, Test Accuracy: 97.87%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0205, Test Loss:0.0852, Test Accuracy: 97.68%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.243, Test Loss:0.131, Test Accuracy: 96.35%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0938, Test Loss:0.163, Test Accuracy: 95.71%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0611, Test Loss:0.0968, Test Accuracy: 97.63%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0427, Test Loss:0.0925, Test Accuracy: 97.76%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0327, Test Loss:0.0966, Test Accuracy: 97.65%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.28, Test Loss:0.181, Test Accuracy: 96.41%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.151, Test Loss:0.164, Test Accuracy: 97.07%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.121, Test Loss:0.148, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.104, Test Loss:0.138, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0906, Test Loss:0.14, Test Accuracy: 97.68%
---------------------------------------------------------------------------------------------
(52, 97.92999999999999, {'h1': 700, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.001, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.226, Test Loss:0.121, Test Accuracy: 96.41%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0753, Test Loss:0.0894, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0411, Test Loss:0.0732, Test Accuracy: 97.81%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0237, Test Loss:0.0737, Test Accuracy: 97.95%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0141, Test Loss:0.0631, Test Accuracy: 98.18%
---------------------------------------------------------------------------------------------
We have done: 60 iterations
This new combination yielded the current best model so far.
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.257, Test Loss:0.145, Test Accuracy: 95.89%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0892, Test Loss:0.107, Test Accuracy: 97.08%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0563, Test Loss:0.0927, Test Accuracy: 97.73%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.038, Test Loss:0.0861, Test Accuracy: 98.00%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0282, Test Loss:0.0845, Test Accuracy: 97.91%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.345, Test Loss:0.208, Test Accuracy: 96.11%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.165, Test Loss:0.18, Test Accuracy: 96.85%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.128, Test Loss:0.155, Test Accuracy: 97.49%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.107, Test Loss:0.148, Test Accuracy: 97.48%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0912, Test Loss:0.135, Test Accuracy: 97.82%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.226, Test Loss:0.117, Test Accuracy: 96.16%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.086, Test Loss:0.0899, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0557, Test Loss:0.177, Test Accuracy: 95.17%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0369, Test Loss:0.0818, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.024, Test Loss:0.0762, Test Accuracy: 97.77%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.23, Test Loss:0.129, Test Accuracy: 96.05%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.09, Test Loss:0.0993, Test Accuracy: 97.13%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0592, Test Loss:0.0896, Test Accuracy: 97.43%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0417, Test Loss:0.0833, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0299, Test Loss:0.0792, Test Accuracy: 97.79%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.265, Test Loss:0.164, Test Accuracy: 96.23%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.131, Test Loss:0.135, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.103, Test Loss:0.127, Test Accuracy: 97.45%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0877, Test Loss:0.122, Test Accuracy: 97.76%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0773, Test Loss:0.12, Test Accuracy: 97.80%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.225, Test Loss:0.123, Test Accuracy: 96.10%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0836, Test Loss:0.0851, Test Accuracy: 97.48%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0515, Test Loss:0.0826, Test Accuracy: 97.52%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.034, Test Loss:0.0696, Test Accuracy: 97.90%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0228, Test Loss:0.0643, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.229, Test Loss:0.124, Test Accuracy: 96.31%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0875, Test Loss:0.0911, Test Accuracy: 97.36%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0569, Test Loss:0.0787, Test Accuracy: 97.75%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0397, Test Loss:0.0729, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0289, Test Loss:0.0698, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.268, Test Loss:0.161, Test Accuracy: 96.31%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.129, Test Loss:0.132, Test Accuracy: 97.17%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.101, Test Loss:0.125, Test Accuracy: 97.57%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.085, Test Loss:0.114, Test Accuracy: 97.89%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0749, Test Loss:0.111, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.229, Test Loss:0.121, Test Accuracy: 96.27%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0812, Test Loss:0.0864, Test Accuracy: 97.33%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0503, Test Loss:0.0717, Test Accuracy: 97.75%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0338, Test Loss:0.0657, Test Accuracy: 97.92%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.024, Test Loss:0.0624, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.229, Test Loss:0.125, Test Accuracy: 96.12%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0846, Test Loss:0.0923, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0547, Test Loss:0.0758, Test Accuracy: 97.81%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0388, Test Loss:0.0693, Test Accuracy: 98.06%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0296, Test Loss:0.0664, Test Accuracy: 98.07%
---------------------------------------------------------------------------------------------
We have done: 70 iterations
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.268, Test Loss:0.164, Test Accuracy: 96.14%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.127, Test Loss:0.133, Test Accuracy: 97.16%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0991, Test Loss:0.122, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0846, Test Loss:0.116, Test Accuracy: 97.74%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0758, Test Loss:0.11, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.326, Test Loss:0.181, Test Accuracy: 94.51%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.139, Test Loss:0.12, Test Accuracy: 96.33%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0934, Test Loss:0.0956, Test Accuracy: 97.04%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.069, Test Loss:0.0829, Test Accuracy: 97.40%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0533, Test Loss:0.0757, Test Accuracy: 97.59%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.326, Test Loss:0.185, Test Accuracy: 94.51%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.142, Test Loss:0.123, Test Accuracy: 96.30%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0969, Test Loss:0.1, Test Accuracy: 97.06%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0735, Test Loss:0.0877, Test Accuracy: 97.46%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0584, Test Loss:0.0806, Test Accuracy: 97.67%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.362, Test Loss:0.219, Test Accuracy: 94.43%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.179, Test Loss:0.161, Test Accuracy: 96.21%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.136, Test Loss:0.138, Test Accuracy: 97.01%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.114, Test Loss:0.126, Test Accuracy: 97.42%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0995, Test Loss:0.119, Test Accuracy: 97.59%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.324, Test Loss:0.182, Test Accuracy: 94.43%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.14, Test Loss:0.122, Test Accuracy: 96.09%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0971, Test Loss:0.0991, Test Accuracy: 96.96%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0744, Test Loss:0.0859, Test Accuracy: 97.34%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0598, Test Loss:0.0772, Test Accuracy: 97.59%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.328, Test Loss:0.185, Test Accuracy: 94.57%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.143, Test Loss:0.126, Test Accuracy: 96.16%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.1, Test Loss:0.101, Test Accuracy: 97.00%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0774, Test Loss:0.0879, Test Accuracy: 97.48%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0632, Test Loss:0.0804, Test Accuracy: 97.67%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.357, Test Loss:0.217, Test Accuracy: 94.60%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.178, Test Loss:0.16, Test Accuracy: 96.22%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.137, Test Loss:0.138, Test Accuracy: 97.06%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.116, Test Loss:0.126, Test Accuracy: 97.43%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.103, Test Loss:0.119, Test Accuracy: 97.68%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.326, Test Loss:0.185, Test Accuracy: 94.33%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.14, Test Loss:0.126, Test Accuracy: 96.06%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0995, Test Loss:0.103, Test Accuracy: 96.83%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0787, Test Loss:0.0904, Test Accuracy: 97.25%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0659, Test Loss:0.0828, Test Accuracy: 97.56%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.332, Test Loss:0.186, Test Accuracy: 94.59%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.147, Test Loss:0.129, Test Accuracy: 96.21%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.105, Test Loss:0.107, Test Accuracy: 96.87%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0839, Test Loss:0.0947, Test Accuracy: 97.24%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.071, Test Loss:0.0874, Test Accuracy: 97.48%
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:905010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.363, Test Loss:0.221, Test Accuracy: 94.40%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.181, Test Loss:0.164, Test Accuracy: 96.10%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.142, Test Loss:0.141, Test Accuracy: 96.89%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.123, Test Loss:0.13, Test Accuracy: 97.22%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.111, Test Loss:0.123, Test Accuracy: 97.48%
---------------------------------------------------------------------------------------------
We have done: 80 iterations
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.825, Test Loss:0.639, Test Accuracy: 81.92%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.533, Test Loss:0.67, Test Accuracy: 86.90%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(60, 98.18, {'h1': 700, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910




Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.124, Test Accuracy: 96.34%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0841, Test Loss:0.0981, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0532, Test Loss:0.0911, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.042, Test Loss:0.112, Test Accuracy: 97.09%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0363, Test Loss:0.124, Test Accuracy: 96.92%
---------------------------------------------------------------------------------------------
We have done: 90 iterations
This new combination yielded the current best model so far.
(90, 96.92, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.132, Test Accuracy: 96.26%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0928, Test Loss:0.123, Test Accuracy: 96.56%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0703, Test Loss:0.0991, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0575, Test Loss:0.109, Test Accuracy: 97.38%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0515, Test Loss:0.107, Test Accuracy: 97.75%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(91, 97.75, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.257, Test Loss:0.223, Test Accuracy: 95.38%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.165, Test Loss:0.175, Test Accuracy: 97.02%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.142, Test Loss:0.184, Test Accuracy: 96.86%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.137, Test Loss:0.177, Test Accuracy: 97.18%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.133, Test Loss:0.196, Test Accuracy: 96.74%
---------------------------------------------------------------------------------------------
(91, 97.75, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.129, Test Accuracy: 96.00%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0773, Test Loss:0.12, Test Accuracy: 96.34%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0451, Test Loss:0.0915, Test Accuracy: 97.45%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0284, Test Loss:0.0942, Test Accuracy: 97.48%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0189, Test Loss:0.0808, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(93, 97.89999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.203, Test Loss:0.132, Test Accuracy: 96.39%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.089, Test Loss:0.125, Test Accuracy: 96.62%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0592, Test Loss:0.0994, Test Accuracy: 97.56%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0413, Test Loss:0.0962, Test Accuracy: 97.73%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0334, Test Loss:0.0873, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(94, 98.0, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.255, Test Loss:0.198, Test Accuracy: 96.02%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.155, Test Loss:0.177, Test Accuracy: 96.72%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.127, Test Loss:0.155, Test Accuracy: 97.38%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.108, Test Loss:0.161, Test Accuracy: 97.16%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0961, Test Loss:0.157, Test Accuracy: 97.07%
---------------------------------------------------------------------------------------------
(94, 98.0, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.128, Test Accuracy: 96.20%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0726, Test Loss:0.0972, Test Accuracy: 97.15%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0387, Test Loss:0.0765, Test Accuracy: 97.88%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0211, Test Loss:0.0687, Test Accuracy: 98.09%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0114, Test Loss:0.0659, Test Accuracy: 98.21%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(96, 98.21, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.135, Test Accuracy: 96.33%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.083, Test Loss:0.0983, Test Accuracy: 97.36%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0505, Test Loss:0.0926, Test Accuracy: 97.69%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0342, Test Loss:0.0845, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0248, Test Loss:0.0739, Test Accuracy: 98.27%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(97, 98.27, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.255, Test Loss:0.204, Test Accuracy: 95.84%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.148, Test Loss:0.163, Test Accuracy: 97.06%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.115, Test Loss:0.153, Test Accuracy: 97.39%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0938, Test Loss:0.131, Test Accuracy: 97.96%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0806, Test Loss:0.125, Test Accuracy: 97.92%
---------------------------------------------------------------------------------------------
(97, 98.27, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.198, Test Loss:0.119, Test Accuracy: 96.30%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0781, Test Loss:0.101, Test Accuracy: 96.96%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0482, Test Loss:0.0838, Test Accuracy: 97.56%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0312, Test Loss:0.0712, Test Accuracy: 97.82%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0216, Test Loss:0.0735, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(97, 98.27, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.205, Test Loss:0.125, Test Accuracy: 96.39%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0844, Test Loss:0.0951, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0549, Test Loss:0.0917, Test Accuracy: 97.43%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0387, Test Loss:0.0782, Test Accuracy: 97.89%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0289, Test Loss:0.0788, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
We have done: 100 iterations
(97, 98.27, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.243, Test Loss:0.166, Test Accuracy: 96.26%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.128, Test Loss:0.148, Test Accuracy: 96.95%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.134, Test Accuracy: 97.45%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0873, Test Loss:0.133, Test Accuracy: 97.58%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0771, Test Loss:0.126, Test Accuracy: 97.70%
---------------------------------------------------------------------------------------------
(97, 98.27, {'h1': 700, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.202, Test Loss:0.124, Test Accuracy: 96.05%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0765, Test Loss:0.0968, Test Accuracy: 97.10%

---------------------------------------------------------------------------




Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.264, Test Loss:0.171, Test Accuracy: 94.79%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.123, Test Loss:0.13, Test Accuracy: 96.36%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0908, Test Loss:0.145, Test Accuracy: 96.10%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0762, Test Loss:0.157, Test Accuracy: 96.02%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0765, Test Loss:0.194, Test Accuracy: 95.83%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(108, 95.83, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.27, Test Loss:0.182, Test Accuracy: 95.09%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.14, Test Loss:0.156, Test Accuracy: 96.34%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.118, Test Loss:0.179, Test Accuracy: 95.68%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.108, Test Loss:0.149, Test Accuracy: 97.17%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.106, Test Loss:0.184, Test Accuracy: 96.63%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(109, 96.63000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.346, Test Loss:0.277, Test Accuracy: 94.76%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.247, Test Loss:0.272, Test Accuracy: 95.69%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.234, Test Loss:0.242, Test Accuracy: 96.64%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.234, Test Loss:0.271, Test Accuracy: 96.11%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.232, Test Loss:0.358, Test Accuracy: 93.11%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
We have done: 110 iterations
(109, 96.63000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.253, Test Loss:0.215, Test Accuracy: 94.12%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.11, Test Loss:0.126, Test Accuracy: 96.27%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0694, Test Loss:0.119, Test Accuracy: 97.01%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0493, Test Loss:0.125, Test Accuracy: 96.73%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0345, Test Loss:0.107, Test Accuracy: 97.57%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(111, 97.57000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.266, Test Loss:0.181, Test Accuracy: 95.03%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.129, Test Loss:0.141, Test Accuracy: 96.71%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.092, Test Loss:0.127, Test Accuracy: 97.15%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0735, Test Loss:0.129, Test Accuracy: 97.18%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0609, Test Loss:0.114, Test Accuracy: 97.54%
---------------------------------------------------------------------------------------------
(111, 97.57000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.349, Test Loss:0.299, Test Accuracy: 94.31%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.226, Test Loss:0.242, Test Accuracy: 95.96%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.199, Test Loss:0.215, Test Accuracy: 96.80%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.169, Test Loss:0.201, Test Accuracy: 96.96%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.146, Test Loss:0.195, Test Accuracy: 96.84%
---------------------------------------------------------------------------------------------
(111, 97.57000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.259, Test Loss:0.258, Test Accuracy: 92.66%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(111, 97.57000000000001, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.265, Test Loss:0.215, Test Accuracy: 94.08%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.117, Test Loss:0.141, Test Accuracy: 96.55%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.074, Test Loss:0.119, Test Accuracy: 97.37%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0513, Test Loss:0.104, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.038, Test Loss:0.094, Test Accuracy: 98.23%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.384, Test Loss:0.282, Test Accuracy: 94.94%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.218, Test Loss:0.245, Test Accuracy: 95.78%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.169, Test Loss:0.197, Test Accuracy: 96.79%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.135, Test Loss:0.177, Test Accuracy: 97.15%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.11, Test Loss:0.151, Test Accuracy: 97.67%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.195, Test Loss:0.116, Test Accuracy: 96.45%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.077, Test Loss:0.103, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0475, Test Loss:0.0821, Test Accuracy: 97.71%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0318, Test Loss:0.0723, Test Accuracy: 98.01%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.023, Test Loss:0.0862, Test Accuracy: 97.82%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.201, Test Loss:0.123, Test Accuracy: 96.38%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.084, Test Loss:0.103, Test Accuracy: 97.10%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0559, Test Loss:0.0887, Test Accuracy: 97.65%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0401, Test Loss:0.0876, Test Accuracy: 97.84%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0319, Test Loss:0.0869, Test Accuracy: 97.95%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.17, Test Accuracy: 96.36%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.133, Test Loss:0.157, Test Accuracy: 96.95%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.108, Test Loss:0.153, Test Accuracy: 96.95%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0952, Test Loss:0.137, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0869, Test Loss:0.133, Test Accuracy: 97.56%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.193, Test Loss:0.116, Test Accuracy: 96.53%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0734, Test Loss:0.093, Test Accuracy: 97.39%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0417, Test Loss:0.0707, Test Accuracy: 97.91%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0256, Test Loss:0.0723, Test Accuracy: 97.88%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0155, Test Loss:0.0699, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
We have done: 120 iterations
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.2, Test Loss:0.121, Test Accuracy: 96.54%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0795, Test Loss:0.1, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0502, Test Loss:0.0793, Test Accuracy: 97.89%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0339, Test Loss:0.0852, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0243, Test Loss:0.0743, Test Accuracy: 98.04%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.239, Test Loss:0.17, Test Accuracy: 96.20%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.127, Test Loss:0.153, Test Accuracy: 96.89%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.101, Test Loss:0.134, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0848, Test Loss:0.124, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0745, Test Loss:0.117, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.121, Test Accuracy: 96.30%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0706, Test Loss:0.0868, Test Accuracy: 97.32%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0387, Test Loss:0.0756, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0227, Test Loss:0.0699, Test Accuracy: 97.98%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.014, Test Loss:0.0634, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.199, Test Loss:0.131, Test Accuracy: 96.15%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0758, Test Loss:0.0892, Test Accuracy: 97.35%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0461, Test Loss:0.0786, Test Accuracy: 97.81%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0301, Test Loss:0.0718, Test Accuracy: 98.01%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0217, Test Loss:0.0702, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.24, Test Loss:0.166, Test Accuracy: 96.18%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.124, Test Loss:0.14, Test Accuracy: 97.17%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0955, Test Loss:0.124, Test Accuracy: 97.74%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0803, Test Loss:0.118, Test Accuracy: 97.83%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0711, Test Loss:0.112, Test Accuracy: 97.95%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.248, Test Loss:0.139, Test Accuracy: 95.74%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0975, Test Loss:0.0951, Test Accuracy: 97.03%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0629, Test Loss:0.0801, Test Accuracy: 97.50%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0432, Test Loss:0.0755, Test Accuracy: 97.65%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0301, Test Loss:0.073, Test Accuracy: 97.77%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.251, Test Loss:0.146, Test Accuracy: 95.58%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.105, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0686, Test Loss:0.0891, Test Accuracy: 97.44%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.05, Test Loss:0.0803, Test Accuracy: 97.68%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0374, Test Loss:0.0771, Test Accuracy: 97.72%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.282, Test Loss:0.178, Test Accuracy: 95.78%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.14, Test Loss:0.141, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.109, Test Loss:0.129, Test Accuracy: 97.38%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0926, Test Loss:0.123, Test Accuracy: 97.51%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0816, Test Loss:0.119, Test Accuracy: 97.70%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.244, Test Loss:0.137, Test Accuracy: 95.84%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0958, Test Loss:0.0967, Test Accuracy: 97.09%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0619, Test Loss:0.0817, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0433, Test Loss:0.0745, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0315, Test Loss:0.0691, Test Accuracy: 97.85%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.248, Test Loss:0.138, Test Accuracy: 95.88%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0997, Test Loss:0.0991, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0663, Test Loss:0.0851, Test Accuracy: 97.50%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0485, Test Loss:0.0765, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.037, Test Loss:0.0715, Test Accuracy: 97.80%
---------------------------------------------------------------------------------------------
We have done: 130 iterations
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.279, Test Loss:0.175, Test Accuracy: 95.77%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.137, Test Loss:0.138, Test Accuracy: 97.03%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.107, Test Loss:0.125, Test Accuracy: 97.42%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0912, Test Loss:0.117, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0808, Test Loss:0.111, Test Accuracy: 97.79%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.246, Test Loss:0.141, Test Accuracy: 95.44%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0951, Test Loss:0.0999, Test Accuracy: 96.94%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0623, Test Loss:0.0811, Test Accuracy: 97.45%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0453, Test Loss:0.071, Test Accuracy: 97.70%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0349, Test Loss:0.0664, Test Accuracy: 97.92%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.248, Test Loss:0.135, Test Accuracy: 96.02%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0991, Test Loss:0.0973, Test Accuracy: 97.21%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0667, Test Loss:0.0831, Test Accuracy: 97.51%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0499, Test Loss:0.0752, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0398, Test Loss:0.0705, Test Accuracy: 97.89%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.281, Test Loss:0.174, Test Accuracy: 95.79%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.137, Test Loss:0.137, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.107, Test Loss:0.121, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0914, Test Loss:0.113, Test Accuracy: 97.83%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0822, Test Loss:0.109, Test Accuracy: 97.93%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.24, Test Loss:0.134, Test Accuracy: 96.06%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.0893, Test Loss:0.0974, Test Accuracy: 97.04%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0587, Test Loss:0.0936, Test Accuracy: 97.20%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0424, Test Loss:0.0973, Test Accuracy: 97.44%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0347, Test Loss:0.103, Test Accuracy: 97.28%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.276, Test Loss:0.132, Test Accuracy: 96.24%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.106, Test Loss:0.116, Test Accuracy: 96.84%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0749, Test Loss:0.119, Test Accuracy: 97.10%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0615, Test Loss:0.105, Test Accuracy: 97.43%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0518, Test Loss:0.102, Test Accuracy: 97.70%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.295, Test Loss:0.21, Test Accuracy: 95.42%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.164, Test Loss:0.184, Test Accuracy: 96.50%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.143, Test Loss:0.176, Test Accuracy: 96.82%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.131, Test Loss:0.182, Test Accuracy: 96.78%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.125, Test Loss:0.176, Test Accuracy: 96.98%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.222, Test Loss:0.167, Test Accuracy: 94.92%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0802, Test Loss:0.0854, Test Accuracy: 97.48%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0457, Test Loss:0.0777, Test Accuracy: 97.85%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0279, Test Loss:0.0731, Test Accuracy: 98.19%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0182, Test Loss:0.0832, Test Accuracy: 97.85%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.235, Test Loss:0.16, Test Accuracy: 95.52%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0903, Test Loss:0.0998, Test Accuracy: 97.23%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0582, Test Loss:0.0977, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0413, Test Loss:0.0907, Test Accuracy: 97.79%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0306, Test Loss:0.0897, Test Accuracy: 97.95%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.29, Test Loss:0.196, Test Accuracy: 96.06%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.154, Test Loss:0.169, Test Accuracy: 96.94%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.125, Test Loss:0.156, Test Accuracy: 97.10%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.105, Test Loss:0.151, Test Accuracy: 97.48%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0919, Test Loss:0.134, Test Accuracy: 97.78%
---------------------------------------------------------------------------------------------
We have done: 140 iterations
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.277, Test Loss:0.119, Test Accuracy: 96.48%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0841, Test Loss:0.106, Test Accuracy: 97.07%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0486, Test Loss:0.0875, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0286, Test Loss:0.0821, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0174, Test Loss:0.0743, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.245, Test Loss:0.137, Test Accuracy: 96.01%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0888, Test Loss:0.101, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0548, Test Loss:0.0853, Test Accuracy: 97.93%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0368, Test Loss:0.0849, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0272, Test Loss:0.0807, Test Accuracy: 98.12%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.291, Test Loss:0.207, Test Accuracy: 95.73%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.148, Test Loss:0.159, Test Accuracy: 97.04%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.114, Test Loss:0.139, Test Accuracy: 97.66%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0934, Test Loss:0.127, Test Accuracy: 97.98%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0807, Test Loss:0.12, Test Accuracy: 98.06%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.222, Test Loss:0.116, Test Accuracy: 96.50%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0845, Test Loss:0.0933, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0535, Test Loss:0.0824, Test Accuracy: 97.55%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0352, Test Loss:0.0828, Test Accuracy: 97.64%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0233, Test Loss:0.0772, Test Accuracy: 97.72%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.226, Test Loss:0.123, Test Accuracy: 96.31%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0894, Test Loss:0.0899, Test Accuracy: 97.40%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.06, Test Loss:0.173, Test Accuracy: 95.53%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0419, Test Loss:0.0808, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0294, Test Loss:0.0791, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.262, Test Loss:0.161, Test Accuracy: 96.35%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.131, Test Loss:0.136, Test Accuracy: 97.24%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.103, Test Loss:0.123, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0874, Test Loss:0.117, Test Accuracy: 97.74%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0768, Test Loss:0.114, Test Accuracy: 97.82%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.224, Test Loss:0.125, Test Accuracy: 96.11%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0828, Test Loss:0.093, Test Accuracy: 97.11%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0515, Test Loss:0.0753, Test Accuracy: 97.64%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0336, Test Loss:0.0709, Test Accuracy: 97.74%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0226, Test Loss:0.0687, Test Accuracy: 97.89%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.227, Test Loss:0.125, Test Accuracy: 96.24%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0881, Test Loss:0.0893, Test Accuracy: 97.41%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0573, Test Loss:0.0806, Test Accuracy: 97.71%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0401, Test Loss:0.0742, Test Accuracy: 97.87%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0294, Test Loss:0.0723, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.261, Test Loss:0.164, Test Accuracy: 96.08%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.127, Test Loss:0.131, Test Accuracy: 97.40%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0995, Test Loss:0.12, Test Accuracy: 97.65%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0842, Test Loss:0.114, Test Accuracy: 97.86%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0744, Test Loss:0.11, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.223, Test Loss:0.122, Test Accuracy: 96.13%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0801, Test Loss:0.0845, Test Accuracy: 97.45%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0497, Test Loss:0.071, Test Accuracy: 97.86%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0334, Test Loss:0.0645, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0241, Test Loss:0.062, Test Accuracy: 98.10%
---------------------------------------------------------------------------------------------
We have done: 150 iterations
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.225, Test Loss:0.128, Test Accuracy: 96.12%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0844, Test Loss:0.0883, Test Accuracy: 97.35%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0545, Test Loss:0.0752, Test Accuracy: 97.80%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0388, Test Loss:0.0713, Test Accuracy: 97.98%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0298, Test Loss:0.0698, Test Accuracy: 97.96%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.263, Test Loss:0.168, Test Accuracy: 95.99%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.126, Test Loss:0.131, Test Accuracy: 97.32%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0977, Test Loss:0.118, Test Accuracy: 97.75%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.083, Test Loss:0.111, Test Accuracy: 97.96%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0744, Test Loss:0.107, Test Accuracy: 98.03%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.323, Test Loss:0.178, Test Accuracy: 94.58%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.138, Test Loss:0.117, Test Accuracy: 96.37%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0937, Test Loss:0.0936, Test Accuracy: 97.19%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0698, Test Loss:0.0811, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0544, Test Loss:0.0743, Test Accuracy: 97.59%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.325, Test Loss:0.189, Test Accuracy: 94.28%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.143, Test Loss:0.126, Test Accuracy: 96.19%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0988, Test Loss:0.101, Test Accuracy: 97.06%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0749, Test Loss:0.0888, Test Accuracy: 97.45%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0594, Test Loss:0.0809, Test Accuracy: 97.65%
---------------------------------------------------------------------------------------------
(115, 98.22999999999999, {'h1': 700, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (700, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:833910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.358, Test Loss:0.219, Test Accuracy: 94.45%




Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.199, Test Loss:0.133, Test Accuracy: 95.89%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0846, Test Loss:0.119, Test Accuracy: 96.56%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0553, Test Loss:0.0899, Test Accuracy: 97.37%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0419, Test Loss:0.103, Test Accuracy: 97.35%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0355, Test Loss:0.108, Test Accuracy: 97.31%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(171, 97.31, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.205, Test Loss:0.145, Test Accuracy: 96.05%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0945, Test Loss:0.125, Test Accuracy: 96.70%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0696, Test Loss:0.12, Test Accuracy: 97.05%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0602, Test Loss:0.111, Test Accuracy: 97.56%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0536, Test Loss:0.123, Test Accuracy: 97.03%
---------------------------------------------------------------------------------------------
(171, 97.31, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.257, Test Loss:0.19, Test Accuracy: 96.42%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.165, Test Loss:0.186, Test Accuracy: 96.72%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.145, Test Loss:0.179, Test Accuracy: 96.90%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.137, Test Loss:0.176, Test Accuracy: 97.06%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.134, Test Loss:0.177, Test Accuracy: 97.23%
---------------------------------------------------------------------------------------------
(171, 97.31, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.198, Test Loss:0.133, Test Accuracy: 95.87%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0794, Test Loss:0.0933, Test Accuracy: 97.21%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0463, Test Loss:0.0812, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.03, Test Loss:0.0862, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0191, Test Loss:0.0857, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(174, 97.89999999999999, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.206, Test Loss:0.134, Test Accuracy: 96.38%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0894, Test Loss:0.105, Test Accuracy: 97.23%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0583, Test Loss:0.0999, Test Accuracy: 97.69%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0431, Test Loss:0.103, Test Accuracy: 97.80%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0323, Test Loss:0.103, Test Accuracy: 97.86%
---------------------------------------------------------------------------------------------
(174, 97.89999999999999, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.256, Test Loss:0.195, Test Accuracy: 96.16%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.154, Test Loss:0.184, Test Accuracy: 96.82%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.128, Test Loss:0.154, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.111, Test Loss:0.154, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0972, Test Loss:0.137, Test Accuracy: 97.98%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(176, 97.98, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0.01}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.197, Test Loss:0.135, Test Accuracy: 95.87%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.074, Test Loss:0.105, Test Accuracy: 96.84%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0382, Test Loss:0.0775, Test Accuracy: 97.83%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0221, Test Loss:0.0757, Test Accuracy: 97.96%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0117, Test Loss:0.0693, Test Accuracy: 98.15%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.127, Test Accuracy: 96.32%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0838, Test Loss:0.118, Test Accuracy: 96.70%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0517, Test Loss:0.0842, Test Accuracy: 97.76%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0342, Test Loss:0.0815, Test Accuracy: 97.94%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0256, Test Loss:0.0862, Test Accuracy: 97.92%
---------------------------------------------------------------------------------------------
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.258, Test Loss:0.183, Test Accuracy: 96.52%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.149, Test Loss:0.171, Test Accuracy: 96.97%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.115, Test Loss:0.146, Test Accuracy: 97.57%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0953, Test Loss:0.142, Test Accuracy: 97.52%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0819, Test Loss:0.127, Test Accuracy: 98.01%
---------------------------------------------------------------------------------------------
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.204, Test Loss:0.122, Test Accuracy: 96.19%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0808, Test Loss:0.092, Test Accuracy: 97.05%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0507, Test Loss:0.0742, Test Accuracy: 97.79%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0331, Test Loss:0.0798, Test Accuracy: 97.68%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0213, Test Loss:0.0739, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
We have done: 180 iterations
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.206, Test Loss:0.129, Test Accuracy: 96.06%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0855, Test Loss:0.11, Test Accuracy: 96.91%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0565, Test Loss:0.0917, Test Accuracy: 97.50%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0397, Test Loss:0.0839, Test Accuracy: 97.71%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0298, Test Loss:0.0913, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.246, Test Loss:0.168, Test Accuracy: 95.99%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.13, Test Loss:0.15, Test Accuracy: 96.87%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.104, Test Loss:0.138, Test Accuracy: 97.22%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0884, Test Loss:0.132, Test Accuracy: 97.46%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.079, Test Loss:0.128, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
(177, 98.15, {'h1': 500, 'h2': 500, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010

---------------------------------------------------------------------------




Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.275, Test Loss:0.285, Test Accuracy: 91.35%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(189, 91.35, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.266, Test Loss:0.197, Test Accuracy: 94.54%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.143, Test Loss:0.169, Test Accuracy: 96.09%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.115, Test Loss:0.15, Test Accuracy: 97.05%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.114, Test Loss:0.182, Test Accuracy: 96.50%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.114, Test Loss:0.167, Test Accuracy: 96.86%
---------------------------------------------------------------------------------------------
We have done: 190 iterations
This new combination yielded the current best model so far.
(190, 96.86, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.348, Test Loss:0.271, Test Accuracy: 94.97%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.249, Test Loss:0.261, Test Accuracy: 95.50%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.228, Test Loss:0.27, Test Accuracy: 95.80%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.225, Test Loss:0.245, Test Accuracy: 96.58%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.227, Test Loss:0.419, Test Accuracy: 91.33%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(190, 96.86, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.259, Test Loss:0.16, Test Accuracy: 95.11%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.11, Test Loss:0.124, Test Accuracy: 96.59%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0728, Test Loss:0.116, Test Accuracy: 96.84%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0472, Test Loss:0.106, Test Accuracy: 97.37%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0293, Test Loss:0.103, Test Accuracy: 97.65%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(192, 97.65, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.274, Test Loss:0.216, Test Accuracy: 94.06%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.13, Test Loss:0.142, Test Accuracy: 96.56%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0942, Test Loss:0.129, Test Accuracy: 97.19%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0715, Test Loss:0.122, Test Accuracy: 97.58%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.06, Test Loss:0.116, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(193, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.351, Test Loss:0.295, Test Accuracy: 94.29%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.226, Test Loss:0.243, Test Accuracy: 95.91%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.191, Test Loss:0.235, Test Accuracy: 96.07%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.169, Test Loss:0.207, Test Accuracy: 96.88%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.146, Test Loss:0.187, Test Accuracy: 97.15%
---------------------------------------------------------------------------------------------
(193, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.255, Test Loss:0.165, Test Accuracy: 95.02%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0964, Test Loss:0.115, Test Accuracy: 96.68%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0558, Test Loss:0.106, Test Accuracy: 97.08%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0303, Test Loss:0.0926, Test Accuracy: 97.53%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0168, Test Loss:0.0848, Test Accuracy: 97.87%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(195, 97.87, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.27, Test Loss:0.217, Test Accuracy: 94.10%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.119, Test Loss:0.141, Test Accuracy: 96.49%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0772, Test Loss:0.118, Test Accuracy: 97.44%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0533, Test Loss:0.11, Test Accuracy: 97.66%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0399, Test Loss:0.0955, Test Accuracy: 98.08%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.355, Test Loss:0.297, Test Accuracy: 94.33%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.211, Test Loss:0.205, Test Accuracy: 96.71%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.165, Test Loss:0.183, Test Accuracy: 97.12%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.132, Test Loss:0.166, Test Accuracy: 97.30%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.11, Test Loss:0.143, Test Accuracy: 97.80%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.197, Test Loss:0.124, Test Accuracy: 96.04%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0776, Test Loss:0.0934, Test Accuracy: 97.04%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0492, Test Loss:0.0856, Test Accuracy: 97.47%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0323, Test Loss:0.0921, Test Accuracy: 97.46%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0227, Test Loss:0.0786, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.203, Test Loss:0.129, Test Accuracy: 96.19%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0847, Test Loss:0.111, Test Accuracy: 96.83%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0567, Test Loss:0.105, Test Accuracy: 97.09%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0419, Test Loss:0.0971, Test Accuracy: 97.39%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0347, Test Loss:0.11, Test Accuracy: 97.11%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.165, Test Accuracy: 96.43%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.132, Test Loss:0.152, Test Accuracy: 97.04%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.108, Test Loss:0.143, Test Accuracy: 97.37%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0949, Test Loss:0.155, Test Accuracy: 97.14%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0889, Test Loss:0.141, Test Accuracy: 97.38%
---------------------------------------------------------------------------------------------
We have done: 200 iterations
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.2, Test Loss:0.137, Test Accuracy: 95.82%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0747, Test Loss:0.113, Test Accuracy: 96.58%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0436, Test Loss:0.0823, Test Accuracy: 97.40%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0265, Test Loss:0.0785, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.017, Test Loss:0.0849, Test Accuracy: 97.62%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.2, Test Loss:0.12, Test Accuracy: 96.45%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0814, Test Loss:0.0937, Test Accuracy: 97.42%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0511, Test Loss:0.0825, Test Accuracy: 97.65%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0348, Test Loss:0.0763, Test Accuracy: 98.06%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.025, Test Loss:0.0799, Test Accuracy: 98.04%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.242, Test Loss:0.168, Test Accuracy: 96.51%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.128, Test Loss:0.152, Test Accuracy: 96.88%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.101, Test Loss:0.139, Test Accuracy: 97.23%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0853, Test Loss:0.13, Test Accuracy: 97.60%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0751, Test Loss:0.122, Test Accuracy: 97.78%
---------------------------------------------------------------------------------------------
(196, 98.08, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.197, Test Loss:0.116, Test Accuracy: 96.25%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0704, Test Loss:0.0856, Test Accuracy: 97.48%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0398, Test Loss:0.0706, Test Accuracy: 97.98%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0238, Test Loss:0.0666, Test Accuracy: 98.12%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0148, Test Loss:0.062, Test Accuracy: 98.22%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.199, Test Loss:0.121, Test Accuracy: 96.34%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0746, Test Loss:0.0945, Test Accuracy: 97.37%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0453, Test Loss:0.0774, Test Accuracy: 97.86%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0306, Test Loss:0.0752, Test Accuracy: 98.06%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0221, Test Loss:0.0691, Test Accuracy: 98.20%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.242, Test Loss:0.168, Test Accuracy: 96.27%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.125, Test Loss:0.142, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0967, Test Loss:0.126, Test Accuracy: 97.72%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.081, Test Loss:0.121, Test Accuracy: 97.88%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0719, Test Loss:0.115, Test Accuracy: 98.01%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.249, Test Loss:0.139, Test Accuracy: 95.72%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0984, Test Loss:0.101, Test Accuracy: 96.85%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0641, Test Loss:0.0858, Test Accuracy: 97.42%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0448, Test Loss:0.0737, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0318, Test Loss:0.0698, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.253, Test Loss:0.141, Test Accuracy: 95.79%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.102, Test Accuracy: 97.03%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.069, Test Loss:0.0934, Test Accuracy: 97.38%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0504, Test Loss:0.0882, Test Accuracy: 97.59%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0377, Test Loss:0.083, Test Accuracy: 97.77%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.29, Test Loss:0.178, Test Accuracy: 95.56%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.142, Test Loss:0.143, Test Accuracy: 96.82%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.111, Test Loss:0.131, Test Accuracy: 97.37%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0935, Test Loss:0.125, Test Accuracy: 97.49%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0823, Test Loss:0.119, Test Accuracy: 97.64%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.252, Test Loss:0.14, Test Accuracy: 95.74%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0981, Test Loss:0.101, Test Accuracy: 96.93%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0641, Test Loss:0.085, Test Accuracy: 97.40%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0455, Test Loss:0.0762, Test Accuracy: 97.64%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0336, Test Loss:0.0709, Test Accuracy: 97.78%
---------------------------------------------------------------------------------------------
We have done: 210 iterations
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.252, Test Loss:0.146, Test Accuracy: 95.52%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.101, Test Loss:0.101, Test Accuracy: 97.08%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0679, Test Loss:0.0859, Test Accuracy: 97.52%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0501, Test Loss:0.0785, Test Accuracy: 97.76%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0386, Test Loss:0.0735, Test Accuracy: 97.98%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.289, Test Loss:0.181, Test Accuracy: 95.68%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.14, Test Loss:0.139, Test Accuracy: 97.02%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.109, Test Loss:0.125, Test Accuracy: 97.48%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0924, Test Loss:0.119, Test Accuracy: 97.55%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0819, Test Loss:0.115, Test Accuracy: 97.71%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.247, Test Loss:0.135, Test Accuracy: 95.82%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0967, Test Loss:0.0968, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0636, Test Loss:0.0815, Test Accuracy: 97.47%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0464, Test Loss:0.0718, Test Accuracy: 97.86%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0361, Test Loss:0.0665, Test Accuracy: 97.89%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.252, Test Loss:0.138, Test Accuracy: 95.89%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.1, Test Loss:0.0985, Test Accuracy: 97.06%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0681, Test Loss:0.0854, Test Accuracy: 97.41%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0516, Test Loss:0.0768, Test Accuracy: 97.75%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0414, Test Loss:0.0723, Test Accuracy: 97.87%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.288, Test Loss:0.179, Test Accuracy: 95.59%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.14, Test Loss:0.141, Test Accuracy: 96.99%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.109, Test Loss:0.125, Test Accuracy: 97.44%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0937, Test Loss:0.116, Test Accuracy: 97.71%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0842, Test Loss:0.111, Test Accuracy: 97.84%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.255, Test Loss:0.129, Test Accuracy: 96.43%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.0935, Test Loss:0.105, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0618, Test Loss:0.0937, Test Accuracy: 97.20%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0467, Test Loss:0.0968, Test Accuracy: 97.25%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0351, Test Loss:0.102, Test Accuracy: 97.48%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.245, Test Loss:0.138, Test Accuracy: 95.90%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.101, Test Loss:0.118, Test Accuracy: 96.64%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0726, Test Loss:0.128, Test Accuracy: 96.78%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0558, Test Loss:0.109, Test Accuracy: 97.39%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0481, Test Loss:0.147, Test Accuracy: 97.00%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.329, Test Loss:0.222, Test Accuracy: 95.26%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.175, Test Loss:0.19, Test Accuracy: 96.38%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.149, Test Loss:0.173, Test Accuracy: 97.03%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.135, Test Loss:0.262, Test Accuracy: 94.73%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.129, Test Loss:0.209, Test Accuracy: 96.03%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.273, Test Loss:0.135, Test Accuracy: 96.16%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0932, Test Loss:0.104, Test Accuracy: 97.08%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0586, Test Loss:0.0851, Test Accuracy: 97.64%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0375, Test Loss:0.0805, Test Accuracy: 97.86%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0262, Test Loss:0.0794, Test Accuracy: 97.96%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.249, Test Loss:0.148, Test Accuracy: 95.64%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0955, Test Loss:0.136, Test Accuracy: 96.06%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0638, Test Loss:0.0954, Test Accuracy: 97.74%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0452, Test Loss:0.0919, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0332, Test Loss:0.0942, Test Accuracy: 97.84%
---------------------------------------------------------------------------------------------
We have done: 220 iterations
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.33, Test Loss:0.211, Test Accuracy: 95.77%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.166, Test Loss:0.179, Test Accuracy: 96.83%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.134, Test Loss:0.166, Test Accuracy: 97.26%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.114, Test Loss:0.146, Test Accuracy: 97.58%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0982, Test Loss:0.148, Test Accuracy: 97.61%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.225, Test Loss:0.126, Test Accuracy: 96.05%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0781, Test Loss:0.0919, Test Accuracy: 97.28%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0422, Test Loss:0.0739, Test Accuracy: 97.78%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0241, Test Loss:0.0704, Test Accuracy: 98.02%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0144, Test Loss:0.0681, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.251, Test Loss:0.132, Test Accuracy: 96.33%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0905, Test Loss:0.106, Test Accuracy: 97.01%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0568, Test Loss:0.1, Test Accuracy: 97.39%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0388, Test Loss:0.0854, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0285, Test Loss:0.0812, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.291, Test Loss:0.208, Test Accuracy: 95.55%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.149, Test Loss:0.155, Test Accuracy: 97.23%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.116, Test Loss:0.151, Test Accuracy: 97.43%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0963, Test Loss:0.13, Test Accuracy: 97.88%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0831, Test Loss:0.125, Test Accuracy: 97.90%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.23, Test Loss:0.117, Test Accuracy: 96.24%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0864, Test Loss:0.0868, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0545, Test Loss:0.0746, Test Accuracy: 97.70%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0368, Test Loss:0.072, Test Accuracy: 97.95%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0246, Test Loss:0.0764, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.229, Test Loss:0.128, Test Accuracy: 96.25%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0907, Test Loss:0.1, Test Accuracy: 97.24%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0597, Test Loss:0.0874, Test Accuracy: 97.47%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0417, Test Loss:0.0843, Test Accuracy: 97.61%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0303, Test Loss:0.0851, Test Accuracy: 97.73%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.271, Test Loss:0.165, Test Accuracy: 96.09%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.134, Test Loss:0.14, Test Accuracy: 97.11%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.106, Test Loss:0.132, Test Accuracy: 97.35%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0902, Test Loss:0.133, Test Accuracy: 97.53%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0797, Test Loss:0.129, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.226, Test Loss:0.122, Test Accuracy: 96.15%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0842, Test Loss:0.0934, Test Accuracy: 97.15%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.053, Test Loss:0.0816, Test Accuracy: 97.46%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0355, Test Loss:0.0737, Test Accuracy: 97.70%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0242, Test Loss:0.0706, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.233, Test Loss:0.129, Test Accuracy: 96.07%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0893, Test Loss:0.0962, Test Accuracy: 97.18%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0579, Test Loss:0.0832, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0407, Test Loss:0.0743, Test Accuracy: 97.82%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0294, Test Loss:0.0705, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.264, Test Loss:0.162, Test Accuracy: 96.24%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.131, Test Loss:0.137, Test Accuracy: 97.18%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.102, Test Loss:0.122, Test Accuracy: 97.68%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0862, Test Loss:0.116, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.076, Test Loss:0.112, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
We have done: 230 iterations
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.226, Test Loss:0.126, Test Accuracy: 96.15%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0813, Test Loss:0.0864, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0504, Test Loss:0.0743, Test Accuracy: 97.68%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0344, Test Loss:0.0702, Test Accuracy: 97.84%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.025, Test Loss:0.0669, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.23, Test Loss:0.125, Test Accuracy: 96.29%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.086, Test Loss:0.0965, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.056, Test Loss:0.0805, Test Accuracy: 97.66%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0401, Test Loss:0.0717, Test Accuracy: 97.89%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0307, Test Loss:0.0688, Test Accuracy: 98.00%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.269, Test Loss:0.163, Test Accuracy: 96.22%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.127, Test Loss:0.128, Test Accuracy: 97.43%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0992, Test Loss:0.118, Test Accuracy: 97.67%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0847, Test Loss:0.111, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0759, Test Loss:0.107, Test Accuracy: 97.87%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.331, Test Loss:0.187, Test Accuracy: 94.47%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.143, Test Loss:0.127, Test Accuracy: 96.23%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0971, Test Loss:0.0998, Test Accuracy: 97.07%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0723, Test Loss:0.0852, Test Accuracy: 97.46%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0561, Test Loss:0.0777, Test Accuracy: 97.64%
---------------------------------------------------------------------------------------------
(204, 98.22, {'h1': 500, 'h2': 500, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.333, Test Loss:0.187, Test Accuracy: 94.38%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.144, Test Loss:0.127, Test Accuracy: 96.17%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.1, Test Loss:0.104, Test Accuracy: 96.93%




Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.334, Test Loss:0.187, Test Accuracy: 94.43%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.145, Test Loss:0.126, Test Accuracy: 96.13%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.1, Test Loss:0.102, Test Accuracy: 96.94%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0761, Test Loss:0.0894, Test Accuracy: 97.35%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0606, Test Loss:0.0821, Test Accuracy: 97.55%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(235, 97.55, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.363, Test Loss:0.221, Test Accuracy: 94.38%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.18, Test Loss:0.162, Test Accuracy: 96.13%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.137, Test Loss:0.139, Test Accuracy: 96.97%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.115, Test Loss:0.127, Test Accuracy: 97.36%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.101, Test Loss:0.121, Test Accuracy: 97.63%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(236, 97.63, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 1, 'regularization': 0.01}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.327, Test Loss:0.188, Test Accuracy: 94.27%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.144, Test Loss:0.126, Test Accuracy: 96.09%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0999, Test Loss:0.1, Test Accuracy: 96.95%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0763, Test Loss:0.0877, Test Accuracy: 97.34%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0612, Test Loss:0.0794, Test Accuracy: 97.66%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.331, Test Loss:0.189, Test Accuracy: 94.36%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.148, Test Loss:0.129, Test Accuracy: 96.09%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.105, Test Loss:0.106, Test Accuracy: 96.98%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0822, Test Loss:0.0928, Test Accuracy: 97.36%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0674, Test Loss:0.0846, Test Accuracy: 97.56%
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.364, Test Loss:0.223, Test Accuracy: 94.33%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.182, Test Loss:0.165, Test Accuracy: 96.04%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.141, Test Loss:0.141, Test Accuracy: 96.91%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.119, Test Loss:0.129, Test Accuracy: 97.24%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.105, Test Loss:0.122, Test Accuracy: 97.50%
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.324, Test Loss:0.183, Test Accuracy: 94.53%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.141, Test Loss:0.124, Test Accuracy: 96.06%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.101, Test Loss:0.0997, Test Accuracy: 96.99%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0795, Test Loss:0.0872, Test Accuracy: 97.33%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0667, Test Loss:0.0801, Test Accuracy: 97.63%
---------------------------------------------------------------------------------------------
We have done: 240 iterations
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.33, Test Loss:0.187, Test Accuracy: 94.48%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.146, Test Loss:0.13, Test Accuracy: 96.21%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.106, Test Loss:0.106, Test Accuracy: 96.97%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0848, Test Loss:0.0935, Test Accuracy: 97.27%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0721, Test Loss:0.0864, Test Accuracy: 97.53%
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 500)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:648010
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.361, Test Loss:0.224, Test Accuracy: 94.29%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.182, Test Loss:0.167, Test Accuracy: 95.96%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.144, Test Loss:0.144, Test Accuracy: 96.76%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.124, Test Loss:0.131, Test Accuracy: 97.26%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.111, Test Loss:0.124, Test Accuracy: 97.55%
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:140: RuntimeWarning: divide by zero encountered in log
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:112: RuntimeWarning: invalid value encountered in subtract
/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:26: RuntimeWarning: invalid value encountered in reduce
  return umr_maximum(a, axis, None, out, keepdims)
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:94: RuntimeWarning: invalid value encountered in less_equal
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:95: RuntimeWarning: invalid value encountered in greater

Epoch: 1, Learning Rate: 1.00e-02, Training Loss:nan, Test Loss:nan, Test Accuracy: 9.80%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.556, Test Loss:0.505, Test Accuracy: 90.97%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.673, Test Loss:0.783, Test Accuracy: 87.75%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:167: RuntimeWarning: invalid value encountered in multiply

Epoch: 1, Learning Rate: 1.00e-02, Training Loss:nan, Test Loss:nan, Test Accuracy: 9.80%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.729, Test Loss:0.674, Test Accuracy: 88.95%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.745, Test Loss:0.831, Test Accuracy: 87.76%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.532, Test Loss:0.532, Test Accuracy: 87.88%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.632, Test Loss:0.781, Test Accuracy: 82.86%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
We have done: 250 iterations
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.726, Test Loss:0.659, Test Accuracy: 89.89%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.198, Test Loss:0.13, Test Accuracy: 96.00%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0867, Test Loss:0.118, Test Accuracy: 96.57%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0576, Test Loss:0.107, Test Accuracy: 97.19%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.043, Test Loss:0.116, Test Accuracy: 97.02%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0358, Test Loss:0.109, Test Accuracy: 97.45%
---------------------------------------------------------------------------------------------
(237, 97.66, {'h1': 500, 'h2': 500, 'mini_batch_size': 32, 'learning_rate': 0.001, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.155, Test Accuracy: 95.58%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0956, Test Loss:0.125, Test Accuracy: 96.72%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0706, Test Loss:0.11, Test Accuracy: 97.17%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.059, Test Loss:0.113, Test Accuracy: 97.35%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0542, Test Loss:0.108, Test Accuracy: 97.71%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(253, 97.71, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.255, Test Loss:0.199, Test Accuracy: 95.64%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.166, Test Loss:0.183, Test Accuracy: 96.57%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.147, Test Loss:0.187, Test Accuracy: 96.66%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.138, Test Loss:0.201, Test Accuracy: 96.64%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.132, Test Loss:0.185, Test Accuracy: 96.92%
---------------------------------------------------------------------------------------------
(253, 97.71, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.198, Test Loss:0.136, Test Accuracy: 95.76%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0809, Test Loss:0.108, Test Accuracy: 96.87%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0471, Test Loss:0.096, Test Accuracy: 97.21%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0296, Test Loss:0.0785, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0192, Test Loss:0.0776, Test Accuracy: 97.97%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(255, 97.97, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.205, Test Loss:0.121, Test Accuracy: 96.60%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0896, Test Loss:0.106, Test Accuracy: 97.14%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0593, Test Loss:0.0961, Test Accuracy: 97.63%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0445, Test Loss:0.088, Test Accuracy: 98.00%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0334, Test Loss:0.1, Test Accuracy: 97.76%
---------------------------------------------------------------------------------------------
(255, 97.97, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.256, Test Loss:0.188, Test Accuracy: 96.43%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.156, Test Loss:0.204, Test Accuracy: 96.05%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.129, Test Loss:0.166, Test Accuracy: 97.15%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.11, Test Loss:0.155, Test Accuracy: 97.36%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0967, Test Loss:0.157, Test Accuracy: 97.21%
---------------------------------------------------------------------------------------------
(255, 97.97, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.197, Test Loss:0.122, Test Accuracy: 96.37%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0742, Test Loss:0.118, Test Accuracy: 96.51%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0403, Test Loss:0.0882, Test Accuracy: 97.39%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0231, Test Loss:0.0806, Test Accuracy: 97.64%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0127, Test Loss:0.0685, Test Accuracy: 98.14%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(258, 98.14, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.204, Test Loss:0.138, Test Accuracy: 96.12%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0846, Test Loss:0.109, Test Accuracy: 96.95%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0527, Test Loss:0.0951, Test Accuracy: 97.62%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0349, Test Loss:0.0833, Test Accuracy: 98.05%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.026, Test Loss:0.079, Test Accuracy: 98.15%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(259, 98.15, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.257, Test Loss:0.198, Test Accuracy: 96.04%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.148, Test Loss:0.172, Test Accuracy: 96.84%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.115, Test Loss:0.155, Test Accuracy: 97.38%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0944, Test Loss:0.136, Test Accuracy: 97.72%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0815, Test Loss:0.132, Test Accuracy: 97.86%
---------------------------------------------------------------------------------------------
We have done: 260 iterations
(259, 98.15, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.202, Test Loss:0.116, Test Accuracy: 96.35%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.079, Test Loss:0.1, Test Accuracy: 96.98%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0499, Test Loss:0.0882, Test Accuracy: 97.43%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0334, Test Loss:0.0848, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0233, Test Loss:0.0976, Test Accuracy: 97.20%
---------------------------------------------------------------------------------------------
(259, 98.15, {'h1': 500, 'h2': 400, 'mini_batch_size': 8, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 8	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.208, Test Loss:0.126, Test Accuracy: 96.20%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0858, Test Loss:0.102, Test Accuracy: 96.98%

---------------------------------------------------------------------------



Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.297, Test Loss:0.223, Test Accuracy: 94.28%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.157, Test Loss:0.221, Test Accuracy: 94.50%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.137, Test Loss:0.165, Test Accuracy: 96.50%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.127, Test Loss:0.193, Test Accuracy: 96.16%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.117, Test Loss:0.179, Test Accuracy: 96.80%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(271, 96.8, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.347, Test Loss:0.266, Test Accuracy: 95.12%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.245, Test Loss:0.279, Test Accuracy: 95.28%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.228, Test Loss:0.253, Test Accuracy: 95.98%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.222, Test Loss:0.278, Test Accuracy: 95.72%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.224, Test Loss:0.261, Test Accuracy: 96.14%
---------------------------------------------------------------------------------------------
(271, 96.8, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 1, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.264, Test Loss:0.223, Test Accuracy: 94.11%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.115, Test Loss:0.123, Test Accuracy: 96.56%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0759, Test Loss:0.127, Test Accuracy: 96.84%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0504, Test Loss:0.12, Test Accuracy: 96.99%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.035, Test Loss:0.117, Test Accuracy: 97.14%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(273, 97.14, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.27, Test Loss:0.171, Test Accuracy: 95.46%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.13, Test Loss:0.15, Test Accuracy: 96.19%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0927, Test Loss:0.3, Test Accuracy: 94.97%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0747, Test Loss:0.138, Test Accuracy: 96.98%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0605, Test Loss:0.125, Test Accuracy: 97.59%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(274, 97.59, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.353, Test Loss:0.285, Test Accuracy: 94.60%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.227, Test Loss:0.245, Test Accuracy: 96.01%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.192, Test Loss:0.22, Test Accuracy: 96.43%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.168, Test Loss:0.222, Test Accuracy: 96.31%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.15, Test Loss:0.186, Test Accuracy: 97.12%
---------------------------------------------------------------------------------------------
(274, 97.59, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.9, 'regularization': 0.001}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.266, Test Loss:0.187, Test Accuracy: 94.66%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.104, Test Loss:0.115, Test Accuracy: 96.60%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0605, Test Loss:0.0957, Test Accuracy: 97.39%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0343, Test Loss:0.0904, Test Accuracy: 97.76%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0184, Test Loss:0.0841, Test Accuracy: 98.07%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.274, Test Loss:0.207, Test Accuracy: 94.75%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.123, Test Loss:0.136, Test Accuracy: 96.66%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0785, Test Loss:0.122, Test Accuracy: 97.20%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.057, Test Loss:0.108, Test Accuracy: 97.64%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.04, Test Loss:0.107, Test Accuracy: 97.80%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.343, Test Loss:0.323, Test Accuracy: 93.82%
Early stop accuracy too low
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.196, Test Loss:0.129, Test Accuracy: 95.93%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.079, Test Loss:0.098, Test Accuracy: 96.96%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0505, Test Loss:0.0842, Test Accuracy: 97.54%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0335, Test Loss:0.0801, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0243, Test Loss:0.0971, Test Accuracy: 97.44%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.201, Test Loss:0.132, Test Accuracy: 95.99%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0852, Test Loss:0.113, Test Accuracy: 96.82%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0571, Test Loss:0.0878, Test Accuracy: 97.68%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0421, Test Loss:0.0848, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0359, Test Loss:0.0929, Test Accuracy: 97.67%
---------------------------------------------------------------------------------------------
We have done: 280 iterations
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.242, Test Loss:0.171, Test Accuracy: 95.98%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.133, Test Loss:0.165, Test Accuracy: 96.55%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.109, Test Loss:0.149, Test Accuracy: 97.21%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0952, Test Loss:0.135, Test Accuracy: 97.59%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0872, Test Loss:0.134, Test Accuracy: 97.75%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.198, Test Loss:0.122, Test Accuracy: 96.22%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0761, Test Loss:0.107, Test Accuracy: 96.60%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0448, Test Loss:0.0828, Test Accuracy: 97.46%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0268, Test Loss:0.0751, Test Accuracy: 97.80%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0158, Test Loss:0.0772, Test Accuracy: 97.93%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.202, Test Loss:0.131, Test Accuracy: 96.22%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0811, Test Loss:0.101, Test Accuracy: 97.10%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0506, Test Loss:0.0835, Test Accuracy: 97.89%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0349, Test Loss:0.0821, Test Accuracy: 97.91%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0243, Test Loss:0.0793, Test Accuracy: 98.05%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.243, Test Loss:0.167, Test Accuracy: 96.29%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.129, Test Loss:0.149, Test Accuracy: 96.98%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.102, Test Loss:0.136, Test Accuracy: 97.46%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0863, Test Loss:0.128, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0755, Test Loss:0.123, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
(276, 98.07000000000001, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.01, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.194, Test Loss:0.117, Test Accuracy: 96.26%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0701, Test Loss:0.0841, Test Accuracy: 97.37%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0394, Test Loss:0.069, Test Accuracy: 97.88%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0234, Test Loss:0.0645, Test Accuracy: 98.06%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0146, Test Loss:0.0608, Test Accuracy: 98.26%
---------------------------------------------------------------------------------------------
This new combination yielded the current best model so far.
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.199, Test Loss:0.129, Test Accuracy: 96.05%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0768, Test Loss:0.091, Test Accuracy: 97.42%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0468, Test Loss:0.0767, Test Accuracy: 97.94%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.031, Test Loss:0.0738, Test Accuracy: 98.06%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0226, Test Loss:0.0711, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.241, Test Loss:0.171, Test Accuracy: 96.12%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.123, Test Loss:0.137, Test Accuracy: 97.28%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0959, Test Loss:0.122, Test Accuracy: 97.76%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0806, Test Loss:0.114, Test Accuracy: 98.00%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0715, Test Loss:0.11, Test Accuracy: 98.13%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.244, Test Loss:0.133, Test Accuracy: 95.94%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.0978, Test Loss:0.0933, Test Accuracy: 97.17%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0639, Test Loss:0.08, Test Accuracy: 97.55%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0444, Test Loss:0.0749, Test Accuracy: 97.58%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0313, Test Loss:0.072, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.255, Test Loss:0.145, Test Accuracy: 95.60%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.104, Test Loss:0.11, Test Accuracy: 96.87%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0703, Test Loss:0.0925, Test Accuracy: 97.34%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0513, Test Loss:0.0846, Test Accuracy: 97.54%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0383, Test Loss:0.0803, Test Accuracy: 97.69%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.288, Test Loss:0.179, Test Accuracy: 95.63%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.14, Test Loss:0.142, Test Accuracy: 96.90%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.109, Test Loss:0.131, Test Accuracy: 97.21%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0931, Test Loss:0.129, Test Accuracy: 97.29%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0824, Test Loss:0.123, Test Accuracy: 97.52%
---------------------------------------------------------------------------------------------
We have done: 290 iterations
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.246, Test Loss:0.133, Test Accuracy: 95.93%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.0956, Test Loss:0.0926, Test Accuracy: 97.13%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0625, Test Loss:0.08, Test Accuracy: 97.50%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0444, Test Loss:0.0733, Test Accuracy: 97.67%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0326, Test Loss:0.0681, Test Accuracy: 97.81%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.249, Test Loss:0.135, Test Accuracy: 96.08%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.101, Test Loss:0.0968, Test Accuracy: 97.16%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.068, Test Loss:0.0842, Test Accuracy: 97.53%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0503, Test Loss:0.0777, Test Accuracy: 97.72%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0387, Test Loss:0.0738, Test Accuracy: 97.82%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.286, Test Loss:0.185, Test Accuracy: 95.39%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.141, Test Loss:0.145, Test Accuracy: 96.89%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.11, Test Loss:0.129, Test Accuracy: 97.27%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0929, Test Loss:0.12, Test Accuracy: 97.53%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0823, Test Loss:0.114, Test Accuracy: 97.70%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.249, Test Loss:0.137, Test Accuracy: 95.68%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0974, Test Loss:0.0964, Test Accuracy: 97.13%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0643, Test Loss:0.0818, Test Accuracy: 97.46%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0471, Test Loss:0.0739, Test Accuracy: 97.70%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0367, Test Loss:0.068, Test Accuracy: 97.83%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.249, Test Loss:0.141, Test Accuracy: 95.78%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.0993, Test Loss:0.102, Test Accuracy: 97.00%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.0673, Test Loss:0.0859, Test Accuracy: 97.52%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0508, Test Loss:0.0778, Test Accuracy: 97.74%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.041, Test Loss:0.073, Test Accuracy: 97.91%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 16	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.284, Test Loss:0.179, Test Accuracy: 95.68%
Epoch: 2, Learning Rate: 8.00e-04, Training Loss:0.139, Test Loss:0.141, Test Accuracy: 96.87%
Epoch: 3, Learning Rate: 6.40e-04, Training Loss:0.108, Test Loss:0.125, Test Accuracy: 97.40%
Epoch: 4, Learning Rate: 5.12e-04, Training Loss:0.0927, Test Loss:0.116, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 4.10e-04, Training Loss:0.0832, Test Loss:0.111, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.245, Test Loss:0.139, Test Accuracy: 95.72%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.0933, Test Loss:0.187, Test Accuracy: 94.55%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0606, Test Loss:0.102, Test Accuracy: 97.17%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0443, Test Loss:0.128, Test Accuracy: 96.86%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0368, Test Loss:0.0963, Test Accuracy: 97.68%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.272, Test Loss:0.158, Test Accuracy: 95.44%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.106, Test Loss:0.142, Test Accuracy: 96.40%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.0759, Test Loss:0.116, Test Accuracy: 97.15%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.0587, Test Loss:0.115, Test Accuracy: 97.31%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.0538, Test Loss:0.114, Test Accuracy: 97.52%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.29, Test Loss:0.2, Test Accuracy: 95.81%
Epoch: 2, Learning Rate: 1.00e-02, Training Loss:0.165, Test Loss:0.196, Test Accuracy: 96.13%
Epoch: 3, Learning Rate: 1.00e-02, Training Loss:0.141, Test Loss:0.169, Test Accuracy: 97.11%
Epoch: 4, Learning Rate: 1.00e-02, Training Loss:0.127, Test Loss:0.182, Test Accuracy: 96.82%
Epoch: 5, Learning Rate: 1.00e-02, Training Loss:0.124, Test Loss:0.158, Test Accuracy: 97.42%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.262, Test Loss:0.135, Test Accuracy: 95.97%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0907, Test Loss:0.0957, Test Accuracy: 97.10%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.056, Test Loss:0.0849, Test Accuracy: 97.66%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0367, Test Loss:0.0798, Test Accuracy: 97.72%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0258, Test Loss:0.0851, Test Accuracy: 97.74%
---------------------------------------------------------------------------------------------
We have done: 300 iterations
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.236, Test Loss:0.141, Test Accuracy: 95.92%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.0921, Test Loss:0.106, Test Accuracy: 97.05%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.0607, Test Loss:0.111, Test Accuracy: 97.22%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.0433, Test Loss:0.0893, Test Accuracy: 97.99%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0329, Test Loss:0.0943, Test Accuracy: 97.88%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.298, Test Loss:0.198, Test Accuracy: 95.98%
Epoch: 2, Learning Rate: 9.00e-03, Training Loss:0.157, Test Loss:0.176, Test Accuracy: 96.71%
Epoch: 3, Learning Rate: 8.10e-03, Training Loss:0.127, Test Loss:0.155, Test Accuracy: 97.29%
Epoch: 4, Learning Rate: 7.29e-03, Training Loss:0.108, Test Loss:0.151, Test Accuracy: 97.50%
Epoch: 5, Learning Rate: 6.56e-03, Training Loss:0.0946, Test Loss:0.142, Test Accuracy: 97.55%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.263, Test Loss:0.133, Test Accuracy: 95.73%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0841, Test Loss:0.0945, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0481, Test Loss:0.0771, Test Accuracy: 97.74%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0293, Test Loss:0.0731, Test Accuracy: 97.92%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0184, Test Loss:0.0757, Test Accuracy: 97.99%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.271, Test Loss:0.147, Test Accuracy: 95.91%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.0939, Test Loss:0.109, Test Accuracy: 97.09%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.0612, Test Loss:0.101, Test Accuracy: 97.47%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0413, Test Loss:0.0859, Test Accuracy: 97.85%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0303, Test Loss:0.0842, Test Accuracy: 98.06%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.01	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-02, Training Loss:0.286, Test Loss:0.21, Test Accuracy: 95.46%
Epoch: 2, Learning Rate: 8.00e-03, Training Loss:0.149, Test Loss:0.154, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 6.40e-03, Training Loss:0.114, Test Loss:0.136, Test Accuracy: 97.80%
Epoch: 4, Learning Rate: 5.12e-03, Training Loss:0.0945, Test Loss:0.136, Test Accuracy: 97.70%
Epoch: 5, Learning Rate: 4.10e-03, Training Loss:0.0814, Test Loss:0.126, Test Accuracy: 97.86%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.223, Test Loss:0.118, Test Accuracy: 96.29%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.086, Test Loss:0.0867, Test Accuracy: 97.27%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0543, Test Loss:0.0781, Test Accuracy: 97.61%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0358, Test Loss:0.0791, Test Accuracy: 97.64%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0244, Test Loss:0.13, Test Accuracy: 96.56%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.23, Test Loss:0.127, Test Accuracy: 96.03%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.0916, Test Loss:0.103, Test Accuracy: 97.02%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.0609, Test Loss:0.0863, Test Accuracy: 97.57%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0426, Test Loss:0.0785, Test Accuracy: 97.94%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.0309, Test Loss:0.0818, Test Accuracy: 97.79%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.264, Test Loss:0.164, Test Accuracy: 96.24%
Epoch: 2, Learning Rate: 3.00e-03, Training Loss:0.132, Test Loss:0.139, Test Accuracy: 97.14%
Epoch: 3, Learning Rate: 3.00e-03, Training Loss:0.104, Test Loss:0.127, Test Accuracy: 97.53%
Epoch: 4, Learning Rate: 3.00e-03, Training Loss:0.0886, Test Loss:0.129, Test Accuracy: 97.69%
Epoch: 5, Learning Rate: 3.00e-03, Training Loss:0.078, Test Loss:0.122, Test Accuracy: 97.74%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.229, Test Loss:0.126, Test Accuracy: 96.02%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0854, Test Loss:0.0887, Test Accuracy: 97.28%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0536, Test Loss:0.0735, Test Accuracy: 97.76%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.0354, Test Loss:0.0648, Test Accuracy: 98.03%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.024, Test Loss:0.0629, Test Accuracy: 98.10%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.227, Test Loss:0.122, Test Accuracy: 96.27%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.0875, Test Loss:0.0937, Test Accuracy: 97.29%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.0568, Test Loss:0.0818, Test Accuracy: 97.75%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.04, Test Loss:0.0796, Test Accuracy: 97.79%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0295, Test Loss:0.0751, Test Accuracy: 97.80%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.265, Test Loss:0.166, Test Accuracy: 96.21%
Epoch: 2, Learning Rate: 2.70e-03, Training Loss:0.131, Test Loss:0.133, Test Accuracy: 97.20%
Epoch: 3, Learning Rate: 2.43e-03, Training Loss:0.102, Test Loss:0.122, Test Accuracy: 97.58%
Epoch: 4, Learning Rate: 2.19e-03, Training Loss:0.086, Test Loss:0.116, Test Accuracy: 97.77%
Epoch: 5, Learning Rate: 1.97e-03, Training Loss:0.0762, Test Loss:0.113, Test Accuracy: 97.85%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.225, Test Loss:0.122, Test Accuracy: 96.23%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0815, Test Loss:0.0856, Test Accuracy: 97.56%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0502, Test Loss:0.0742, Test Accuracy: 97.73%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0338, Test Loss:0.0674, Test Accuracy: 97.87%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0243, Test Loss:0.064, Test Accuracy: 98.01%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.228, Test Loss:0.126, Test Accuracy: 96.20%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.0868, Test Loss:0.0919, Test Accuracy: 97.22%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0561, Test Loss:0.0826, Test Accuracy: 97.56%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0401, Test Loss:0.0771, Test Accuracy: 97.88%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0307, Test Loss:0.0719, Test Accuracy: 97.97%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.003	learning_rate_decay: 0.8	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 3.00e-03, Training Loss:0.264, Test Loss:0.172, Test Accuracy: 95.83%
Epoch: 2, Learning Rate: 2.40e-03, Training Loss:0.126, Test Loss:0.13, Test Accuracy: 97.33%
Epoch: 3, Learning Rate: 1.92e-03, Training Loss:0.0984, Test Loss:0.12, Test Accuracy: 97.65%
Epoch: 4, Learning Rate: 1.54e-03, Training Loss:0.0839, Test Loss:0.115, Test Accuracy: 97.89%
Epoch: 5, Learning Rate: 1.23e-03, Training Loss:0.0752, Test Loss:0.111, Test Accuracy: 97.94%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.318, Test Loss:0.176, Test Accuracy: 94.76%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.137, Test Loss:0.118, Test Accuracy: 96.40%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0943, Test Loss:0.0964, Test Accuracy: 97.19%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0706, Test Loss:0.0849, Test Accuracy: 97.41%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0551, Test Loss:0.0781, Test Accuracy: 97.57%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.001	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.325, Test Loss:0.186, Test Accuracy: 94.48%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.143, Test Loss:0.124, Test Accuracy: 96.25%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.0994, Test Loss:0.0991, Test Accuracy: 96.99%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.0757, Test Loss:0.0865, Test Accuracy: 97.34%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.0602, Test Loss:0.0803, Test Accuracy: 97.58%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0.01	learning_rate: 0.001	learning_rate_decay: 1	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.36, Test Loss:0.221, Test Accuracy: 94.43%
Epoch: 2, Learning Rate: 1.00e-03, Training Loss:0.181, Test Loss:0.163, Test Accuracy: 96.16%
Epoch: 3, Learning Rate: 1.00e-03, Training Loss:0.139, Test Loss:0.14, Test Accuracy: 97.00%
Epoch: 4, Learning Rate: 1.00e-03, Training Loss:0.117, Test Loss:0.128, Test Accuracy: 97.32%
Epoch: 5, Learning Rate: 1.00e-03, Training Loss:0.102, Test Loss:0.122, Test Accuracy: 97.53%
---------------------------------------------------------------------------------------------
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0}) 

Hyperparameters:	hidden_dims: (500, 400)	mini_batch_size: 32	regularization: 0	learning_rate: 0.001	learning_rate_decay: 0.9	Initialization method: glorot_initializer	Activation: RELU
number of trainable parameters:596910
Epoch: 1, Learning Rate: 1.00e-03, Training Loss:0.322, Test Loss:0.182, Test Accuracy: 94.59%
Epoch: 2, Learning Rate: 9.00e-04, Training Loss:0.141, Test Loss:0.121, Test Accuracy: 96.22%
Epoch: 3, Learning Rate: 8.10e-04, Training Loss:0.0984, Test Loss:0.0992, Test Accuracy: 96.93%
Epoch: 4, Learning Rate: 7.29e-04, Training Loss:0.0756, Test Loss:0.0877, Test Accuracy: 97.31%
Epoch: 5, Learning Rate: 6.56e-04, Training Loss:0.0609, Test Loss:0.0803, Test Accuracy: 97.49%
---------------------------------------------------------------------------------------------
We have done: 310 iterations
(285, 98.26, {'h1': 500, 'h2': 400, 'mini_batch_size': 16, 'learning_rate': 0.003, 'learning_rate_decay': 0.8, 'regularization': 0})