Strange behavior in cartpole.py

[tcfuji]

Hi,

When running cartpole.py, I get the following (go to step 119):

% python cartpole.py
INFO:evograd.noise:Sampling 10000000 random numbers with seed 42
INFO:evograd.noise:Sampled 40000000 bytes
/Users/tcf/anaconda3/envs/evo/lib/python3.7/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
step: 0, mean fitness: 120.77
step: 1, mean fitness: 124.54
step: 2, mean fitness: 132.03
step: 3, mean fitness: 136.29
step: 4, mean fitness: 137.41
step: 5, mean fitness: 126.86
step: 6, mean fitness: 146.96
step: 7, mean fitness: 150.17
step: 8, mean fitness: 154.71
step: 9, mean fitness: 152.27
step: 10, mean fitness: 159.49
step: 11, mean fitness: 163.4
step: 12, mean fitness: 162.69
step: 13, mean fitness: 159.5
step: 14, mean fitness: 150.78
step: 15, mean fitness: 165.73
step: 16, mean fitness: 169.43
step: 17, mean fitness: 176.08
step: 18, mean fitness: 166.3
step: 19, mean fitness: 178.74
step: 20, mean fitness: 181.06
step: 21, mean fitness: 181.42
step: 22, mean fitness: 182.07
step: 23, mean fitness: 175.19
step: 24, mean fitness: 181.09
step: 25, mean fitness: 178.61
step: 26, mean fitness: 184.12
step: 27, mean fitness: 185.27
step: 28, mean fitness: 184.36
step: 29, mean fitness: 181.19
step: 30, mean fitness: 184.31
step: 31, mean fitness: 182.96
step: 32, mean fitness: 188.73
step: 33, mean fitness: 188.25
step: 34, mean fitness: 189.97
step: 35, mean fitness: 193.47
step: 36, mean fitness: 191.24
step: 37, mean fitness: 192.58
step: 38, mean fitness: 187.92
step: 39, mean fitness: 188.66
step: 40, mean fitness: 189.48
step: 41, mean fitness: 191.93
step: 42, mean fitness: 191.03
step: 43, mean fitness: 191.72
step: 44, mean fitness: 191.17
step: 45, mean fitness: 192.76
step: 46, mean fitness: 192.89
step: 47, mean fitness: 191.14
step: 48, mean fitness: 196.5
step: 49, mean fitness: 194.53
step: 50, mean fitness: 191.08
step: 51, mean fitness: 193.84
step: 52, mean fitness: 195.39
step: 53, mean fitness: 194.18
step: 54, mean fitness: 191.56
step: 55, mean fitness: 193.55
step: 56, mean fitness: 195.62
step: 57, mean fitness: 190.07
step: 58, mean fitness: 194.5
step: 59, mean fitness: 199.6
step: 60, mean fitness: 195.21
step: 61, mean fitness: 193.79
step: 62, mean fitness: 194.38
step: 63, mean fitness: 198.5
step: 64, mean fitness: 198.08
step: 65, mean fitness: 197.03
step: 66, mean fitness: 193.46
step: 67, mean fitness: 195.71
step: 68, mean fitness: 197.98
step: 69, mean fitness: 192.48
step: 70, mean fitness: 198.83
step: 71, mean fitness: 199.3
step: 72, mean fitness: 197.35
step: 73, mean fitness: 199.26
step: 74, mean fitness: 198.08
step: 75, mean fitness: 195.37
step: 76, mean fitness: 199.07
step: 77, mean fitness: 198.37
step: 78, mean fitness: 199.12
step: 79, mean fitness: 194.08
step: 80, mean fitness: 198.94
step: 81, mean fitness: 197.03
step: 82, mean fitness: 199.53
step: 83, mean fitness: 198.38
step: 84, mean fitness: 195.94
step: 85, mean fitness: 198.77
step: 86, mean fitness: 199.41
step: 87, mean fitness: 195.99
step: 88, mean fitness: 197.38
step: 89, mean fitness: 195.78
step: 90, mean fitness: 197.69
step: 91, mean fitness: 199.55
step: 92, mean fitness: 199.54
step: 93, mean fitness: 197.72
step: 94, mean fitness: 197.13
step: 95, mean fitness: 192.07
step: 96, mean fitness: 199.21
step: 97, mean fitness: 197.8
step: 98, mean fitness: 199.78
step: 99, mean fitness: 198.93
step: 100, mean fitness: 199.01
step: 101, mean fitness: 199.42
step: 102, mean fitness: 199.06
step: 103, mean fitness: 199.55
step: 104, mean fitness: 193.92
step: 105, mean fitness: 198.15
step: 106, mean fitness: 199.96
step: 107, mean fitness: 199.94
step: 108, mean fitness: 198.41
step: 109, mean fitness: 196.36
step: 110, mean fitness: 199.65
step: 111, mean fitness: 199.8
step: 112, mean fitness: 199.69
step: 113, mean fitness: 199.12
step: 114, mean fitness: 196.0
step: 115, mean fitness: 199.94
step: 116, mean fitness: 199.6
step: 117, mean fitness: 199.88
step: 118, mean fitness: 200.0
step: 119, mean fitness: 9.352
step: 120, mean fitness: 9.36
step: 121, mean fitness: 9.332
step: 122, mean fitness: 9.332
step: 123, mean fitness: 9.302
step: 124, mean fitness: 9.346
step: 125, mean fitness: 9.346
step: 126, mean fitness: 9.372
step: 127, mean fitness: 9.326
step: 128, mean fitness: 9.302
step: 129, mean fitness: 9.378
step: 130, mean fitness: 9.376
step: 131, mean fitness: 9.338
step: 132, mean fitness: 9.396
step: 133, mean fitness: 9.336
step: 134, mean fitness: 9.388
step: 135, mean fitness: 9.344
step: 136, mean fitness: 9.356
step: 137, mean fitness: 9.326
step: 138, mean fitness: 9.338
step: 139, mean fitness: 9.376
step: 140, mean fitness: 9.354
step: 141, mean fitness: 9.314
step: 142, mean fitness: 9.35
step: 143, mean fitness: 9.4
step: 144, mean fitness: 9.386
step: 145, mean fitness: 9.38
step: 146, mean fitness: 9.312
step: 147, mean fitness: 9.334
step: 148, mean fitness: 9.374
step: 149, mean fitness: 9.302
step: 150, mean fitness: 9.324
step: 151, mean fitness: 9.39
step: 152, mean fitness: 9.342
step: 153, mean fitness: 9.372
step: 154, mean fitness: 9.38
step: 155, mean fitness: 9.428
step: 156, mean fitness: 9.41
step: 157, mean fitness: 9.346
step: 158, mean fitness: 9.366
step: 159, mean fitness: 9.384
step: 160, mean fitness: 9.348
step: 161, mean fitness: 9.282
step: 162, mean fitness: 9.312
step: 163, mean fitness: 9.338
step: 164, mean fitness: 9.32
step: 165, mean fitness: 9.39
step: 166, mean fitness: 9.302
step: 167, mean fitness: 9.374
step: 168, mean fitness: 9.356
step: 169, mean fitness: 9.33
step: 170, mean fitness: 9.356
step: 171, mean fitness: 9.366
step: 172, mean fitness: 9.41
step: 173, mean fitness: 9.366
step: 174, mean fitness: 9.364
step: 175, mean fitness: 9.378
step: 176, mean fitness: 9.4
step: 177, mean fitness: 9.386
step: 178, mean fitness: 9.43
step: 179, mean fitness: 9.374
step: 180, mean fitness: 9.42
step: 181, mean fitness: 9.316
step: 182, mean fitness: 9.31
step: 183, mean fitness: 9.348
step: 184, mean fitness: 9.384
step: 185, mean fitness: 9.396
step: 186, mean fitness: 9.436
step: 187, mean fitness: 9.36
step: 188, mean fitness: 9.32
step: 189, mean fitness: 9.396
step: 190, mean fitness: 9.34
step: 191, mean fitness: 9.328
step: 192, mean fitness: 9.354
step: 193, mean fitness: 9.408
step: 194, mean fitness: 9.334
step: 195, mean fitness: 9.346
step: 196, mean fitness: 9.388
step: 197, mean fitness: 9.366
step: 198, mean fitness: 9.318
step: 199, mean fitness: 9.358
step: 200, mean fitness: 9.412
step: 201, mean fitness: 9.308
step: 202, mean fitness: 9.338
step: 203, mean fitness: 9.406
step: 204, mean fitness: 9.392
step: 205, mean fitness: 9.388
step: 206, mean fitness: 9.44
step: 207, mean fitness: 9.344
step: 208, mean fitness: 9.384
step: 209, mean fitness: 9.338
step: 210, mean fitness: 9.272
step: 211, mean fitness: 9.36
step: 212, mean fitness: 9.326
step: 213, mean fitness: 9.404
step: 214, mean fitness: 9.416
step: 215, mean fitness: 9.31
step: 216, mean fitness: 9.364
step: 217, mean fitness: 9.334
step: 218, mean fitness: 9.376
step: 219, mean fitness: 9.364
step: 220, mean fitness: 9.296
step: 221, mean fitness: 9.344
step: 222, mean fitness: 9.354
step: 223, mean fitness: 9.384
step: 224, mean fitness: 9.284
step: 225, mean fitness: 9.382
step: 226, mean fitness: 9.344
step: 227, mean fitness: 9.354
step: 228, mean fitness: 9.364
step: 229, mean fitness: 9.346
step: 230, mean fitness: 9.308
step: 231, mean fitness: 9.332
step: 232, mean fitness: 9.326
step: 233, mean fitness: 9.344
step: 234, mean fitness: 9.332
step: 235, mean fitness: 9.334
step: 236, mean fitness: 9.356
step: 237, mean fitness: 9.328
step: 238, mean fitness: 9.346
step: 239, mean fitness: 9.286
step: 240, mean fitness: 9.354
step: 241, mean fitness: 9.374
step: 242, mean fitness: 9.372
step: 243, mean fitness: 9.376
step: 244, mean fitness: 9.328
step: 245, mean fitness: 9.354
step: 246, mean fitness: 9.382
step: 247, mean fitness: 9.328
step: 248, mean fitness: 9.346
step: 249, mean fitness: 9.362
step: 250, mean fitness: 9.362
step: 251, mean fitness: 9.324
step: 252, mean fitness: 9.354
step: 253, mean fitness: 9.36
step: 254, mean fitness: 9.366
step: 255, mean fitness: 9.4
step: 256, mean fitness: 9.392
step: 257, mean fitness: 9.352
step: 258, mean fitness: 9.348
step: 259, mean fitness: 9.372
step: 260, mean fitness: 9.312
step: 261, mean fitness: 9.33
step: 262, mean fitness: 9.396
step: 263, mean fitness: 9.36
step: 264, mean fitness: 9.34
step: 265, mean fitness: 9.338
step: 266, mean fitness: 9.354
step: 267, mean fitness: 9.246
step: 268, mean fitness: 9.404
step: 269, mean fitness: 9.342
step: 270, mean fitness: 9.302
step: 271, mean fitness: 9.302
step: 272, mean fitness: 9.354
step: 273, mean fitness: 9.37
step: 274, mean fitness: 9.316
step: 275, mean fitness: 9.318
step: 276, mean fitness: 9.34
step: 277, mean fitness: 9.344
step: 278, mean fitness: 9.348
step: 279, mean fitness: 9.338
step: 280, mean fitness: 9.382
step: 281, mean fitness: 9.322
step: 282, mean fitness: 9.324
step: 283, mean fitness: 9.396
step: 284, mean fitness: 9.394
step: 285, mean fitness: 9.384
step: 286, mean fitness: 9.428
step: 287, mean fitness: 9.412
step: 288, mean fitness: 9.398
step: 289, mean fitness: 9.332
step: 290, mean fitness: 9.326
step: 291, mean fitness: 9.354
step: 292, mean fitness: 9.374
step: 293, mean fitness: 9.368
step: 294, mean fitness: 9.312
step: 295, mean fitness: 9.39
step: 296, mean fitness: 9.41
step: 297, mean fitness: 9.372
step: 298, mean fitness: 9.342
step: 299, mean fitness: 9.35
step: 300, mean fitness: 9.386
step: 301, mean fitness: 9.338

Is this supposed to happen while maximizing mean fitness? If so, why?

[jf---]

same here - looks like it might have something todo with the fitness hitting 200? That there's no more variation in the population? Huh, let me found out:

step: 121, mean fitness: 195.59, std: 13.277
step: 122, mean fitness: 198.24, std: 8.447
step: 123, mean fitness: 193.96, std: 28.307
>>> step: 124, mean fitness: 200.0, std: 0.0
step: 125, mean fitness: 9.322, std: 0.26053
step: 126, mean fitness: 9.324, std: 0.25679
step: 127, mean fitness: 9.342, std: 0.23482
step: 128, mean fitness: 9.3, std: 0.22131

Yep, that looks suspicious...

[ekorudiawan]

Got same issue here, I try to change the policy with neural network, also got same result, after reach fitness 200, then drop to 9.

Then how to take weights after training? the example only shows how to train, but not explain how to inference with the weights

[hoang-tn-nguyen]

you need to avoid division by 0, set std = (fitnesses.std() + eps). This will fix the problem.