This repository has been archived by the owner on Dec 11, 2022. It is now read-only.

Tf2 migration#430

Open

dandanelbaz wants to merge 153 commits intomasterfrom tf2_migration

+4,912-1,820

Commits on Jul 7, 2019

Added WIP warning in the README.md file header
Dan Elbaz
committed

Commits on Jul 14, 2019

Added tf2 components folder. The folder contains the same files as tensorflow components after applying tf migration script and the manual adjustments required for starting the migration twords tf2
Dan Elbaz
committed

Commits on Jul 16, 2019

Commits on Jul 24, 2019

Started input embeder migration to Keras
Dan Elbaz
committed

Commits on Jul 28, 2019

Input embedder migrated to tf2
Dan Elbaz
committed

Commits on Jul 29, 2019

MiddleWare migrated to tf2. Added DnnModel class, this class instantiate the network components
Dan Elbaz
committed

Commits on Jul 30, 2019

initial version of general_network migrated to tf2. The network is implemented in dnn_model. The loss was seperated from the head and will be moved to a different class
Dan Elbaz
committed

Commits on Aug 1, 2019

Forward pass is running, not tested for correctness. Training is not functioning yet
Dan Elbaz
committed

Commits on Aug 5, 2019

Separated the loss function from the DNN head. loss is now in a separate class
Dan Elbaz
committed

Commits on Aug 6, 2019

Commits on Aug 7, 2019

Commits on Aug 8, 2019

Added batchnorm, dropout and activation functions to middleware
Dan Elbaz
committed
Loss was removed from DNN head
Dan Elbaz
committed
get_input embedder is called with explicit vector embedder parameters and without dynamic import
Dan Elbaz
committed

Commits on Aug 11, 2019

get_middleware is called with explicit vector embedder parameters and without dynamic import
Dan Elbaz
committed
Cleaning network head constructor
Dan Elbaz
committed

Commits on Aug 12, 2019

get_output_head is called with explicit vector embedder parameters and without dynamic import
Dan Elbaz
committed

Commits on Aug 13, 2019

Checkpoint before mxnet enablement
Dan Elbaz
committed
Rolled back API changes to re enable mxnet
Dan Elbaz
committed
Single worker model is instantiated in a seperate class. Need to check with model build if model inputs are one for each embedder
Dan Elbaz
committed

Commits on Aug 14, 2019

general_network migrated to tf2 (mostly)
Dan Elbaz
committed
Deleting redundant files
Dan Elbaz
committed

Commits on Aug 15, 2019

GeneralTensorFlowNetwork extends TensorFlowArchitecture. We should consider changing this in the future
Dan Elbaz
committed
Fixed devision bug and embbeder name
Dan Elbaz
committed
Full batch forward through the DNN model
Dan Elbaz
committed

Commits on Aug 19, 2019

changed model inputs to one input for each embedder. parallel predict and predict were adjusted accordingly
Dan Elbaz
committed

Commits on Aug 20, 2019

Loss is calculated on input batch
Dan Elbaz
committed
Gradient of the loss is calculated w.r.t model parameters
Dan Elbaz
committed

Commits on Aug 21, 2019

Gradients accumulation converted to tf2
Dan Elbaz
committed
Model weights are updated with optimization step
Dan Elbaz
committed
DQN is runing. Not tested.
Dan Elbaz
committed
Shallow testing on DQN shows convergence
Dan Elbaz
committed
Cosmetic changes
Dan Elbaz
committed
Removed generalLoss class. Loss class is defined in the same file as the Head class
Dan Elbaz
committed

Commits on Aug 22, 2019

Added temporary preparation for gradient rescaling, needs testing that indeed does as expected
Dan Elbaz
committed
Removed gradient rescaling wrapper class
Dan Elbaz
committed
get_input_embedder, get_middleware and get_output head are member functions inside DnnModel
Dan Elbaz
committed
DNN Model moved to a separate module
Dan Elbaz
committed
Added losses directory with loss base class
Dan Elbaz
committed

Commits on Aug 27, 2019

Starting loss separation from head.
Dan Elbaz
committed
DNN Head is separeted from loss function
Dan Elbaz
committed
Modules renaming
Dan Elbaz
committed
Removed loss from head module
Dan Elbaz
committed

Commits on Aug 28, 2019

Image embedder converted to tf1 compatible version
Dan Elbaz
committed

Commits on Aug 29, 2019

Added small code snippet that enables running the script from command line without rl-coach package installation
Dan Elbaz
committed
GalN code review fixes
Dan Elbaz
committed

Commits on Sep 1, 2019

Adding Gal Leibovitch code review comments
Dan Elbaz
committed
Adding Gal Leibovitch code review comments
Dan Elbaz
committed
Added gradient clipping
Dan Elbaz
committed

Commits on Sep 2, 2019

Commits on Sep 4, 2019

Saver save and restore partly migrated, only saving the weights, not the architecture
Dan Elbaz
committed

Commits on Sep 8, 2019

Removed concurrency support in order to avoid broken pipe error
Dan Elbaz
committed

Commits on Sep 9, 2019

Fixed memory leak bug
Dan Elbaz
committed

Commits on Sep 10, 2019

Commits on Sep 11, 2019

Starting clipped PPO infrastructure
Dan Elbaz
committed
Added value loss
Dan Elbaz
committed

Commits on Sep 12, 2019

Adding a RL loss base class ontop of keras loss class
Dan Elbaz
committed
Renamed output_heads to heads
Dan Elbaz
committed
Adding a RL loss base class ontop of keras loss class
Dan Elbaz
committed

Commits on Sep 15, 2019

Commits on Sep 16, 2019

dummy_model_inputs moved to dnn_model
Dan Elbaz
committed

Commits on Sep 18, 2019

Spliting the targets into targets per loss
Dan Elbaz
committed
Head Loss expands keras layer and not keras loss. This is due to call function signature constraint
Dan Elbaz
committed
Added output schema to the head loss output
Dan Elbaz
committed

Commits on Sep 19, 2019

Added output schema to the head loss output
Dan Elbaz
committed
Added output schema to the head loss output
Dan Elbaz
committed
Merge branch 'tf2_migration' of https://github.com/NervanaSystems/coach into tf2_migration
Dan Elbaz
committed
Bug fix in tensorflow architecture. Dimensionality check added
Dan Elbaz
committed

Commits on Sep 22, 2019

Added _num_outputs property to output head to be compared against numer of inputs to the loss class
Dan Elbaz
committed

Commits on Sep 23, 2019

Changed loss output to dictionary, added a check on the loss outputs
Dan Elbaz
committed

Commits on Sep 24, 2019

Clipped PPO is running. Gradient tape is not working as expected with tf.probability
Dan Elbaz
committed
Checkpoint
Dan Elbaz
committed

Commits on Sep 25, 2019

Commits on Sep 26, 2019

Changed PPO agent to output probability distribution
Dan Elbaz
committed
Dimensions Bug fix in PPO loss
Dan Elbaz
committed
PPO learning is extreamly slow
Dan Elbaz
committed
DQN is learning OK. Memory usage is very high
Dan Elbaz
committed

Commits on Oct 2, 2019

Commits on Oct 3, 2019

Removed loss from model.compaile
Dan Elbaz
committed
Actor is outputing log std instead of std in order to to constrain positive stds
Dan Elbaz
committed

Commits on Oct 7, 2019

Detaching The value of the std from the network output. Reaching reward of 1000 on inverted pendulum.
Dan Elbaz
committed

Commits on Oct 10, 2019

Input shape is derived from emmbeder parameters. Instead of from instantiated network
Dan Elbaz
committed

Commits on Oct 13, 2019

Commits on Oct 15, 2019

Commits on Oct 17, 2019

Changed PPO Head to functional form. Reaching reward 0f 1000
Dan Elbaz
committed
Changed model_wrapper class to create_model function
Dan Elbaz
committed
Changed actor critic to functional form. DQN gives errors
Dan Elbaz
committed

Commits on Oct 18, 2019

DQN Bug fixed PPO is Buggy
Dan Elbaz
committed
Sanity PPO reaching 1000
Dan Elbaz
committed
Functional top model DQN and PPO running
Dan Elbaz
committed

Commits on Oct 19, 2019

Changed head value output shape and value loss accordingly
Dan Elbaz
committed

Commits on Oct 20, 2019

Reverted unnecessary changes from initial commit. PPO reaches 1000
Dan Elbaz
committed
Changed back ppo agent to the original fetches form
Dan Elbaz
committed

Commits on Oct 21, 2019

Starting single network migration to functional form
Dan Elbaz
committed

Commits on Oct 22, 2019

Both DQN and PPO are running

Dan Elbaz
authored and
Dan Elbaz
committed
Removed SingleDNN PPO Head models subclass impementation. PPO Reward 1000
Dan Elbaz
committed
Changed heads to functional form. PPO 1000
Dan Elbaz
committed

Commits on Oct 23, 2019

Added functional losses
Dan Elbaz
committed

Commits on Nov 20, 2019

PPO is working, everything else is broken. Unintentionally shared parameters between middlewares in actor and critic
Dan Elbaz
committed

Commits on Nov 21, 2019

Rolled back unintentional change to mxnet
Dan Elbaz
committed
Back to old accumulate_gradients, not verified
Dan Elbaz
committed

Commits on Nov 26, 2019

Commits on Nov 27, 2019

humanoid is runing. inverted pendulum is checked with clipped PPO
Dan Elbaz
committed

Commits on Dec 1, 2019

Humanoid reaches benchmark. Number of timesteps is OK. Wall clock time is slow
Dan Elbaz
committed

Commits on Dec 2, 2019

Commits on Dec 4, 2019

Inverted Pendulum verifiedd
Dan Elbaz
committed
Reverted coach main to original version. Renaming and coments
Dan Elbaz
committed
Checkpoint, before fixing additional input
Dan Elbaz
committed
Report generation
Dan Elbaz
committed

Commits on Dec 5, 2019

Presentation
Dan Elbaz
committed

Commits on Dec 15, 2019

Humanoid multiple seeds experiment done on this checkpoint
Dan Elbaz
committed
Removed output schema
Dan Elbaz
committed
Changed loss input schema to split inputs based on trainable and non trainable arguments
Dan Elbaz
committed
Updated copyriCopyright header in TF2 files
Dan Elbaz
committed

Commits on Dec 16, 2019

Each head_loss is responsible for extracting its own args. Inverted pendulum verified
Dan Elbaz
committed
Updated input schema and input checking
Dan Elbaz
committed
Removed PolicyHead and PPO VHead
Dan Elbaz
committed
Added comments and typings
Dan Elbaz
committed
Removed LSTM middleware, not supported in TF2
Dan Elbaz
committed
Added documentation
Dan Elbaz
committed
Added documentation
Dan Elbaz
committed

Commits on Dec 17, 2019

Added wrappers on dense layers and changed casting, Should test for correctnes here
Dan Elbaz
committed
loss type is not hard coded, can be configured from agent params
Dan Elbaz
committed
QHead and VHead are generated via function call, not class constructor
Dan Elbaz
committed
Added GPU support
Dan Elbaz
committed
Checkpoint before runing experiments
Dan Elbaz
committed

Commits on Dec 18, 2019

Comments and typing hints were inserted
Dan Elbaz
committed

Commits on Dec 22, 2019

Restrict TensorFlow to only allocate 2GB of memory on the first GPU, for multi seeds run
Dan Elbaz
committed

Commits on Dec 25, 2019

Added benchmark for humanoid_clipped_pp
Dan Elbaz
committed

Commits on Dec 29, 2019

Added benchmark for DQN pong
Dan Elbaz
committed
Added benchmark for DQN breakout
Dan Elbaz
committed
Added benchmark for inverted pendulum, only have 2.5 M timesteps
Dan Elbaz
committed

Commits on Dec 30, 2019

Added benchmark for space invaders DQN
Dan Elbaz
committed

Commits on Jan 8, 2020

Added benchmarks for clipped PPO double pendulum and half cheetah
Dan Elbaz
committed

Commits on Jan 12, 2020

Added ant clipped ppo benchmark
Dan Elbaz
committed