onert-micro / onert checkpoint format

[chunseoklee]

parent : #12997

Let's design checkpoint format for training.

Checkpoint should contains current trainable weight(tensor), optimizer variable(tensor), and training parameter state(kv pair).

[BalyshevArtem]

What do you mean by optimizer variable(tensor), and training parameter state(kv pair).?

[chunseoklee]

• Optimizer variable is e.g. ADAM optimizer's state weight(IDK exact term)
• training parameter state is like current step number, epoch number, which is any values to resume the training.

[chunseoklee]

In checkpoint, we should save three tensors(please let me know if wrong) : weight tensor, ADAM's 2 state tensors(m, v). And those three tensors have the same shape. Thus, save the ADAM's two state tensors in the same way of weight tensor as follows.

0	2		4	8	12	16	…	16 + N * 4					m offset				v offset			parameter offset
MAGIC NUMBER	SCHEMA VERSION	RESERVED	m offset	v offset	other parameter offset	n_buffers (=N)	Offset 1	…	Offset N	Buffer 1 Data	Buffer 2 Data	…	# of m	offset for m[0]	offset for m[1]	...	# of v	offset for v[1]	...	# of parameter

Q. there is any parameter not a number(e.g. string format) ?

[BalyshevArtem]

Q. there is any parameter not a number(e.g. string format) ? - I think no, due to we can all such parameter to encode as enum: for example for loss type (if we will save loss information), then we can decide, for example that MSE is is 1, MAE is 2 and so on. So, I think we need to save the following information: Optimizer (ADAM, SGD), Optimizer specific params (lambda, beta (for Adam), epsilon (for Adam)), epoch number, step number. Smth like:

0	2		4	8	12	16	…	16 + N * 4					m offset				v offset			start of the parameters offset	start of the parameters offset + 4	start of the parameters offset + 8	start of the parameters offset + 12	start of the parameters offset + 16	start of the parameters offset + 20	start of the parameters offset + 24	start of the parameters offset + 28
MAGIC NUMBER	SCHEMA VERSION	RESERVED	m offset	v offset	start of the parameters offset	n_buffers (=N)	Offset 1	…	Offset N	Buffer 1 Data	Buffer 2 Data	…	# of m	offset for m[0]	offset for m[1]	...	# of v	offset for v[1]	...	encoded optimizer type	current epoch	current step	encoded loss type	batch size	lambda value	beta	beta_squares

[chunseoklee]

lambda, beta and epsilon will be saved on circle's meta information(e.g. generated by https://github.com/Samsung/ONE/tree/master/tools/circle_plus_gen). I'd like to keep information which can be updated during training in checkpoint : weight, adam state, step number, so on.

[BalyshevArtem]

lambda, beta and epsilon will be saved on circle's meta information(e.g. generated by https://github.com/Samsung/ONE/tree/master/tools/circle_plus_gen). I'd like to keep information which can be updated during training in checkpoint : weight, adam state, step number, so on.

Ah, I see. Then I think, that - weight, adam state, current step number and current epoch num all information that can change.

[BalyshevArtem]

In #13107 I used the following format:

0	2	3	4	8	12	16	20	...	20 + N * 4 - 4	20 + N * 4	...	Buffers_size + 20 + N * 4	m offset	...	m offset + N * 4 - 4	m offset + N * 4	...	m offset + Buffer_size + N * 4	v offset	...	v offset + N * 4 - 4	v offset + N * 4	...	v offset + Buffer_size + N * 4	other param offset	other param offset + 4
MAGIC NUMBER	SCHEMA VERSION	RESERVED	m offset	v offset	other parameters offset	n_buffers (=N)	Offset 1	…	Offset N	Buffer 1 Data	…	Buffer N Data	Offset 1 for m	...	Offset N for m	Buffer 1 for m	...	Buffer N for m	Offset 1 for v	...	Offset N for v	Buffer 1 for v	...	Buffer N for v	cur step value	cur epoch value

[jyoungyun]

I'm introducing the checkpoint API to onert. During this task, I faced some issues with the checkopoint file format. I left several issues to discuss in the future.

• Redesign optimizer variable format to be flexible
  • SGD does not need m offset and v offset. If the model uses SGD optimizer, these are redundant.
• Reconsideration of epoch usability
  • When is the epoch used when resuming the model training?
• Schema version
  • [onert] Introduce checkpoint.h file #13671 (comment)