Looking for a reference for action masking in DQN

[BFAnas]

Hi, do you know any publication reference for the implementation of action masking in DQN?

[Trinkle23897]

Search for “legal action” in https://arxiv.org/abs/1908.09453

[BFAnas]

Hi,
Thanks for the answer, unfortunately, it talks about setting the logits of policy-gradient algorithms to −∞. This is the quote that I found:

For policy gradient methods, a masked softmax is used to set the logits of the illegal actions to −∞ to force the policy to sets probability zero to illegal actions.

I'm looking for a reference that talks about setting Q-values of illegal actions to be less than the min(Q-values of legal actions). This part of the code from dqn.py: https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/modelfree/dqn.py?plain=1#L116-120

[Trinkle23897]

hmm I guess you are struggling to answer the reviewer's question. Unfortunately, IMHO there are no papers dedicated to this issue. You can easily transfer the idea from policy-gradient to Q-values. Setting illegal action's Q value to −∞ doesn't affect the gradient update from value iteration and can achieve the feature of masked action. That's all I can tell.