-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slight result difference #6
Comments
Thanks Wead! Meng |
Thanks a lot. I am conducting some experiments based on this repo. If I get some positive results, I'd like to share my idea and look forward to your suggestion! |
It seems due to the default hyperparameters after inspecting the log. The predictor_num_epoch is so small that the learning is not adequate. |
This is indeed a problem. The codes were refactorized before release, especially for the predictor, so the optimal hyperparameters might change. We are also trying to tune the hyperparameters to match the results of the previous codes. |
Hi Wead, |
Thanks for your help! When changing the max rule length, should I also change the length_time parameter in the rule_sample function to {1: 1, 2: 10, 3: 10, 4: 10, 5: 10}? For FB15k-237, I actually pre-trained a model using the KnowledgeGraphEmbedding repo with the best configuration and move it to the data directory. Is that OK? I have modified the code and am running the experiments following your suggestions. By the way, I noticed that the relation_embedding is reset after training for validation (line 719-721 in model_rnnlogic.py), does it matter for model performance? Sincerely. |
Hi, I have tried to run the experiments with your suggestion, however, the results remain similar, i.e., mr~7500+. For wn18rr, I have tried to (1) increase the max_rule_length to 5 (2) increase the predictor_num_epoch to 8000 and (3) change to length_time in the rule_sample function to {1:1, 2:10, 3:10, 4:10, 5:10}. There are many details in the code, I do not know how to tune these parameters. If you have any other suggestions, please tell me. |
Hi Weidi, Sorry for the inconvenience. I have found a major problem in our code: the hyperparameters of code I have updated Thank you again for pointing out the problems. Sincerely, |
Thanks for your response. I didn't realize that the pgnd is disabled by previous configurations. I have updated the code and rerun the experiments. |
Hi, meng. I was recently rerunning the code to reproduce the results.
Appreciate your wonderful code. I successfully reproduce�d the results in the kinship and umls datasets. However, the results on the wn18rr seem inconsistent (rnnlogic w emb: 7261, 0.45, 0.42, 0.53). I wonder if i made something wrong.
I did the following to run the experiment:
And I also found the same problem on the FB15k-237 dataset. Could you please give some help?
The text was updated successfully, but these errors were encountered: