RMD only reproductibility

[aurelien-renault]

Hello,

I'm very interested in your model and I was trying to reproduce the results displayed in the Ablation studies section of the paper. To replicate RMD only results I followed the provided procedure and ran the following : python train_ag.py --rtd_loss_weight 0 and got those results for the first split (business) :

as well as for the second split (sci), python train_ag.py --subset 2 --rtd_loss_weight 0 :

Those don't seem to match with the mean value displayed in the paper (85,9), are there anything I'm doing wrong ?
Thank you in advance for your help.

[andreimano]

Hi Aurelien!

Could you give us more details about the hyperparameters? For how many epochs did you run the experiment?

I will try to reproduce the experiments ASAP, but maybe we can figure the cause this way in the meantime :).

Thanks,
Andrei.

[aurelien-renault]

Hi Andrei !

Thanks for your reply :)

I've basically used the same hyperparameters as setted up by default in the train_ag.py file of the repo, running the experiments for num_train_epochs=20 with train_batch_size=16.

I have also dumped train_args.csv if that can help you in any way.

Thank you for your help !

[andreimano]

Thanks! I just got access to my old machines, I'll check it up next week.