FTS3 job status updating failed when using multi-hop transfer

[zhangxt-ihep]

Hello,
We are using multi-hop for IHEP -> JINR transfer for JUNO experiment in IHEP's DIRAC instance. The transfer will go through CNAF storage, like: IHEP->CNAF->JINR. Basically it works fine, and most of the transfers could be successfully handled.
But we still met an issue: When the transfer failed in IHEP->CNAF part (as figure 1 shows), the FTS3Agent can no longer update job status (as figure 2 shows). Then the transformation requests got stuck with status "Submitted". Neither re-submit, nor cancelled, just stay in submitted.
Is their any methods to re-activate those stuck requests?

Thank you,
Xuantong

[chaen]

Can you please get the output of

import fts3.rest.client.easy as fts3
context = fts3.Context(endpoint=<yourFTS>)
fts3.get_job_status(context, <yourFtsGUID>, list_files=True)

Also, can you please copy paste errors rather than screenshots ? It's easier to search :-)

[zhangxt-ihep]

Here is a typical transfer status. We are using the multihop transfer, the transfer will go through IHEP->CNAF->JINR.
In this failure, when it failed at IHEP->CNAF part, the CNAF->JINR part state would be "Not_used".
But in DIRAC/DataManagementSystem/Client/FTS3Job.py, line 181, the for-loop cannot handle the 2 file states with "Failed" and "Not_used" pair. It just goes to line 221 and raise an error. Then, this transfer will get stuck, and no longer resumed.

[zhangxuantong@lxlogin003:155 16:49 ~]$ fts-rest-transfer-status -s https://fts3.ihep.ac.cn:8446 03861268-a80c-11f0-8acc-7cde78c49152 -j | jq .
{
  "source_se": "https://junoeos01.ihep.ac.cn",
  "vo_name": "juno",
  "overwrite_flag": "Y",
  "retry_delay": 0,
  "source_spacetoken": null,
  "dest_se": "https://juno-se-dr01.jinr.ru",
  "reason": "One or more files failed. Please have a look at the details for more information",
  "job_finished": "2025-10-13T08:41:51",
  "os_project_id": null,
  "destination_spacetoken": null,
  "job_state": "FAILED",
  "submit_time": "2025-10-13T08:10:00",
  "copy_pin_lifetime": -1,
  "job_type": "H",
  "priority": 3,
  "bring_online": -1,
  "max_time_in_queue": null,
  "verify_checksum": "t",
  "cancel_job": false,
  "archive_timeout": -1,
  "submit_host": "fts3.ihep.ac.cn",
  "internal_job_params": "nostreams:1",
  "job_metadata": {
    "operationID": 9146792,
    "rmsReqID": 11009597,
    "sourceSE": "IHEP-JUNOEOS",
    "targetSE": "JINR-DCACHE",
    "useTokens": false,
    "auth_method": "certificate"
  },
  "job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
  "user_dn": "/C=CN/O=HEP/O=IHEP/OU=PHYS/CN=Kairui Huang",
  "dst_file_report": false,
  "cred_id": "5446836be3531bcf",
  "retry": 3,
  "http_status": "200 Ok",
  "files": [
    {
      "dest_surl_uuid": null,
      "source_surl": "https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
      "archive_start_time": null,
      "retry": 0,
      "log_file": "/var/log/fts3/transfers/2025-10-13/junoeos01.ihep.ac.cn__xfer-archive.cr.cnaf.infn.it/2025-10-13-0841__junoeos01.ihep.ac.cn__xfer-archive.cr.cnaf.infn.it__8890253857__03861268-a80c-11f0-8acc-7cde78c49152",
      "vo_name": "juno",
      "dest_surl": "https://xfer-archive.cr.cnaf.infn.it:8443/juno/storm/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
      "archive_finish_time": null,
      "user_filesize": 293300104,
      "activity": "default",
      "source_se": "https://junoeos01.ihep.ac.cn",
      "staging_host": null,
      "internal_file_params": "nostreams:1,timeout:1160,buffersize:0",
      "file_metadata": {
        "desc": "MultiHop 29457036"
      },
      "scitag": null,
      "log_debug": 0,
      "dest_se": "https://xfer-archive.cr.cnaf.infn.it",
      "reason": "TRANSFER [5] TRANSFER ERROR: Copy failed (3rd pull). Last attempt: Transfer failure: SocketTimeoutException while fetching https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd: Read timed out",
      "pid": 2658338,
      "staging_metadata": null,
      "src_token_id": null,
      "file_id": 8890253857,
      "recoverable": true,
      "priority": 3,
      "filesize": 293300104,
      "archive_metadata": null,
      "staging_finished": null,
      "dst_token_id": null,
      "hashed_id": 64402,
      "file_state": "FAILED",
      "checksum": "ADLER32:9331e14a",
      "tx_duration": 60.698,
      "staging_start": null,
      "file_state_initial": "",
      "file_index": 0,
      "transfer_host": "fts3.ihep.ac.cn",
      "finish_time": "2025-10-13T08:41:51",
      "throughput": 0.0,
      "selection_strategy": "auto",
      "job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
      "start_time": "2025-10-13T08:40:49",
      "bringonline_token": null
    },
    {
      "dest_surl_uuid": null,
      "source_surl": "https://xfer-archive.cr.cnaf.infn.it:8443/juno/storm/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
      "archive_start_time": null,
      "retry": 0,
      "log_file": null,
      "vo_name": "juno",
      "dest_surl": "https://juno-se-dr01.jinr.ru:2880/pnfs/jinr.ru/data/juno/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
      "archive_finish_time": null,
      "user_filesize": 293300104,
      "activity": "default",
      "source_se": "https://xfer-archive.cr.cnaf.infn.it",
      "staging_host": null,
      "internal_file_params": null,
      "file_metadata": {
        "desc": "Transfer 29457036",
        "fileID": 29457036
      },
      "scitag": null,
      "log_debug": null,
      "dest_se": "https://juno-se-dr01.jinr.ru",
      "reason": null,
      "pid": null,
      "staging_metadata": null,
      "src_token_id": null,
      "file_id": 8890253858,
      "recoverable": null,
      "priority": 3,
      "filesize": null,
      "archive_metadata": null,
      "staging_finished": null,
      "dst_token_id": null,
      "hashed_id": 64402,
      "file_state": "NOT_USED",
      "checksum": "ADLER32:9331e14a",
      "tx_duration": null,
      "staging_start": null,
      "file_state_initial": "",
      "file_index": 1,
      "transfer_host": null,
      "finish_time": null,
      "throughput": null,
      "selection_strategy": "auto",
      "job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
      "start_time": null,
      "bringonline_token": null
    }
  ],
  "dm": []
}

[zhangxiaomei]

Another problem related is that the request jobs containing these multi-hop transfers got stuck in "scheduled" status after 2 attempts:

How to deal with those requests?

[chaen]

Could you please try to apply the following PR to your servers ?
#8386

This should solve the problem hopefully. We never saw that problem in LHCb because the FTS servers are configured with CancelUnusedMultihopFiles=True, which means that we never have any Not_used transfer left.

[zhangxt-ihep]

We still don't try it yet as we found the large amount of "Not_used" transfers were caused by incorrect configuration of multi-hop sites. We used CNAF site as a bridge site for IHEP->JINR transfer (it works as IHEP->CNAF->JINR), but the we also submit a IHEP->CNAF transfer at the same time because the data need to be copied both to JINR and CNAF. So those 2 transfer running at the same time and caused a lot of failure because they are writing in same files in CNAF storage, which led to the second part (CNAF->JINR) got "not_used".

After we found this problem, we just use another CNAF storage path as bridge site, then this "Not_used" issue not happened again. So basically we don't trigger this issue any more. But any way, the issue still exist and we will applied this PR after we upgraded to 9.0 version of DIRAC. Thank you!

[chaen]

This advice is in the doc :-)

[zhangxt-ihep]

My apologies. If only I could read the documentation more carefully...