21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 421572.72@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 421572.72@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7908 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-25 08:23:39 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-06-25 13:46:01 | |
| From worker node | Hostname | node2b08.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-25 09:00:01 | |
| Input files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root | |
| Jobscript | Exit code | 0 |
| Real time | 4h (17000s) | |
| CPU time | 4h (16663s = 98%) | |
| Max RSS bytes | 2534137856 (2416 MiB) | |
| Outputting started | 2025-06-25 13:43:22 | |
| Output files | fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root | |
| Finished | 2025-06-25 13:46:01 | |
| Saved logs | justin-logs:421572.72-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG %MSG-e WireIDIntersectionCheck: PMAlgTrackMaker:pmtracktc@BeginModule 25-Jun-2025 14:40:58 BST run: 5013 subRun: 0 event: 22610 Comparing two wires in the same plane: return failure %MSG [2025-06-25 14:43:16.578212 +0100][Debug ][File ] [0xdbc0720@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/54/f9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e1055351-761c-4118-b055-b8bbbcc55c0b] Sending a read command for handle 0x0 to pubstor2306.fnal.gov:23489 [2025-06-25 14:43:16.578283 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0x19108a30 (message: kXR_read (handle: 0x00000000, offset: 2968773678, size: 1374004) ). [2025-06-25 14:43:16.578496 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0x19108a30 (message: kXR_read (handle: 0x00000000, offset: 2968773678, size: 1374004) ) from out-queu to in-queue. [2025-06-25 14:43:16.729392 +0100][Debug ][ExDbgMsg ] [msg: 0x444f7770] Assigned MsgHandler: 0x19108a30. [2025-06-25 14:43:16.729428 +0100][Debug ][ExDbgMsg ] [handler: 0x19108a30] Removed MsgHandler: 0x19108a30 from the in-queue. [2025-06-25 14:43:17.346817 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0x19108a30 (message: kXR_read (handle: 0x00000000, offset: 2968773678, size: 1374004) ) with status: [SUCCESS] . [2025-06-25 14:43:17.346858 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0x19108a30. 25-Jun-2025 14:43:18 BST Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root" [2025-06-25 14:43:18.491930 +0100][Debug ][File ] [0xdbc0720@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/54/f9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e1055351-761c-4118-b055-b8bbbcc55c0b] Sending a close command for handle 0x0 to pubstor2306.fnal.gov:23489 [2025-06-25 14:43:18.491973 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0xed4a190 (message: kXR_close (handle: 0x00000000) ). [2025-06-25 14:43:18.492150 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0xed4a190 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-06-25 14:43:18.607332 +0100][Debug ][ExDbgMsg ] [msg: 0x325d73c0] Assigned MsgHandler: 0xed4a190. [2025-06-25 14:43:18.607376 +0100][Debug ][ExDbgMsg ] [handler: 0xed4a190] Removed MsgHandler: 0xed4a190 from the in-queue. [2025-06-25 14:43:18.607420 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0xed4a190 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-06-25 14:43:18.607509 +0100][Debug ][File ] [0xdbc0720@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/54/f9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e1055351-761c-4118-b055-b8bbbcc55c0b] Close returned from pubstor2306.fnal.gov:23489 with: [SUCCESS] [2025-06-25 14:43:18.607564 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0xed4a190. 25-Jun-2025 14:43:18 BST Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/54/f9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 1402.56 1671.03 1906.48 1666.82 213.865 10 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00283957 0.0362752 0.16915 0.00634506 0.0616453 10 reco:pandora:StandardPandora 1255.43 1515.74 1750.2 1516.12 210.715 10 reco:pandoraTrack:LArPandoraTrackCreation 0.000206553 0.00555631 0.0403553 0.00259887 0.0116639 10 reco:pandoraShower:LArPandoraModularShowerCreation 0.000228701 0.00147152 0.00599834 0.000844914 0.00168677 10 reco:pandoracalo:Calorimetry 0.000155757 0.000774187 0.00232818 0.000302242 0.000795311 10 reco:pandorapid:Chi2ParticleID 3.5907e-05 0.000123 0.000636569 7.64115e-05 0.000171972 10 reco:linecluster:LineCluster 1.00019 1.29192 1.85171 1.27516 0.210425 10 reco:trajcluster:TrajCluster 0.228804 0.247988 0.266152 0.249653 0.0133373 10 reco:pmtracktc:PMAlgTrackMaker 3.97528 4.63018 5.10211 4.66638 0.355508 10 reco:emtrkmichelid:EmTrackMichelId 137.038 147.511 154.628 148.745 5.20136 10 reco:solarflash:SolarOpFlash 1.08936 1.37806 1.70172 1.35175 0.21809 10 [art]:TriggerResults:TriggerResultInserter 2.3239e-05 2.68489e-05 4.5542e-05 2.54525e-05 6.31432e-06 10 end_path:out1:RootOutput 4.768e-06 7.0454e-06 1.5669e-05 5.6305e-06 3.28129e-06 10 end_path:out1:RootOutput(write) 0.172388 0.184202 0.204859 0.182092 0.00864758 10 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3999.95 MB Peak resident set size usage (VmHWM): 2534.14 MB ==================================================================================================== [2025-06-25 14:43:18.960786 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-06-25 14:43:18.961306 +0100][Debug ][JobMgr ] Job manager stopped [2025-06-25 14:43:18.961369 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-06-25 14:43:18.961508 +0100][Debug ][TaskMgr ] Task manager stopped [2025-06-25 14:43:18.961513 +0100][Debug ][Poller ] Stopping the poller... [2025-06-25 14:43:18.961690 +0100][Debug ][AsyncSock ] [pubstor2306.fnal.gov:23489.0] Closing the socket [2025-06-25 14:43:18.961700 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:44124><--><[::ffff:131.225.69.92]:23489> Removing socket from the poller [2025-06-25 14:43:18.961746 +0100][Debug ][PostMaster ] [pubstor2306.fnal.gov:23489] Destroying stream [2025-06-25 14:43:18.961752 +0100][Debug ][AsyncSock ] [pubstor2306.fnal.gov:23489.0] Closing the socket Art has completed and will exit with status 0. Moving on to metadata extractor ============OUTPUT FILE: prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root Extracted metadata checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco.root MD conversion to MetaCat OK all-input-dids.txt debugprod.log jobscript.log justin-processed-pfns.txt old_md.json prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root.ext.json prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T223636Z_gen_002261_supernova_g4stage1_g4stage2_detsim_reco_20250625T090050Z_reco2.root.json reco2_hist.root