Jobsub ID 79943.93@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 79943.93@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 3515 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-27 07:59:49 |
| Site | US_UCSD |
| Entry | CMSHTPC_T2_US_UCSD_gw6 |
| Last heartbeat | 2024-09-27 16:37:40 |
| From worker node | Hostname | mh-7662-13.t2.ucsd.edu |
| cpuinfo | AMD EPYC 7662 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-27 12:54:19 |
| Input files | hd-protodune:np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12110s) |
| CPU time | 55m (3334s = 27%) |
| Outputting started | 2024-09-27 16:16:10 |
| Output files | |
| Finished | 2024-09-27 16:37:40 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
63 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 56063 traces tagged "gauss"
FrameSaver: q=2.07434e+06 n=980661 tag=gauss
wclsFrameSaver: saving 73872 traces tagged "wiener"
FrameSaver: q=2.25897e+06 n=893549 tag=wiener
5314 hits in TPC 0
5587 hits in TPC 1
5690 hits in TPC 2
4012 hits in TPC 3
5678 hits in TPC 4
9519 hits in TPC 5
4696 hits in TPC 6
2185 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12 XUs and 11 XVs -> 0 XUVs
C:0 T:1 22476 XUs and 16030 XVs -> 827 XUVs
C:0 T:2 9712 XUs and 8911 XVs -> 312 XUVs
C:0 T:3 406 XUs and 357 XVs -> 17 XUVs
C:0 T:4 37 XUs and 38 XVs -> 1 XUVs
C:0 T:5 81902 XUs and 32900 XVs -> 1284 XUVs
C:0 T:6 8518 XUs and 7619 XVs -> 160 XUVs
C:0 T:7 842 XUs and 1285 XVs -> 61 XUVs
2662 XUVs total
1435 collection wire objects
2662 potential space points
Neighbour search...
47254 tests to find 28150 neighbours
Iterating with no regularization...
Begin: 3.82655e+07
0 3.65203e+07
1 3.64314e+07
2 3.64246e+07
Now with regularization...
Begin: 3.53892e+07
0 3.53868e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
27-Sep-2024 09:16:02 PDT Closed output file "np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup.root"
27-Sep-2024 09:16:02 PDT Closed input file "np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 51.0923 73.6564 100.438 74.7453 10.8618 35
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000317456 0.00167472 0.00331742 0.00151513 0.000722223 35
produce:fembfilter:PDHDFEMBFilter 0.560537 0.590118 0.641359 0.587439 0.0192298 35
produce:wclsdatahd:WireCellToolkit 27.9561 29.5126 31.5183 29.4466 0.726076 35
produce:gaushit:GausHitFinder 0.370899 0.66034 0.940147 0.674181 0.12785 35
produce:nhitsfilter:NumberOfHitsFilter 0.000170083 0.000339942 0.000533179 0.000340816 7.59089e-05 35
produce:reco3d:SpacePointSolver 0.196131 0.831836 2.79298 0.619797 0.549405 35
produce:hitpdune:DisambigFromSpacePoints 0.130751 0.441833 0.778345 0.45142 0.142889 35
produce:pandora:StandardPandora 4.89087 12.6579 24.8975 12.7296 4.70744 35
produce:pandoraWriter:StandardPandora 0.063458 0.131695 0.179039 0.137981 0.0255504 35
produce:pandoraTrack:LArPandoraTrackCreation 0.418641 1.01585 1.53423 1.08935 0.251853 35
produce:pandoraShower:LArPandoraShowerCreation 0.302572 1.18078 2.28795 1.18499 0.40961 35
produce:pandoracalo:Calorimetry 0.208749 0.501062 0.730263 0.516799 0.110103 35
produce:pandoracalonosce:Calorimetry 0.182913 0.457892 0.672392 0.473106 0.103466 35
produce:pandorapid:Chi2ParticleID 0.00128949 0.00167819 0.00290595 0.00161452 0.000285726 35
produce:pandoraShowercalo:ShowerCalorimetry 0.263304 1.04764 1.96134 1.06526 0.34098 35
produce:pandoraShowercalonosce:ShowerCalorimetry 0.25016 1.00517 1.87355 1.02348 0.326251 35
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0621482 0.108622 0.156866 0.110623 0.0192167 35
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0206965 0.0376653 0.0528264 0.0381779 0.00637015 35
produce:emtrkmichelid:EmTrackMichelId 12.3143 22.1804 30.8169 23.0334 3.92423 35
[art]:TriggerResults:TriggerResultInserter 1.408e-05 2.37201e-05 7.8341e-05 2.1831e-05 1.19686e-05 35
end_path:out1:RootOutput 3.45e-06 8.31311e-06 4.2871e-05 6.221e-06 6.45785e-06 35
end_path:michelremoving:michelremoving 0.00956749 0.0568173 0.305117 0.0207003 0.0653547 35
end_path:michelremovinggnocchi:michelremoving 0.0091636 0.054886 0.302742 0.0185068 0.0650044 35
end_path:calibanagnocchi:CalibAnaTree 0.135419 0.378395 0.529233 0.372213 0.0918376 35
end_path:out1:RootOutput(write) 0.394048 0.71432 1.03383 0.744478 0.129494 35
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4994.36 MB
Peak resident set size usage (VmHWM): 2640.65 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup.root
\tHists: np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup.root
Ran successfully
{
"name": "np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup.root",
"namespace": "hd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v09_91_02d01",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage2_calibration_protodunehd_keepup.fcl",
"dune.campaign": "hd-protodune-reco-keepup-v0",
"core.start_time": 1727451153.0,
"core.end_time": 1727453762.0,
"core.events": [
131322,
131326,
131330,
131334,
131338,
131342,
131346,
131350,
131354,
131358,
131362,
131366,
131370,
131374,
131378,
131382,
131386,
131390,
131394,
131398,
131402,
131406,
131410,
131414,
131418,
131422,
131426,
131430,
131434,
131438,
131442,
131446,
131450,
131454,
131458
],
"core.event_count": 35,
"core.first_event_number": 131322,
"core.last_event_number": 131458,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "hd-protodune",
"core.runs": [
27388
],
"core.runs_subruns": [
2738800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics",
"origin.applications.config_files": {
"reco1": "standard_reco_stage1_protodunehd_keepup.fcl"
},
"origin.applications.versions": {
"reco1": "v09_91_02d01"
},
"origin.applications.names": [
"reco1"
]
},
"parents": [
{
"did": "hd-protodune:np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517_reco_stage1_reco_stage2_20240927T161602_keepup_hists.root",
"namespace": "hd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v09_91_02d01",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage2_calibration_protodunehd_keepup.fcl",
"dune.campaign": "hd-protodune-reco-keepup-v0",
"core.start_time": 1727451153.0,
"core.end_time": 1727453762.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "hd-protodune",
"core.runs": [
27388
],
"core.runs_subruns": [
2738800001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics",
"origin.applications.config_files": {
"reco1": "standard_reco_stage1_protodunehd_keepup.fcl"
},
"origin.applications.versions": {
"reco1": "v09_91_02d01"
},
"origin.applications.names": [
"reco1"
]
},
"parents": [
{
"did": "hd-protodune:np04hd_raw_run027388_0938_dataflow2_datawriter_0_20240623T043517.hdf5"
}
]
} 