Jobsub ID 79989.48@justin-prod-sched02.dune.hep.ac.uk
Jobsub ID | 79989.48@justin-prod-sched02.dune.hep.ac.uk |
Workflow ID | 3518 |
Stage ID | 1 |
User name | calcuttj@fnal.gov |
HTCondor Group | group_dune.prod_mcsim |
Requested | Processors | 1 |
RSS bytes | 4193255424 (3999 MiB) |
Wall seconds limit | 18000 (5 hours) |
Submitted time | 2024-09-27 08:41:25 |
Site | CA_SFU |
Entry | DUNE_CA_SFU_lcg-ce3 |
Last heartbeat | 2024-09-27 16:40:56 |
From worker node | Hostname | cdr166.int.cedar.computecanada.ca |
cpuinfo | Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz |
OS release | Scientific Linux release 7.9 (Nitrogen) |
Processors | 1 |
RSS bytes | 4193255424 (3999 MiB) |
Wall seconds limit | 84598 (23 hours) |
Inner Apptainer? | True |
Job state | stalled |
Allocator name | justin-allocator-pro.dune.hep.ac.uk |
Started | 2024-09-27 14:20:45 |
Input files | hd-protodune:np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030.hdf5
|
Outputting started | 2024-09-27 16:17:48 |
Output files | |
Finished | 2024-09-27 17:36:32 |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Retagger: tagging trace set: gauss with 49820 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 49820 traces tagged "gauss"
FrameSaver: q=1.77506e+06 n=880561 tag=gauss
wclsFrameSaver: saving 65711 traces tagged "wiener"
FrameSaver: q=1.94839e+06 n=806392 tag=wiener
5360 hits in TPC 0
4309 hits in TPC 1
3333 hits in TPC 2
1541 hits in TPC 3
7441 hits in TPC 4
8550 hits in TPC 5
5443 hits in TPC 6
3020 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11 XUs and 0 XVs -> 0 XUVs
C:0 T:1 12767 XUs and 7391 XVs -> 405 XUVs
C:0 T:2 11509 XUs and 6274 XVs -> 159 XUVs
C:0 T:3 37 XUs and 11 XVs -> 3 XUVs
C:0 T:4 39 XUs and 40 XVs -> 0 XUVs
C:0 T:5 45093 XUs and 41288 XVs -> 1090 XUVs
C:0 T:6 9085 XUs and 8594 XVs -> 489 XUVs
C:0 T:7 280 XUs and 285 XVs -> 7 XUVs
2153 XUVs total
1213 collection wire objects
2153 potential space points
Neighbour search...
32437 tests to find 19522 neighbours
Iterating with no regularization...
Begin: 2.17207e+07
0 2.07993e+07
1 2.07279e+07
2 2.07226e+07
Now with regularization...
Begin: 1.9717e+07
0 1.97156e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
27-Sep-2024 09:17:39 PDT Closed output file "np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup.root"
27-Sep-2024 09:17:39 PDT Closed input file "np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 81.5193 124.26 203.826 117.346 27.0608 35
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000505905 0.00195474 0.00454169 0.00193378 0.000989313 35
produce:fembfilter:PDHDFEMBFilter 0.898871 0.916288 0.957958 0.914368 0.0103407 35
produce:wclsdatahd:WireCellToolkit 51.5678 53.9183 56.3959 53.8224 1.06636 35
produce:gaushit:GausHitFinder 0.517393 1.02084 1.66304 0.953418 0.279649 35
produce:nhitsfilter:NumberOfHitsFilter 0.000202783 0.000359795 0.000618676 0.000337531 0.00010663 35
produce:reco3d:SpacePointSolver 0.274024 1.07586 6.89224 0.749241 1.11833 35
produce:hitpdune:DisambigFromSpacePoints 0.261368 0.846027 1.83206 0.729893 0.441271 35
produce:pandora:StandardPandora 5.7441 21.4118 60.5701 18.3201 11.2235 35
produce:pandoraWriter:StandardPandora 0.0899139 0.179443 0.294899 0.168456 0.0488404 35
produce:pandoraTrack:LArPandoraTrackCreation 0.774858 1.88201 5.16748 1.66641 0.964383 35
produce:pandoraShower:LArPandoraShowerCreation 0.499596 2.38341 9.15446 1.93137 1.60872 35
produce:pandoracalo:Calorimetry 0.403083 0.945084 2.30817 0.808457 0.430584 35
produce:pandoracalonosce:Calorimetry 0.357171 0.879986 2.15338 0.761762 0.416429 35
produce:pandorapid:Chi2ParticleID 0.00169107 0.00232537 0.00425751 0.00224928 0.000555954 35
produce:pandoraShowercalo:ShowerCalorimetry 0.437294 2.15268 6.25315 1.81412 1.3056 35
produce:pandoraShowercalonosce:ShowerCalorimetry 0.424971 2.07764 6.23932 1.74844 1.27755 35
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0926749 0.166778 0.268693 0.161953 0.0456333 35
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.033223 0.0610954 0.0992882 0.060478 0.0163389 35
produce:emtrkmichelid:EmTrackMichelId 17.4538 32.2901 52.1176 30.3915 8.37115 35
[art]:TriggerResults:TriggerResultInserter 2.4649e-05 3.47903e-05 8.127e-05 2.8751e-05 1.48819e-05 35
end_path:out1:RootOutput 1.0478e-05 1.35832e-05 4.5343e-05 1.1312e-05 6.44266e-06 35
end_path:michelremoving:michelremoving 0.0228969 0.0899033 0.295947 0.0458607 0.073363 35
end_path:michelremovinggnocchi:michelremoving 0.0202261 0.0855913 0.290045 0.0394857 0.0737813 35
end_path:calibanagnocchi:CalibAnaTree 0.311581 0.751107 2.04469 0.644518 0.379838 35
end_path:out1:RootOutput(write) 0.533271 1.0324 1.69734 0.954812 0.276838 35
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5002.15 MB
Peak resident set size usage (VmHWM): 2682.31 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup.root
\tHists: np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup_hists.root
Forming reco metadata
Successfully opened file np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup.root
Ran successfully
{
"name": "np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup.root",
"namespace": "hd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v09_91_02d01",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage2_calibration_protodunehd_keepup.fcl",
"dune.campaign": "hd-protodune-reco-keepup-v0",
"core.start_time": 1727449461.0,
"core.end_time": 1727453860.0,
"core.events": [
17924,
17928,
17932,
17936,
17940,
17944,
17948,
17952,
17956,
17960,
17964,
17968,
17972,
17976,
17980,
17984,
17988,
17992,
17996,
18000,
18004,
18008,
18012,
18016,
18020,
18024,
18028,
18032,
18036,
18040,
18044,
18048,
18052,
18056,
18060
],
"core.event_count": 35,
"core.first_event_number": 17924,
"core.last_event_number": 18060,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "hd-protodune",
"core.runs": [
27406
],
"core.runs_subruns": [
2740600001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics",
"origin.applications.config_files": {
"reco1": "standard_reco_stage1_protodunehd_keepup.fcl"
},
"origin.applications.versions": {
"reco1": "v09_91_02d01"
},
"origin.applications.names": [
"reco1"
]
},
"parents": [
{
"did": "hd-protodune:np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030_reco_stage1_reco_stage2_20240927T161739_keepup_hists.root",
"namespace": "hd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v09_91_02d01",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage2_calibration_protodunehd_keepup.fcl",
"dune.campaign": "hd-protodune-reco-keepup-v0",
"core.start_time": 1727449461.0,
"core.end_time": 1727453860.0,
"core.data_stream": "cosmics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "hd-protodune",
"core.runs": [
27406
],
"core.runs_subruns": [
2740600001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics",
"origin.applications.config_files": {
"reco1": "standard_reco_stage1_protodunehd_keepup.fcl"
},
"origin.applications.versions": {
"reco1": "v09_91_02d01"
},
"origin.applications.names": [
"reco1"
]
},
"parents": [
{
"did": "hd-protodune:np04hd_raw_run027406_0128_dataflow0_datawriter_0_20240624T142030.hdf5"
}
]
}