justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301824.15@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301824.15@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 05:51:28
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 07:10:44
From worker nodeHostnamecomp12-03
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 05:53:22
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2.root
JobscriptExit code0
Real time1h (4626s)
CPU time1h (4450s = 96%)
Outputting started2024-11-23 07:10:28
Output files
Finished2024-11-23 07:10:44
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

07:10:08 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:08 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:08 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:08 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:09 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:09 GMT run: 6457551 subRun: 1 event: 19300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:09 GMT run: 6457551 subRun: 1 event: 19300
1st wire C:0 T:15 P:2 W:669 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 456, 454, 527
minwire 0: 1837
minwire 1: 830
minwire 2: 1628
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0177387, 0.00176646, 0.980495, 
Output 1: 0.847137, 0.136244, 0.0132042, 0.00341491, 
Output 2: 0.950079, 0.0499207, 

600
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2013!
23-Nov-2024 07:10:27 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2_reco_data_2024-11-23T_055326Z.root"
23-Nov-2024 07:10:27 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/48/20/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.08821       44.4744       715.248       9.66643       115.392        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00989615     0.0390322     0.223492      0.025844      0.0430653       100    
reco:linecluster:LineCluster                           0.100176       1.2039        48.837       0.130877       6.13714        100    
reco:trajcluster:TrajCluster                           0.0291367      2.90202       83.732       0.0445805      11.3694        100    
reco:pandora:StandardPandora                            1.00675       4.55359       87.873        1.12588       14.6139        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000134369    0.0119361     0.316264     0.00146944     0.0378231       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000166687    0.0130426     0.338708     0.00257815     0.0379161       100    
reco:pandoracalo:Calorimetry                          9.8578e-05     0.0517142      1.58367     0.000912598    0.225177        100    
reco:pandorapid:Chi2ParticleID                         3.749e-05    0.000159764   0.00122479    0.000107347   0.000166019      100    
reco:trkshowersplit:TrackShowerHits                   0.000101577    0.0465098      1.34055     0.000717135     0.17695        100    
reco:pmtrack:PMAlgTrackMaker                          0.00112632      3.91865       108.85       0.168653       14.0359        100    
reco:pmtrackcalo:Calorimetry                          7.4927e-05     0.023603      0.655013     0.000811551    0.0855285       100    
reco:pmtrackpid:Chi2ParticleID                        3.0104e-05    0.000319787   0.00346754    0.000119529   0.000594704      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000189554     3.41161       87.847       0.236562       12.2825        100    
reco:pmtrajfitcalo:Calorimetry                        6.1001e-05     0.0539129      1.64455     0.00114787     0.232695        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6075e-05    0.000179743   0.00115607    0.000132118   0.000194206      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00104586      4.29062       79.8341      0.153557       13.4475        100    
reco:pmtracktccalo:Calorimetry                        6.9921e-05     0.015059      0.266038     0.000857874    0.0473235       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0512e-05    0.000298186   0.00240846    0.000122884   0.000487959      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000175697     2.34611       41.1093      0.226332       6.68369        100    
reco:pmtrajfittccalo:Calorimetry                      6.4814e-05     0.0377213     0.857605     0.000944201    0.130244        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6779e-05    0.000407674   0.00391497    0.00012733    0.000778827      100    
reco:blurredcluster:BlurredClustering                  0.0117253      1.25227       8.72169      0.410506       1.94446        100    
reco:emtrkmichelid:EmTrackMichelId                     0.270942       16.9853       203.571       3.24091       38.3105        100    
reco:emshower:EMShower                                0.000233101    0.0324088     0.516443     0.00467167     0.0680533       100    
reco:cvnmap:CVNMapper                                 2.1064e-05     0.0114675     0.035712      0.0127344    0.00933008       100    
reco:cvneva:CVNEvaluator                              1.5094e-05     0.524626       1.1689       0.737804      0.363578        100    
reco:energyrecnumu:EnergyReco                         0.000250679    0.0501795      1.1182       0.005172      0.150886        100    
reco:energyrecnue:EnergyReco                          0.000131385   0.00377767     0.0945695    0.000584643    0.0116151       100    
reco:energyrecnc:EnergyReco                           0.00179314    0.00535009     0.0963358    0.00214956     0.0116489       100    
reco:energyrecnumurange:EnergyReco                    0.000189657    0.0479131      1.10973     0.00194276     0.150255        100    
reco:energyrecnumumcs:EnergyReco                      0.00176513     0.0491889      1.11762     0.00267426     0.151125        100    
reco:opdec:Deconvolution                               0.0643576      0.28619       1.51836      0.245075      0.198488        100    
reco:ophitspe:OpHitFinderDeco                           0.22235       2.07553       5.04656       2.0824        0.92509        100    
reco:opflash:OpFlashFinder                            0.000220957    0.0011884    0.00276569    0.00115295    0.000532361      100    
reco:rns:RandomNumberSaver                            1.7483e-05    2.89381e-05   0.000231399   2.5574e-05    2.18547e-05      100    
reco:anglereconue:NuAngularReco                       0.00013399    0.00306924     0.0843003    0.000334663    0.0103286       100    
reco:anglereconumu:NuAngularReco                      6.3689e-05    0.00285684     0.0828971    0.00017912     0.0100895       100    
reco:anglereconuepfps:NuAngularReco                   0.000107269   0.00298564     0.0842269    0.000298985    0.0102284       100    
reco:anglereconumupfps:NuAngularReco                  8.9904e-05    0.00296758     0.0837495    0.000249506    0.0102959       100    
[art]:TriggerResults:TriggerResultInserter             8.526e-06    1.11119e-05   3.9594e-05    1.0424e-05    3.32192e-06      100    
end_path:out1:RootOutput                               2.235e-06    4.31402e-06   1.3152e-05     3.239e-06    2.19837e-06      100    
end_path:out1:RootOutput(write)                        0.0112784     0.215062       1.08966      0.202921      0.199159        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9765.89 MB
  Peak resident set size usage (VmHWM): 2615.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2_reco_2024-11-23T_055326Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_192_20231207T142940Z_gen_g4_detsim_hitreco__20240510T061029Z_reco2_reco_data_2024-11-23T_055326Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:53:45 UTC       justIN version: 01.01.09