justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301653.100@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301653.100@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 02:35:32
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 03:39:32
From worker nodeHostnamecomp08-18
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 02:43:46
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2.root
JobscriptExit code0
Real time55m (3330s)
CPU time49m (2974s = 89%)
Outputting started2024-11-23 03:39:17
Output files
Finished2024-11-23 03:39:32
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

9:07 GMT run: 6220954 subRun: 1 event: 17200
1st wire C:0 T:6 P:2 W:705 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:39:07 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:39:07 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:39:07 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:39:07 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:39:07 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:39:10 GMT run: 6220954 subRun: 1 event: 17200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 87, 73, 80
minwire 0: 945
minwire 1: 1650
minwire 2: 679
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.971478, 0.00445216, 0.0240702, 
Output 1: 0.0583916, 0.367107, 0.561293, 0.0132079, 
Output 2: 0.931906, 0.0680938, 

492
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1727!
23-Nov-2024 03:39:15 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2_reco_data_2024-11-23T_024350Z.root"
23-Nov-2024 03:39:15 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/75/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.808899       30.1365       324.122       10.5306       58.7678        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0112627     0.112695      0.800619      0.0928324     0.0946699       100    
reco:linecluster:LineCluster                           0.0871483     0.575207       11.7943      0.307775       1.46563        100    
reco:trajcluster:TrajCluster                           0.0319344      1.30911       31.299       0.113716       5.11662        100    
reco:pandora:StandardPandora                           0.072363       2.07954       17.339        1.40158       2.82247        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139823   0.00635329     0.082575     0.00146412     0.0144711       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000177256    0.0117916     0.219023     0.00268865     0.0301164       100    
reco:pandoracalo:Calorimetry                          0.000109851    0.0217324     0.504559     0.000903079    0.0773283       100    
reco:pandorapid:Chi2ParticleID                        3.8843e-05    0.000157514   0.00167345    0.000121099   0.000177508      100    
reco:trkshowersplit:TrackShowerHits                   0.000115256    0.0228015     0.634364     0.000622162    0.0883303       100    
reco:pmtrack:PMAlgTrackMaker                          0.00128158      2.1109        39.718       0.148101       6.49964        100    
reco:pmtrackcalo:Calorimetry                          8.5381e-05    0.00933437     0.171803     0.000845016    0.0283416       100    
reco:pmtrackpid:Chi2ParticleID                        3.4408e-05    0.000253377   0.00187051    0.00014164    0.000361514      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000195557     2.14558       38.7199      0.221822       6.30738        100    
reco:pmtrajfitcalo:Calorimetry                        6.2615e-05     0.0226306     0.498212     0.00132018     0.0783315       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7517e-05    0.000174182   0.000612282   0.000147476   0.000126012      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00109755      2.6226        44.9002      0.173795       7.64132        100    
reco:pmtracktccalo:Calorimetry                        7.5837e-05    0.00723883     0.0986615    0.000836259    0.0185764       100    
reco:pmtracktcpid:Chi2ParticleID                       2.983e-05    0.000230973   0.00148441    0.000139361   0.000283239      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000189652     1.39518       17.4338      0.190501       3.29388        100    
reco:pmtrajfittccalo:Calorimetry                      6.4879e-05     0.0150292     0.260997     0.000948108    0.0478016       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7784e-05    0.000319328   0.00269959    0.000146405   0.000495926      100    
reco:blurredcluster:BlurredClustering                 6.6311e-05      1.12001       7.14756      0.501918       1.47342        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000115338     12.8068       128.086       3.24575       25.0657        100    
reco:emshower:EMShower                                0.000168673    0.0249347     0.251435     0.00891671     0.0409838       100    
reco:cvnmap:CVNMapper                                 1.9413e-05     0.0142529     0.0408211     0.0136462    0.00955345       100    
reco:cvneva:CVNEvaluator                              1.5788e-05     0.657344       1.04699      0.828873      0.362032        100    
reco:energyrecnumu:EnergyReco                         0.000370882    0.0216893     0.406275     0.00502526     0.0564846       100    
reco:energyrecnue:EnergyReco                          0.000153797   0.00206754     0.0280717    0.000597519   0.00454319       100    
reco:energyrecnc:EnergyReco                           0.00194825    0.00375847     0.0298337    0.00234351    0.00461179       100    
reco:energyrecnumurange:EnergyReco                    0.000234334    0.0190055     0.402033     0.00203733     0.0559558       100    
reco:energyrecnumumcs:EnergyReco                      0.000908372    0.0202149     0.402858     0.00282825     0.0558113       100    
reco:opdec:Deconvolution                               0.0910795     0.402375       1.31043      0.377659      0.170911        100    
reco:ophitspe:OpHitFinderDeco                          0.102919       2.33966       4.1191        2.29313      0.916353        100    
reco:opflash:OpFlashFinder                            0.000241788   0.00130131    0.00382059    0.00123918    0.000567773      100    
reco:rns:RandomNumberSaver                            2.0436e-05    3.60995e-05   0.000297522   3.29985e-05   2.82726e-05      100    
reco:anglereconue:NuAngularReco                       0.00014605    0.00148455     0.023862     0.000357252   0.00374226       100    
reco:anglereconumu:NuAngularReco                      6.8371e-05    0.00126349     0.0226105    0.000208027   0.00361233       100    
reco:anglereconuepfps:NuAngularReco                   0.000113368   0.00138067     0.0226788    0.000302789   0.00361413       100    
reco:anglereconumupfps:NuAngularReco                  0.000100439   0.00133452     0.0227394    0.000284731   0.00362738       100    
[art]:TriggerResults:TriggerResultInserter             9.519e-06    1.33438e-05   6.6476e-05    1.19715e-05   6.23406e-06      100    
end_path:out1:RootOutput                               2.974e-06    5.47323e-06   1.9378e-05    5.1225e-06    2.47628e-06      100    
end_path:out1:RootOutput(write)                        0.0117827     0.226744      0.783068      0.217992      0.151867        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9562.48 MB
  Peak resident set size usage (VmHWM): 2412.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2_reco_2024-11-23T_024350Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_171_20231122T082847Z_gen_g4_detsim_hitreco__20240503T065111Z_reco2_reco_data_2024-11-23T_024350Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-27 03:11:10 UTC       justIN version: 01.01.09