Jobsub ID 108787.61@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.61@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat	2024-11-23 06:11:10
From worker node	Hostname	wn-sate-003.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2.root
Jobscript	Exit code	0
	Real time	50m (3018s)
	CPU time	47m (2863s = 94%)
Outputting started	2024-11-23 06:10:55
Output files
Finished	2024-11-23 06:11:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:45 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:45 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:45 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:45 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:10:50 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:10:50 CET run: 74552530 subRun: 1 event: 71500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 211, 167, 179
minwire 0: 1570
minwire 1: 612
minwire 2: 1840
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.854055, 0.115353, 0.0305922, 
Output 1: 0.616255, 0.303053, 0.0673964, 0.0132953, 
Output 2: 0.977122, 0.0228784, 

396
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1209!
23-Nov-2024 07:10:54 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2_reco_data_2024-11-23T_052040Z.root"
23-Nov-2024 07:10:54 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/1e/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.833404       28.1801       728.876       9.83809       80.1417        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0155234     0.0390173     0.107071      0.0375538     0.0132203       100    
reco:linecluster:LineCluster                           0.105703      0.554071       34.1729      0.160298       3.38607        100    
reco:trajcluster:TrajCluster                           0.0434708      1.15182       78.6213      0.0586359      7.87651        100    
reco:pandora:StandardPandora                           0.054401       3.85289       159.128       1.75471       15.8146        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00015968    0.00591908     0.103113     0.00198121     0.0144262       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000213852   0.00858786      0.14874     0.00338477     0.0178738       100    
reco:pandoracalo:Calorimetry                          0.000135284    0.0171276     0.709014      0.0012586     0.0779605       100    
reco:pandorapid:Chi2ParticleID                        4.9313e-05    0.000179168   0.00101434    0.000142532   0.00014978       100    
reco:trkshowersplit:TrackShowerHits                   0.000150143    0.0221862      1.1236      0.000980309    0.116469        100    
reco:pmtrack:PMAlgTrackMaker                          0.00159997      2.51195       99.5524      0.223711       10.6925        100    
reco:pmtrackcalo:Calorimetry                          8.6723e-05     0.0112014     0.420698     0.00108583     0.0461248       100    
reco:pmtrackpid:Chi2ParticleID                        3.9865e-05    0.000324084   0.00406424    0.000155828   0.000545851      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000235673     2.13418       60.668       0.339752       7.29586        100    
reco:pmtrajfitcalo:Calorimetry                        8.3406e-05     0.0179488      0.74002     0.00159853     0.0808031       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.7641e-05    0.000203563   0.00117554    0.000162165   0.000170319      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00142092      2.8233        97.9339      0.257812       11.0416        100    
reco:pmtracktccalo:Calorimetry                        8.5291e-05    0.00654031     0.139225     0.00115747     0.0191976       100    
reco:pmtracktcpid:Chi2ParticleID                       3.707e-05    0.000293846   0.00245298    0.000173903   0.000388298      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000216898     1.54664       36.1742      0.316295       4.45185        100    
reco:pmtrajfittccalo:Calorimetry                      8.4288e-05     0.0153494     0.504114     0.00128994     0.0601466       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5998e-05    0.000401942    0.0041289    0.000174242   0.000657003      100    
reco:blurredcluster:BlurredClustering                 8.3607e-05      1.0057        6.29593      0.501009       1.29557        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000150774     9.56153       144.799       2.85072       20.9167        100    
reco:emshower:EMShower                                0.000216728    0.0304896     0.194604     0.00688019     0.0475663       100    
reco:cvnmap:CVNMapper                                 2.7622e-05     0.0129412     0.039412      0.0146925    0.00989646       100    
reco:cvneva:CVNEvaluator                              2.1601e-05     0.763167       1.23816       1.0082       0.454067        100    
reco:energyrecnumu:EnergyReco                         0.000428627    0.0348213     0.558395     0.00514414     0.105053        100    
reco:energyrecnue:EnergyReco                          0.00018618    0.00199409     0.0317819    0.000611426   0.00439694       100    
reco:energyrecnc:EnergyReco                            0.0027714    0.00464318     0.0342306    0.00337628    0.00451246       100    
reco:energyrecnumurange:EnergyReco                    0.000330071    0.0325897     0.568131     0.00309759     0.104577        100    
reco:energyrecnumumcs:EnergyReco                      0.00196531     0.0343754     0.564671     0.00376231     0.105168        100    
reco:opdec:Deconvolution                               0.0803038     0.293219      0.584029      0.293672      0.0975165       100    
reco:ophitspe:OpHitFinderDeco                          0.248412       1.41623       2.81423       1.38946      0.552963        100    
reco:opflash:OpFlashFinder                            0.000287881   0.00144877    0.00402385    0.00138114    0.000694726      100    
reco:rns:RandomNumberSaver                            2.0989e-05    7.7363e-05    0.000570744   4.0732e-05    0.000106082      100    
reco:anglereconue:NuAngularReco                       0.00015947    0.00134321     0.0228249    0.000386913    0.0033891       100    
reco:anglereconumu:NuAngularReco                      9.0691e-05    0.00114245      0.02187     0.000222328    0.0033064       100    
reco:anglereconuepfps:NuAngularReco                   0.000144923    0.0013158     0.0220339    0.000381033   0.00333792       100    
reco:anglereconumupfps:NuAngularReco                  0.000136648   0.00124824     0.0250122    0.000299047   0.00350088       100    
[art]:TriggerResults:TriggerResultInserter             1.092e-05    1.44251e-05   6.2388e-05    1.2979e-05    5.54599e-06      100    
end_path:out1:RootOutput                               3.396e-06    7.51918e-06    2.126e-05     7.098e-06    2.78752e-06      100    
end_path:out1:RootOutput(write)                        0.0176907     0.258971       1.20959      0.257829      0.186426        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8303.19 MB
  Peak resident set size usage (VmHWM): 2333.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2_reco_2024-11-23T_052040Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_714_20231207T120938Z_gen_g4_detsim_hitreco__20240510T060306Z_reco2_reco_data_2024-11-23T_052040Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work