Jobsub ID 301641.175@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301641.175@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:12:27
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 04:00:48
From worker node	Hostname	comp23-22
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:23:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5832s)
	CPU time	1h (5375s = 92%)
Outputting started	2024-11-23 04:00:33
Output files
Finished	2024-11-23 04:00:48
List job events Wrapper job log
Jobscript log (last 10,000 characters)

in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:00:27 GMT run: 6209163 subRun: 1 event: 12600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:00:27 GMT run: 6209163 subRun: 1 event: 12600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:00:27 GMT run: 6209163 subRun: 1 event: 12600
1st wire C:0 T:13 P:2 W:685 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:00:27 GMT run: 6209163 subRun: 1 event: 12600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:00:28 GMT run: 6209163 subRun: 1 event: 12600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 42, 35, 31
minwire 0: 1823
minwire 1: 1381
minwire 2: 1592
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 47, 246
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.995413, 0.000300879, 0.0042856, 
Output 1: 0.822192, 0.1708, 0.00624164, 0.000766409, 
Output 2: 0.997417, 0.00258289, 

477
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1445!
23-Nov-2024 04:00:32 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2_reco_data_2024-11-23T_022325Z.root"
23-Nov-2024 04:00:32 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/89/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.608815       27.4572       437.728       10.4945       53.6518        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0281884     0.090538       0.55516      0.071779      0.0631897       200    
reco:linecluster:LineCluster                           0.0942618     0.493207       14.7167      0.272118       1.3488         200    
reco:trajcluster:TrajCluster                           0.0314836     0.872155       35.7975      0.112834       3.59411        200    
reco:pandora:StandardPandora                           0.0742047      2.00001       42.9667       1.26692       3.80345        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000128891    0.006339      0.140709     0.00168244     0.015368        200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000158711    0.0135955     0.499636     0.00245776     0.0447668       200    
reco:pandoracalo:Calorimetry                          9.2642e-05     0.0216338      1.31292     0.000959273    0.111417        200    
reco:pandorapid:Chi2ParticleID                         3.644e-05    0.000135776    0.0010297    0.000107515   0.000108354      200    
reco:trkshowersplit:TrackShowerHits                   8.1474e-05     0.0158181     0.539494     0.000671088    0.0601092       200    
reco:pmtrack:PMAlgTrackMaker                          0.00113649      1.91148       62.5655       0.14668       6.27139        200    
reco:pmtrackcalo:Calorimetry                          7.6115e-05    0.00944336       0.329      0.000755138    0.0340963       200    
reco:pmtrackpid:Chi2ParticleID                        2.9845e-05    0.000237136   0.00257605    0.000119061   0.000337774      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000195968     1.75538       49.7998      0.267869       5.07395        200    
reco:pmtrajfitcalo:Calorimetry                        6.1934e-05     0.0226664      1.34885     0.00126546     0.114457        200    
reco:pmtrajfitpid:Chi2ParticleID                      2.6092e-05    0.000158057   0.000746359   0.000130525   0.000120917      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00104759      2.11827       58.4089       0.17708       6.59342        200    
reco:pmtracktccalo:Calorimetry                         7.036e-05    0.00758527     0.249018     0.000775879    0.0234553       200    
reco:pmtracktcpid:Chi2ParticleID                      2.7936e-05    0.000219016   0.00156598    0.000126695   0.000261457      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000169508     1.20394       26.8566       0.21406       3.02778        200    
reco:pmtrajfittccalo:Calorimetry                      6.0425e-05     0.0145613     0.514718     0.000994572    0.0534189       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6084e-05    0.000293326   0.00321143    0.000135791   0.00043452       200    
reco:blurredcluster:BlurredClustering                 5.9513e-05      1.03885       8.00676      0.438268       1.35047        200    
reco:emtrkmichelid:EmTrackMichelId                    9.8549e-05      12.6644       178.483       3.86043       24.266         200    
reco:emshower:EMShower                                0.000150567    0.0244187     0.227683     0.00627906     0.039668        200    
reco:cvnmap:CVNMapper                                 1.7446e-05     0.012872      0.043975      0.0130844    0.00974906       200    
reco:cvneva:CVNEvaluator                              1.4492e-05     0.543852      0.841794      0.743969      0.340185        200    
reco:energyrecnumu:EnergyReco                         0.000253967    0.0288359     0.773198     0.00494372     0.0886318       200    
reco:energyrecnue:EnergyReco                          0.000131004   0.00198457     0.0521074    0.000521322   0.00496149       200    
reco:energyrecnc:EnergyReco                           0.00160435    0.00341093     0.0478067     0.0019508    0.00470676       200    
reco:energyrecnumurange:EnergyReco                    0.000195437    0.0262052     0.761264     0.00183529     0.0872559       200    
reco:energyrecnumumcs:EnergyReco                      0.00116589     0.0275509     0.757259     0.00254666     0.0883086       200    
reco:opdec:Deconvolution                               0.102636      0.372139       1.63808      0.362941      0.175214        200    
reco:ophitspe:OpHitFinderDeco                          0.105291       1.94465       3.53489       1.93677       0.73516        200    
reco:opflash:OpFlashFinder                            0.00026869    0.00122661    0.00290866    0.00121679    0.000485619      200    
reco:rns:RandomNumberSaver                             1.694e-05    2.68995e-05   0.000215767   2.36665e-05   1.5161e-05       200    
reco:anglereconue:NuAngularReco                       0.000140728    0.0014886     0.0388658    0.000326281   0.00387587       200    
reco:anglereconumu:NuAngularReco                      6.6572e-05    0.00129653     0.0391091    0.000195808   0.00382326       200    
reco:anglereconuepfps:NuAngularReco                   0.00010909     0.0013976     0.0399301    0.000289727   0.00387021       200    
reco:anglereconumupfps:NuAngularReco                  9.3173e-05    0.00136721     0.0396665    0.000255255   0.00385868       200    
[art]:TriggerResults:TriggerResultInserter             7.897e-06    9.94374e-06   3.3952e-05    9.4335e-06    2.44924e-06      200    
end_path:out1:RootOutput                               2.22e-06     3.9431e-06    1.2407e-05     2.968e-06    1.70258e-06      200    
end_path:out1:RootOutput(write)                        0.0123154     0.202049      0.907716      0.202963      0.138691        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9690.26 MB
  Peak resident set size usage (VmHWM): 2524.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2_reco_2024-11-23T_022325Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_62_20231122T131711Z_gen_g4_detsim_hitreco__20240507T181827Z_reco2_reco_data_2024-11-23T_022325Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work