Jobsub ID 301749.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301749.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:12:39
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:20:26
From worker node	Hostname	comp23-04
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:13:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3995s)
	CPU time	1h (3798s = 95%)
Outputting started	2024-11-23 06:20:11
Output files
Finished	2024-11-23 06:20:26
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ailure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:20:04 GMT run: 50577113 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:20:04 GMT run: 50577113 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:20:04 GMT run: 50577113 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:04 GMT run: 50577113 subRun: 1 event: 87200
1st wire C:0 T:9 P:2 W:645 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:04 GMT run: 50577113 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:20:05 GMT run: 50577113 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 57, 35, 40
minwire 0: 1384
minwire 1: 1962
minwire 2: 1056
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.995865, 0.000736076, 0.00339942, 
Output 1: 0.102872, 0.819264, 0.0735132, 0.00435104, 
Output 2: 0.996001, 0.00399901, 

553
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1878!
23-Nov-2024 06:20:09 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2_reco_data_2024-11-23T_051339Z.root"
23-Nov-2024 06:20:09 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/2f/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.718654       38.1311       988.452       9.82543       104.967        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0101262     0.0811137      1.38681      0.0266156     0.163162        100    
reco:linecluster:LineCluster                           0.101528       1.0902        68.6997      0.131826       6.83357        100    
reco:trajcluster:TrajCluster                           0.0286709      2.27103       140.415      0.0484593      14.1529        100    
reco:pandora:StandardPandora                           0.0615762      2.64886       98.9276       1.16031       9.8012         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000123507    0.0091909     0.320444     0.00172112     0.0331319       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000168498    0.0123922     0.257691     0.00292306     0.0313423       100    
reco:pandoracalo:Calorimetry                          0.000100543    0.0435623      2.96258     0.00101344     0.295222        100    
reco:pandorapid:Chi2ParticleID                        3.6408e-05    0.000158999   0.00114519    0.000120902   0.00015203       100    
reco:trkshowersplit:TrackShowerHits                   0.000109788    0.0350397      1.64047     0.000736723    0.169642        100    
reco:pmtrack:PMAlgTrackMaker                          0.00109956      2.85588       103.102      0.177872       10.893         100    
reco:pmtrackcalo:Calorimetry                          7.2578e-05     0.017487      0.892628     0.000928703    0.0902443       100    
reco:pmtrackpid:Chi2ParticleID                        2.9714e-05    0.000304442   0.00428526    0.000145592   0.000503524      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000194549     3.0377        118.284      0.292352       12.2113        100    
reco:pmtrajfitcalo:Calorimetry                        5.6459e-05     0.0431501      2.78594     0.00144217     0.277886        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7013e-05    0.000212488   0.00167853    0.000148285   0.000227026      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00103864      3.59723       116.029      0.207612       12.752         100    
reco:pmtracktccalo:Calorimetry                        7.0565e-05     0.0102902     0.364575     0.000950255    0.0389957       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8642e-05    0.000271016   0.00295627    0.000147073    0.0003758       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000189235     1.87741       57.8878       0.29836       6.19115        100    
reco:pmtrajfittccalo:Calorimetry                      6.2692e-05     0.0274338      1.32134     0.00127231     0.135536        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5169e-05    0.000388896   0.00488811    0.000163948   0.000626103      100    
reco:blurredcluster:BlurredClustering                 0.00166161      1.18252       9.44521       0.47079       1.67523        100    
reco:emtrkmichelid:EmTrackMichelId                     0.041569       15.7385       255.216       3.68286       32.1073        100    
reco:emshower:EMShower                                0.000225213    0.0299458     0.317532      0.0137989     0.0472659       100    
reco:cvnmap:CVNMapper                                 1.9493e-05     0.0147447     0.0470932     0.014015      0.0107126       100    
reco:cvneva:CVNEvaluator                               1.503e-05     0.570899      0.844253      0.732021      0.312867        100    
reco:energyrecnumu:EnergyReco                         0.000305852    0.0455994     0.910364     0.00514022     0.148112        100    
reco:energyrecnue:EnergyReco                          0.000134114   0.00297278     0.100435     0.000476399    0.0104477       100    
reco:energyrecnc:EnergyReco                           0.00172732    0.00469743     0.103278     0.00220703     0.0106083       100    
reco:energyrecnumurange:EnergyReco                    0.000206686    0.0430732     0.896089     0.00205734      0.14729        100    
reco:energyrecnumumcs:EnergyReco                      0.00169636     0.0443316     0.902075     0.00321374     0.147468        100    
reco:opdec:Deconvolution                               0.055874      0.310784       2.68969      0.264879      0.292044        100    
reco:ophitspe:OpHitFinderDeco                          0.351522       2.24396       5.41476       2.29694      0.902458        100    
reco:opflash:OpFlashFinder                            0.00031598     0.0014358    0.00519667    0.00135723    0.000735992      100    
reco:rns:RandomNumberSaver                            1.8724e-05    3.49072e-05   0.000247674   3.1495e-05    2.50278e-05      100    
reco:anglereconue:NuAngularReco                       0.000148696    0.0025496     0.0927002    0.000368546    0.0096125       100    
reco:anglereconumu:NuAngularReco                      6.6902e-05    0.00222431      0.09079     0.000208935   0.00935691       100    
reco:anglereconuepfps:NuAngularReco                   0.000113072   0.00235016     0.0916401    0.000317962   0.00943941       100    
reco:anglereconumupfps:NuAngularReco                  9.4789e-05    0.00229874     0.0923362    0.000275357    0.0095079       100    
[art]:TriggerResults:TriggerResultInserter             8.779e-06    1.24434e-05   4.2217e-05     1.103e-05    5.22158e-06      100    
end_path:out1:RootOutput                               2.557e-06    5.89551e-06   1.6947e-05     5.004e-06    2.75351e-06      100    
end_path:out1:RootOutput(write)                        0.0152216     0.227026       1.28638      0.205614      0.169702        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9736.92 MB
  Peak resident set size usage (VmHWM): 2584.15 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2_reco_2024-11-23T_051339Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_871_20231204T032724Z_gen_g4_detsim_hitreco__20240509T194724Z_reco2_reco_data_2024-11-23T_051339Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work