Jobsub ID 108775.116@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108775.116@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:59:49
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:42:27
From worker node	Hostname	b9p11p6229.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:01:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2.root
Jobscript	Exit code	0
	Real time	40m (2435s)
	CPU time	43m (2600s = 106%)
Outputting started	2024-11-23 05:42:09
Output files
Finished	2024-11-23 05:42:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:42:01 CET run: 66009532 subRun: 1 event: 113300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:42:01 CET run: 66009532 subRun: 1 event: 113300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:42:03 CET run: 66009532 subRun: 1 event: 113300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:42:03 CET run: 66009532 subRun: 1 event: 113300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:42:03 CET run: 66009532 subRun: 1 event: 113300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 68, 65, 60
minwire 0: 2614
minwire 1: 225
minwire 2: 2685
Used alternate method to get min and max wires due to vertex determination failure: 64, 263
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 153, 352
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 18, 217
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.989696, 0.000544978, 0.0097589, 
Output 1: 0.0664086, 0.363124, 0.465763, 0.104704, 
Output 2: 0.989305, 0.0106952, 

398
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1341!
23-Nov-2024 06:42:07 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2_reco_data_2024-11-23T_050138Z.root"
23-Nov-2024 06:42:07 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b4/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.714129       22.4929       245.627       7.85083       44.0882        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0168763     0.0393894     0.0796384     0.0411026     0.0115092       100    
reco:linecluster:LineCluster                           0.144906      0.254205       3.69805      0.166145      0.427928        100    
reco:trajcluster:TrajCluster                           0.0472816     0.546656       11.7997      0.0572579      1.7876         100    
reco:pandora:StandardPandora                           0.058918       1.83566       23.1286       1.25096       2.83218        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000107971   0.00546829     0.0726243    0.00119929     0.0120771       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000149322    0.0142262     0.413615     0.00266266     0.0463075       100    
reco:pandoracalo:Calorimetry                          8.4011e-05     0.0145172     0.300237     0.000795855    0.0465843       100    
reco:pandorapid:Chi2ParticleID                         3.224e-05    0.00015553    0.00122666    0.000124476   0.000145802      100    
reco:trkshowersplit:TrackShowerHits                   7.4881e-05     0.0172053     0.349854     0.000540577    0.0537382       100    
reco:pmtrack:PMAlgTrackMaker                          0.000918512     1.82309       29.6381      0.135629       4.95617        100    
reco:pmtrackcalo:Calorimetry                          5.7371e-05    0.00772674     0.143225     0.000751725    0.0222227       100    
reco:pmtrackpid:Chi2ParticleID                         2.25e-05     0.000256286   0.00209513    0.000137207   0.000344632      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000137862     1.63422       38.4341      0.195972       4.65926        100    
reco:pmtrajfitcalo:Calorimetry                        5.2331e-05     0.0155888     0.369999     0.00109327     0.0536846       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.223e-05    0.000182488   0.00111042    0.000141237   0.000161745      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00076483      2.52096       46.6962      0.139134       7.69147        100    
reco:pmtracktccalo:Calorimetry                        5.7061e-05     0.0069767      0.14694     0.000801955    0.0212765       100    
reco:pmtracktcpid:Chi2ParticleID                       2.179e-05    0.000253132   0.00222766    0.000129781   0.000389913      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000125252     1.38466       16.4913      0.228708       3.33619        100    
reco:pmtrajfittccalo:Calorimetry                      5.1361e-05     0.0125741     0.194634     0.000942962    0.0374695       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.104e-05    0.000325382   0.00259946    0.000148932   0.000515691      100    
reco:blurredcluster:BlurredClustering                 7.9181e-05     0.864134       7.40377      0.410987       1.25361        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0178566      8.51479       81.2653       2.1288        17.5782        100    
reco:emshower:EMShower                                0.000190832    0.0288179     0.323392     0.00560909     0.0584093       100    
reco:cvnmap:CVNMapper                                  1.924e-05     0.0105627     0.0546235     0.0104189     0.0100058       100    
reco:cvneva:CVNEvaluator                               1.462e-05     0.691466       4.64218      0.744346      0.574313        100    
reco:energyrecnumu:EnergyReco                         0.000269753    0.0214075      0.59152     0.00371219     0.0805683       100    
reco:energyrecnue:EnergyReco                          0.000108052   0.00195492     0.0237436    0.000457951   0.00426173       100    
reco:energyrecnc:EnergyReco                           0.00155981    0.00427161     0.027996     0.00307153    0.00477669       100    
reco:energyrecnumurange:EnergyReco                    0.000138862    0.0194732     0.574143     0.00192597     0.0791199       100    
reco:energyrecnumumcs:EnergyReco                      0.00147861     0.0191386     0.475184     0.00357263     0.0664312       100    
reco:opdec:Deconvolution                               0.0756528      0.25945      0.472712      0.262752      0.0857638       100    
reco:ophitspe:OpHitFinderDeco                          0.290953       1.69223       4.19242       1.64949      0.861492        100    
reco:opflash:OpFlashFinder                            0.000211933   0.00133189    0.00663007    0.00119305    0.000873218      100    
reco:rns:RandomNumberSaver                             1.626e-05    5.01772e-05   0.000527436   3.16305e-05   7.77108e-05      100    
reco:anglereconue:NuAngularReco                       0.000130671   0.00150729     0.0280791    0.000367014   0.00388658       100    
reco:anglereconumu:NuAngularReco                       5.842e-05    0.00126822     0.0277197    0.000181618   0.00383938       100    
reco:anglereconuepfps:NuAngularReco                   9.8421e-05    0.00139354     0.0271433    0.000328149   0.00379705       100    
reco:anglereconumupfps:NuAngularReco                  8.0011e-05    0.00131543     0.026602     0.000254918    0.0037543       100    
[art]:TriggerResults:TriggerResultInserter             8.14e-06     1.26098e-05   4.0171e-05     1.134e-05    4.68578e-06      100    
end_path:out1:RootOutput                               2.61e-06     5.70229e-06    1.948e-05     5.44e-06     2.84504e-06      100    
end_path:out1:RootOutput(write)                        0.0174758     0.222129      0.973308      0.200657      0.182719        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21651 MB
  Peak resident set size usage (VmHWM): 2219.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2_reco_2024-11-23T_050138Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1132_20231203T235822Z_gen_g4_detsim_hitreco__20240509T202705Z_reco2_reco_data_2024-11-23T_050138Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work