Jobsub ID 301620.129@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301620.129@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:21:37
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2024-11-23 02:22:02
From worker node	Hostname	iut2-c412.iu.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:25:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3350s)
	CPU time	51m (3090s = 92%)
Outputting started	2024-11-23 02:21:47
Output files
Finished	2024-11-23 02:22:02
List job events Wrapper job log
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
1st wire C:0 T:17 P:2 W:998 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
1st wire C:0 T:19 P:2 W:700 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:21:39 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:21:42 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:21:42 EST run: 66009532 subRun: 1 event: 5700
Comparing two wires in the same plane: return failure
%MSG
364
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1073!
22-Nov-2024 21:21:45 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2_reco_data_2024-11-23T_012600Z.root"
22-Nov-2024 21:21:45 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/1b/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.913212       30.5564       637.016       8.28087       81.8883        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0270668     0.0886145     0.265742      0.0813454     0.041529        100    
reco:linecluster:LineCluster                           0.204715      0.615057        26.52       0.267427       2.65219        100    
reco:trajcluster:TrajCluster                           0.0770083      1.09418       40.9397      0.0838881      5.16554        100    
reco:pandora:StandardPandora                           0.094989       2.64953       69.8612       1.27679       7.72308        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136776   0.00617072     0.122439     0.00151777     0.0161391       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187532    0.0137497     0.493294      0.0028225     0.0510623       100    
reco:pandoracalo:Calorimetry                          0.000124634    0.0215028     0.742604     0.000893963    0.0941537       100    
reco:pandorapid:Chi2ParticleID                        4.0807e-05    0.000202543   0.00152086    0.000153538   0.00018825       100    
reco:trkshowersplit:TrackShowerHits                   0.000115727    0.0323545      1.00466     0.000601753    0.139234        100    
reco:pmtrack:PMAlgTrackMaker                          0.00112935      2.84963       101.999      0.0945314      11.8364        100    
reco:pmtrackcalo:Calorimetry                           8.551e-05     0.0127504     0.273952     0.000775449    0.0458577       100    
reco:pmtrackpid:Chi2ParticleID                        3.1459e-05    0.000335869   0.00289292    0.000173119   0.000513406      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000183764     2.78345       117.562      0.123451       12.3756        100    
reco:pmtrajfitcalo:Calorimetry                        6.1605e-05     0.0242166     0.984986      0.001049      0.115055        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7221e-05    0.000214932   0.00140302    0.000155856   0.000205146      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000839713     3.12254       73.5376      0.113379       11.5956        100    
reco:pmtracktccalo:Calorimetry                        6.5122e-05    0.00839791     0.178944     0.000703399    0.0277831       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7261e-05    0.000318074   0.00279063    0.000168711   0.000487991      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000157575     1.66297       30.8229      0.151816       4.82205        100    
reco:pmtrajfittccalo:Calorimetry                      6.8248e-05     0.0177249     0.583923     0.000817989    0.0707128       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7031e-05    0.000388774   0.00431966    0.000165861   0.000655981      100    
reco:blurredcluster:BlurredClustering                 0.000155462    0.983443       8.09162      0.417471       1.40554        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000173145     11.0892       156.456       2.04021       26.3514        100    
reco:emshower:EMShower                                0.000252002    0.0296455     0.509703     0.00305307     0.069591        100    
reco:cvnmap:CVNMapper                                 2.5778e-05     0.0113105     0.044544     0.00978112     0.0107502       100    
reco:cvneva:CVNEvaluator                              1.7282e-05     0.775568       13.1874      0.902424       1.33071        100    
reco:energyrecnumu:EnergyReco                         0.000411852    0.0508884      1.36719     0.00374598     0.182456        100    
reco:energyrecnue:EnergyReco                          0.000141496   0.00236137     0.0416347    0.000588399   0.00531877       100    
reco:energyrecnc:EnergyReco                           0.00160685    0.00468733     0.0471885    0.00314416    0.00593859       100    
reco:energyrecnumurange:EnergyReco                    0.000181089    0.049415       1.30675      0.0029198     0.187903        100    
reco:energyrecnumumcs:EnergyReco                      0.00110776     0.0553204      1.74537     0.00339593      0.21949        100    
reco:opdec:Deconvolution                               0.0814947      0.35826      0.815599      0.319854      0.160373        100    
reco:ophitspe:OpHitFinderDeco                          0.259376       1.89135       3.91678       1.94303      0.829965        100    
reco:opflash:OpFlashFinder                            0.000344246   0.00164135    0.00420443    0.00154215    0.000746758      100    
reco:rns:RandomNumberSaver                            2.2572e-05    6.09108e-05   0.000316102   5.70725e-05   3.32634e-05      100    
reco:anglereconue:NuAngularReco                       0.000174157    0.0017254     0.0359911    0.000494332   0.00448501       100    
reco:anglereconumu:NuAngularReco                      7.1885e-05    0.00140016     0.0352279    0.00022912    0.00432178       100    
reco:anglereconuepfps:NuAngularReco                   0.000136787   0.00159897     0.0355975    0.000408566   0.00435785       100    
reco:anglereconumupfps:NuAngularReco                  9.1522e-05    0.00147245     0.0355809    0.00028789     0.0043679       100    
[art]:TriggerResults:TriggerResultInserter             9.467e-06    1.78276e-05    4.815e-05    1.6115e-05    6.49504e-06      100    
end_path:out1:RootOutput                               3.146e-06    7.81061e-06   2.1179e-05     7.639e-06    2.69636e-06      100    
end_path:out1:RootOutput(write)                        0.0158991     0.240202       1.15942      0.242633      0.202976        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17330.1 MB
  Peak resident set size usage (VmHWM): 2382.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2_reco_2024-11-23T_012600Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_56_20231203T223046Z_gen_g4_detsim_hitreco__20240509T205342Z_reco2_reco_data_2024-11-23T_012600Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work