Jobsub ID 108776.54@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108776.54@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:00:53
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:30:29
From worker node	Hostname	b9p08p3365.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:02:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2.root
Jobscript	Exit code	0
	Real time	28m (1682s)
	CPU time	35m (2145s = 127%)
Outputting started	2024-11-23 05:30:15
Output files
Finished	2024-11-23 05:30:29
List job events Wrapper job log
Jobscript log (last 10,000 characters)

640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:05 CET run: 50640579 subRun: 1 event: 25900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 295, 430, 309
minwire 0: 2511
minwire 1: 526
minwire 2: 2500
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.232239, 0.0151696, 0.752592, 
Output 1: 0.0801226, 0.360539, 0.29392, 0.265418, 
Output 2: 0.0575003, 0.9425, 

576
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1989!
23-Nov-2024 06:30:14 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2_reco_data_2024-11-23T_050217Z.root"
23-Nov-2024 06:30:14 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a6/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.142901       16.011        233.737       5.12706       33.0222        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0042937    0.00659828     0.0103915    0.00581424    0.00191126       100    
reco:linecluster:LineCluster                           0.0223508     0.0883159      1.38665      0.0452945     0.187204        100    
reco:trajcluster:TrajCluster                           0.0111425     0.364263       11.3013      0.0178416      1.46359        100    
reco:pandora:StandardPandora                           0.0192859      1.20683       14.8203      0.856147       1.62551        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000108091   0.00413943      0.09814     0.000849637    0.0115439       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000159642   0.00701196     0.109714     0.00189247     0.0135834       100    
reco:pandoracalo:Calorimetry                          8.4761e-05    0.00838293     0.223493     0.00056012     0.0302661       100    
reco:pandorapid:Chi2ParticleID                         3.038e-05    0.000121445   0.000874548   9.4406e-05    0.000113224      100    
reco:trkshowersplit:TrackShowerHits                   8.2391e-05     0.0127896     0.385318     0.000411118    0.0484933       100    
reco:pmtrack:PMAlgTrackMaker                          0.000873448     1.23146       22.3155      0.0641815      3.73401        100    
reco:pmtrackcalo:Calorimetry                          5.5451e-05    0.00619572     0.218376     0.000494644    0.0247433       100    
reco:pmtrackpid:Chi2ParticleID                         2.144e-05    0.000185188   0.00175698    9.74905e-05   0.000284385      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000147781     1.05042       26.6199       0.14808       3.12653        100    
reco:pmtrajfitcalo:Calorimetry                        5.1701e-05    0.00985214     0.256981     0.000867828    0.0343419       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.097e-05    0.000140274   0.000803097   0.000105886   0.00012831       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000708116     1.64892       33.1831      0.126632       5.15398        100    
reco:pmtracktccalo:Calorimetry                        5.3761e-05     0.0040207     0.128103     0.000509069    0.013945        100    
reco:pmtracktcpid:Chi2ParticleID                       2.157e-05    0.000173924   0.00156249    9.98905e-05   0.000226446      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000122731    0.931451       15.5673      0.152186       2.28348        100    
reco:pmtrajfittccalo:Calorimetry                       5.15e-05     0.00809189     0.250327     0.000568135    0.0295142       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.067e-05    0.000234486   0.00254514    0.000105091   0.00037982       100    
reco:blurredcluster:BlurredClustering                 9.5781e-05     0.759221       6.40575      0.314949       1.15459        100    
reco:emtrkmichelid:EmTrackMichelId                    9.6691e-05      6.66889       95.2853        1.489        14.0054        100    
reco:emshower:EMShower                                0.000137601    0.0171193     0.309206     0.00225553     0.0393848       100    
reco:cvnmap:CVNMapper                                  1.613e-05     0.0070155     0.0294128    0.00493567    0.00678557       100    
reco:cvneva:CVNEvaluator                               1.189e-05     0.512888      0.896295      0.672398      0.332766        100    
reco:energyrecnumu:EnergyReco                         0.000275592    0.0389789      1.51745     0.00267308     0.176388        100    
reco:energyrecnue:EnergyReco                          0.000101351   0.00142165     0.0323568    0.000337373   0.00379285       100    
reco:energyrecnc:EnergyReco                           0.00133174    0.00304184     0.035055     0.00189308     0.0040236       100    
reco:energyrecnumurange:EnergyReco                    0.000108441    0.0376505      1.51778     0.00169435     0.177787        100    
reco:energyrecnumumcs:EnergyReco                      0.000581895    0.039282       1.52798     0.00227193     0.181361        100    
reco:opdec:Deconvolution                               0.0144649     0.137693      0.264105      0.132597      0.0528475       100    
reco:ophitspe:OpHitFinderDeco                          0.0331671      1.03123       2.20254       0.97523      0.454746        100    
reco:opflash:OpFlashFinder                            9.4331e-05    0.000939877    0.0036458    0.000856568   0.000485171      100    
reco:rns:RandomNumberSaver                             1.43e-05     3.08814e-05   0.000277843   2.83355e-05   2.7654e-05       100    
reco:anglereconue:NuAngularReco                       7.7341e-05     0.0011213     0.0296045    0.000276743   0.00332164       100    
reco:anglereconumu:NuAngularReco                       5.354e-05    0.00094791     0.0285176    0.000143201   0.00322009       100    
reco:anglereconuepfps:NuAngularReco                   8.4561e-05    0.00106258     0.0285752    0.000237837   0.00322597       100    
reco:anglereconumupfps:NuAngularReco                  7.8011e-05    0.000994332    0.0289068    0.000192281   0.00325127       100    
[art]:TriggerResults:TriggerResultInserter              7.4e-06     9.80856e-06   4.5431e-05     8.96e-06     4.12977e-06      100    
end_path:out1:RootOutput                               2.59e-06     5.63486e-06    1.657e-05     5.695e-06    2.65195e-06      100    
end_path:out1:RootOutput(write)                       0.00745183     0.160506      0.717801      0.161663       0.11236        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21690.8 MB
  Peak resident set size usage (VmHWM): 2246.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2_reco_2024-11-23T_050217Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_258_20231207T162355Z_gen_g4_detsim_hitreco__20240510T051535Z_reco2_reco_data_2024-11-23T_050217Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work