Jobsub ID 301684.64@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301684.64@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:50:09
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce13-multicore
Last heartbeat	2024-11-23 05:30:12
From worker node	Hostname	tds442.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:51:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5883s)
	CPU time	1h (5309s = 90%)
Outputting started	2024-11-23 05:29:48
Output files
Finished	2024-11-23 05:30:12
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:29:35 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:29:36 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:29:36 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:29:36 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:29:36 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:29:36 CET run: 50581281 subRun: 1 event: 98700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 26, 122, 75
minwire 0: 1542
minwire 1: 1240
minwire 2: 1545
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00174526, 0.000326976, 0.997928, 
Output 1: 0.00338062, 0.995274, 0.00132986, 1.56906e-05, 
Output 2: 0.998775, 0.00122477, 

457
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1457!
23-Nov-2024 06:29:46 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2_reco_data_2024-11-23T_035150Z.root"
23-Nov-2024 06:29:46 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ab/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.35659       54.1712       788.622       18.2141       114.846        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0435361     0.0836313     0.212249      0.0859076     0.0278062       100    
reco:linecluster:LineCluster                           0.255065      0.833738       15.1269      0.408492       1.91307        100    
reco:trajcluster:TrajCluster                           0.125908       2.35049       67.4761      0.144759       9.39235        100    
reco:pandora:StandardPandora                           0.150742       4.17881       74.5797       2.19943       9.08101        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000175994    0.0113217     0.271253     0.00263663     0.0304278       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000243596    0.0207042     0.536113     0.00523899     0.0585071       100    
reco:pandoracalo:Calorimetry                          0.000136984    0.0324642     0.902633     0.00176227     0.110661        100    
reco:pandorapid:Chi2ParticleID                        5.1381e-05    0.000276821    0.0040422    0.000196965   0.000419596      100    
reco:trkshowersplit:TrackShowerHits                   0.000147863    0.0466401      1.2286      0.00145421      0.17165        100    
reco:pmtrack:PMAlgTrackMaker                          0.00177711      4.40729       82.5535      0.373624       12.5589        100    
reco:pmtrackcalo:Calorimetry                          9.8563e-05     0.0341877      1.33818     0.00166125     0.157923        100    
reco:pmtrackpid:Chi2ParticleID                        3.9081e-05    0.000590292   0.00802132    0.000217345   0.00111129       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000280446     3.67762       75.6976      0.498606       10.5107        100    
reco:pmtrajfitcalo:Calorimetry                        8.9162e-05     0.0359011     0.946424     0.00270171     0.121094        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.8801e-05    0.000305459   0.00161395    0.00023592    0.000256829      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0015573      5.81888       116.307      0.488767       16.9748        100    
reco:pmtracktccalo:Calorimetry                        9.6262e-05     0.0110489     0.242651     0.00163948     0.0305184       100    
reco:pmtracktcpid:Chi2ParticleID                      3.8181e-05    0.000460506   0.00367822    0.00023271    0.000566364      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000255866     3.10075       67.6396      0.477629       8.34078        100    
reco:pmtrajfittccalo:Calorimetry                      9.3912e-05     0.0348491     0.880465     0.00200626      0.12844        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.7061e-05    0.000698426   0.00710127    0.000272051   0.00118261       100    
reco:blurredcluster:BlurredClustering                 0.000143783     1.58812       11.511       0.876625       2.03488        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000156683     23.1571       276.486       6.78335       45.0276        100    
reco:emshower:EMShower                                0.000262566    0.044866      0.504432     0.00533176     0.0809211       100    
reco:cvnmap:CVNMapper                                  2.676e-05     0.0158888     0.0665682     0.0142735     0.0140538       100    
reco:cvneva:CVNEvaluator                              2.2861e-05      1.15005       2.38249       1.36962      0.727318        100    
reco:energyrecnumu:EnergyReco                         0.000397919    0.0638353      1.54635      0.0104699     0.198856        100    
reco:energyrecnue:EnergyReco                          0.000169954   0.00418036     0.0980603     0.0010943     0.0111034       100    
reco:energyrecnc:EnergyReco                           0.00245552     0.0072384     0.0997591    0.00435121     0.0111252       100    
reco:energyrecnumurange:EnergyReco                    0.000303777    0.0523823      1.06992      0.0046191     0.159252        100    
reco:energyrecnumumcs:EnergyReco                       0.0015703     0.0539844      1.02159      0.0064537     0.154972        100    
reco:opdec:Deconvolution                               0.159472      0.425775      0.675736      0.442044      0.121021        100    
reco:ophitspe:OpHitFinderDeco                          0.391132       2.51462       4.51523       2.60251      0.900046        100    
reco:opflash:OpFlashFinder                            0.000381549   0.00231367    0.00588425    0.00205652     0.0011632       100    
reco:rns:RandomNumberSaver                             2.034e-05    5.91702e-05   0.000288986   4.7061e-05    4.28287e-05      100    
reco:anglereconue:NuAngularReco                       0.000206205   0.00299666     0.0821224    0.000732647   0.00915054       100    
reco:anglereconumu:NuAngularReco                      0.000102043   0.00272247     0.0819025    0.000390119   0.00923345       100    
reco:anglereconuepfps:NuAngularReco                   0.000165354   0.00289551     0.0808169    0.000538127   0.00906252       100    
reco:anglereconumupfps:NuAngularReco                  0.000159054   0.00281023     0.0806509    0.00044884    0.00900581       100    
[art]:TriggerResults:TriggerResultInserter             1.119e-05    2.21799e-05   0.000109352   1.6866e-05    1.51025e-05      100    
end_path:out1:RootOutput                               3.35e-06     9.79079e-06    2.943e-05    8.2555e-06    6.17176e-06      100    
end_path:out1:RootOutput(write)                        0.0178573     0.395295       1.67173      0.357306      0.293747        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21535.1 MB
  Peak resident set size usage (VmHWM): 2388.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2_reco_2024-11-23T_035150Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_986_20231204T053434Z_gen_g4_detsim_hitreco__20240509T214231Z_reco2_reco_data_2024-11-23T_035150Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work