justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301713.4@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301713.4@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:45:13
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2024-11-23 05:15:47
From worker nodeHostnameb9p08p3365.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:45:40
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2.root
JobscriptExit code0
Real time29m (1791s)
CPU time33m (2030s = 113%)
Outputting started2024-11-23 05:15:32
Output files
Finished2024-11-23 05:15:47
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:15:27 CET run: 6418365 subRun: 1 event: 22799
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:15:27 CET run: 6418365 subRun: 1 event: 22799
Comparing two wires in the same plane: return failure
%MSG
263
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 557!
Begin processing the 100th record. run: 6418365 subRun: 1 event: 22800 at 23-Nov-2024 06:15:28 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:15:29 CET run: 6418365 subRun: 1 event: 22800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:15:29 CET run: 6418365 subRun: 1 event: 22800
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:15:30 CET run: 6418365 subRun: 1 event: 22800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:15:30 CET run: 6418365 subRun: 1 event: 22800
Comparing two wires in the same plane: return failure
%MSG
281
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 635!
23-Nov-2024 06:15:31 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2_reco_data_2024-11-23T_044544Z.root"
23-Nov-2024 06:15:31 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/46/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.558253       16.0595       208.599       5.87415       28.056         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0163679     0.0380965     0.0663258     0.0397115     0.0112532       100    
reco:linecluster:LineCluster                           0.100106      0.201732        1.249       0.168648      0.157621        100    
reco:trajcluster:TrajCluster                           0.0474878     0.294477       5.12491      0.0533057     0.792016        100    
reco:pandora:StandardPandora                           0.0555083      1.2922        19.3229      0.901881       1.94744        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00010795    0.00383792     0.072623     0.000910993   0.00858688       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000160822   0.00815322     0.0826079    0.00186778     0.0146089       100    
reco:pandoracalo:Calorimetry                          8.7781e-05     0.0082833     0.171424     0.000614885    0.0233312       100    
reco:pandorapid:Chi2ParticleID                         3.416e-05    0.00012919    0.00161338    9.4781e-05    0.000167687      100    
reco:trkshowersplit:TrackShowerHits                   6.7601e-05     0.0101212     0.242533     0.000402858    0.0316374       100    
reco:pmtrack:PMAlgTrackMaker                          0.000911998     1.20345       22.4952      0.0921126      3.18961        100    
reco:pmtrackcalo:Calorimetry                          5.4671e-05    0.00528435     0.148544     0.00056726     0.0166998       100    
reco:pmtrackpid:Chi2ParticleID                         2.231e-05    0.000183402   0.00167439    0.000100936   0.000252368      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154291     1.05107       21.4338      0.135448       2.72858        100    
reco:pmtrajfitcalo:Calorimetry                         5.264e-05    0.00880739     0.199401     0.000803797    0.0254625       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.191e-05    0.000127628   0.000616016   9.8911e-05    0.000103171      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000731666     1.47972       27.9098      0.122207       4.06362        100    
reco:pmtracktccalo:Calorimetry                         5.325e-05    0.00379795     0.0570623    0.00056259    0.00856691       100    
reco:pmtracktcpid:Chi2ParticleID                      2.1631e-05    0.000164801    0.0011391    0.000107241   0.000201098      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000120021    0.950919       18.3901      0.149865        2.292         100    
reco:pmtrajfittccalo:Calorimetry                      5.0941e-05    0.00766694     0.211345     0.000715286    0.0246416       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.147e-05    0.000227972   0.00231478    0.000105141   0.000342743      100    
reco:blurredcluster:BlurredClustering                 8.6121e-05      0.8177        6.82802      0.356239       1.11322        100    
reco:emtrkmichelid:EmTrackMichelId                    8.8881e-05      6.51061       80.249        1.42355       11.9091        100    
reco:emshower:EMShower                                0.000120371    0.0184845     0.303505     0.00257636     0.0391705       100    
reco:cvnmap:CVNMapper                                  2.325e-05    0.00687689     0.0453312    0.00580951     0.0072693       100    
reco:cvneva:CVNEvaluator                              1.3951e-05     0.475527      0.849878      0.676177      0.336125        100    
reco:energyrecnumu:EnergyReco                         0.000260222    0.027027      0.900265     0.00278599     0.100013        100    
reco:energyrecnue:EnergyReco                          0.000114491   0.00128043     0.0215062    0.000370419   0.00262166       100    
reco:energyrecnc:EnergyReco                           0.00127786    0.00280337     0.0242303    0.00181638    0.00292111       100    
reco:energyrecnumurange:EnergyReco                    0.000117381    0.0257648      0.91918     0.00158783      0.10149        100    
reco:energyrecnumumcs:EnergyReco                      0.000898867    0.026189       0.87098     0.00187696     0.0970505       100    
reco:opdec:Deconvolution                               0.0525787     0.232252      0.386595      0.234397      0.0672047       100    
reco:ophitspe:OpHitFinderDeco                          0.175944       1.18026       2.12202       1.19499      0.464038        100    
reco:opflash:OpFlashFinder                            0.000161642   0.00110085    0.00572615    0.00103593    0.000637313      100    
reco:rns:RandomNumberSaver                            1.6251e-05    3.55751e-05   0.000411034    3.246e-05    3.89318e-05      100    
reco:anglereconue:NuAngularReco                       0.000129001   0.000967049    0.0184378    0.000303018   0.00215997       100    
reco:anglereconumu:NuAngularReco                      5.8891e-05    0.000788686     0.01804     0.000142486   0.00209909       100    
reco:anglereconuepfps:NuAngularReco                   0.000103041   0.000923895    0.0181638    0.000265542   0.00211517       100    
reco:anglereconumupfps:NuAngularReco                  8.0321e-05    0.000839532    0.0181952    0.000200242   0.00211113       100    
[art]:TriggerResults:TriggerResultInserter             7.45e-06     9.97233e-06    4.435e-05    8.9855e-06    4.41188e-06      100    
end_path:out1:RootOutput                               2.91e-06     6.59563e-06    2.894e-05     6.765e-06    3.35195e-06      100    
end_path:out1:RootOutput(write)                        0.0102987      0.16015       0.61235      0.167832      0.108521        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22406.4 MB
  Peak resident set size usage (VmHWM): 2843.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2_reco_2024-11-23T_044544Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2_reco_data_2024-11-23T_044544Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 13:51:23 UTC       justIN version: 01.01.09