Commit 58f0c26f authored by Femi Kadri's avatar Femi Kadri

README.TXT

parent 7a020bf2
......@@ -33,12 +33,13 @@ and method = LBM or LST
dos2unix *
dos2unix lsf*
chmod 755 lsf* *cmd
module load intel/2016a #this line is only need when running the job on schooner.Not needed if running on kong.
mpicc -c timer.c
./make_cmd sooner
tcsh lsf_chain.tcsh sooner 1 #FOR EXAMPLE IF YOU JUST WANT TO RUN A SINGLE JOB FOR TESTING YOUR CODE
./make_cmd schooner
tcsh lsf_chain.tcsh schooner 1 #FOR EXAMPLE IF YOU JUST WANT TO RUN A SINGLE JOB FOR TESTING YOUR CODE
tcsh lsf_chain.tcsh sooner 100 #FOR RUNNING
tcsh lsf_chain.tcsh schooner 100 #FOR RUNNING
OR
......@@ -127,4 +128,4 @@ March 25,2010
I should write here more often... now we have tricubic and trilinear with subtrilinear at the wall lst interpolation. also the square and the semi-pipe ducts. matlab postprocessing has been improved tremendously. i am sure i am forgetting other things.
=================================================================
NEEDS TO BE FIXED: RESUMING WITH FIRST ITERATION LAUNCHES INITIAL BSUB, write out a complete velocity array (not just nonwalls)
\ No newline at end of file
NEEDS TO BE FIXED: RESUMING WITH FIRST ITERATION LAUNCHES INITIAL BSUB, write out a complete velocity array (not just nonwalls)
INSTRUCTIONS TO RUN THE CODE:
NOTE: your unix shell must be tcsh, change your shell by typing chsh -s /bin/tcsh
1)UPLOAD ALL THE FILES IN THE CURRENT DIRECTORY TO THE SUPERCOMPUTER (you must use an SFTP client, like FileZilla or WinSCP), TYPE THE FOLLOWING INTO THE TERMINAL
#make clean #THIS IS OPTIONAL, IN CASE YOU WANT TO RECOMPILE COMPLETELY
mkdir debug
mkdir backup
mkdir results
mkdir results_LST
mkdir results_LBM
mkdir results_LBM/stresses
2)SETUP THE input.txt WITH USER INPUT;
NOTE: if using a user generated geometry, upload global_walls.txt to /results folders and make sure that all the auto-geometry options are set to .false.
3)SETUP THE *.bsub BATCH SCRIPTS FOR THE SUPERCOMPUTER (the script requests the amount of processes and the duration of the job from the machine)
NOTE: the code requires three LBM batch scripts and three LST batch scrips (six total), the naming convention is
machinename_initial_method.bsub
machinename_resume_method.bsub
machinename_finalize_method.bsub
where machinename = sooner, lonestar, etc
and method = LBM or LST
(for example sooner_finalize_LBM.bsub)
4)COPY AND PASTE THE FOLLOWING LINES (notice dos2unix command) INTO THE TERMINAL SCREEN
dos2unix *
dos2unix lsf*
chmod 755 lsf* *cmd
module load intel/2016a #this line is only need when running the job on schooner.Not needed if running on kong.
mpicc -c timer.c
./make_cmd schooner
tcsh lsf_chain.tcsh sooner 1 #FOR EXAMPLE IF YOU JUST WANT TO RUN A SINGLE JOB FOR TESTING YOUR CODE
tcsh lsf_chain.tcsh sooner 100 #FOR RUNNING
OR
./make_cmd lonestar
tcsh lsf_chain.tcsh lonestar 1 #FOR FOR EXAMPLE IF YOU JUST WANT TO RUN A SINGLE JOB FOR TESTING YOUR CODE
tcsh lsf_chain.tcsh lonestar 50 #FOR RUNNING
(this sets the machine machine name and the maximum number of chain jobs allowed... should be at least 5 for a non-debug job)
NOTE ABOUT COMPILING: if you want to compile the code with "bounds checking" turned on (this is where the compiler will actually check if any of the arrays were referenced out of bounds (because by default it will NOT do this, for computational efficiency), you must edit the make_cmd file such that the -C option is included in the compilation flags
make sure that you understand the different ways to compile. Here is a description of some useful compiler flags for the Intel compiler:
-O3 this one is optimization, there are different levels of optimization. 3 is an "aggressive" level. you are not guaranteed that you will get the same results when you use optimization (the compiler could mess up the flow of calculations), so you have to check that you are getting the same results with and without optimization!
-C this one checks for references outside of the bounds of the array. if you don't use this flag the compiler will *NOT* check for this and your code will either crash without telling you why or, the worse scenario, is that it will work as if nothing wrong happened, but your results will be crap and you will not know why. however, including the extra checks slows down your performance, so you should only compile without this flag when you are absolutely sure that your code is working 100% correctly!
-fpe0 this one checks for for overflow/underflow. again, the compiler will simply ignore if you have something blowing up to +/- infinity, if you don't use this flag. however, using this flag slows down performance.
-traceback this flag also slows down performance, but provides you with extra information regarding where the problem in your code occured that you otherwise wouldn't get
these flags tell the compiler what kind of hardware you are running on, so it can compile your code more efficiently. these are machine specific!
-xT -shared-intel on Lonestar
-march=core2 -mtune=core2 -shared-intel on Sooner
other compilers have analogous flags, but you have to look up the syntax.
NOTE ABOUT DEBUG OUTPUT: if you want to write DEBUG output (this is output that might help you debug the code that has been put in throughout the code by the code author), you must edit the file makefile and change the value of -DALL_DEBUG=false to -DALL_DEBUG=true (notice that NO periods are used here). this will give you a lot more debug output, but will slow down the code tremendously and might also generate huge output files if you run large or long simulations.
=========================================================
HERE IS HOW TO CHECK YOUR DISK USAGE:
The du command ("disk usage") tells you how much consumption is
in a particular directory tree. For example:
du -sk ~
This tells you how many KB of files are in your home directory.
If you want a subdirectory-by-subdirectory breakdown, remove the s.
The df command ("disk free") tells you how much disk space is
available in a particular partition as a whole (which may contain
many users). For example:
df -h ~
This tells you the total size of the partition containing your
home directory (and perhaps many others), how much is already
used, and how much is still free.
=================================================================
PERSONAL NOTES:
April 20th, 2009:
HERE WE AVOID COPYING FROM ONE 4D ARRAY INTO ANOTHER DURING THE STREAM_BC SUBROUTINE BY SWAPPING POINTERS TO THE ARRAYS. PLEASE TAKE NOTE OF THE EXTRA LINE IN THE input.txt file:
.false. /spread_LST_markers_in_a_plane /initial positions are placed randomly throughout entrance YZ plane
(THIS LINE IS FOR THE LST PART OF THE CODE)
remember that we had to comment out all references to file unit 30 from generic.F90 in order to get the code to work on bigben
=================================================================
May 30th, 2009:
Added LST_fluid_surf_rxn_count.F90
!THIS SUBROUTINE WILL COUNT HOW MANY LST MARKERS REACTED AT THE FLUID SURFACE NODES
!AND IT WILL KEEP TRACK OF THE REACTION PROBABILITIES THAT OCCURED
!IT IS ASSUMED THAT A REACTION OCCURED WITHIN THIS UNIT CUBE, IF THIS SUBROUTINE IS CALLED
!IT IS ALSO ASSUMED THAT EVERY VERTEX IN THE UNIT CUBE SURROUNDING THE REACTED MARKER
!BELONGS TO THE REACTING FLUID SURFACE (AS LONG AS ITS NOT A SOLID VERTEX)
=================================================================
August 28, 2009
For stability, I changed the user input from courant number to relaxation time tau (1 is recommended value). the courant number is calculated from the following formula, in order to satisfy the tau:
courant = ( 2 * tau [lattice units] -1) * dx / (6 * kinematic viscosity [physical units])
=================================================================
March 25,2010
I should write here more often... now we have tricubic and trilinear with subtrilinear at the wall lst interpolation. also the square and the semi-pipe ducts. matlab postprocessing has been improved tremendously. i am sure i am forgetting other things.
=================================================================
NEEDS TO BE FIXED: RESUMING WITH FIRST ITERATION LAUNCHES INITIAL BSUB, write out a complete velocity array (not just nonwalls)
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:52 2016
MODULE ADD_A_SURFACE_LAYER_TO_WALLS__genmod
INTERFACE
SUBROUTINE ADD_A_SURFACE_LAYER_TO_WALLS(MY_MPI_PROCESS_ID, &
&LOCAL_WALLS,LOCAL_X_MIN_REAL,LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL, &
&LOCAL_Y_MAX_REAL,LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL, &
&GLOBAL_X_MIN_REAL,GLOBAL_X_MAX_REAL,GLOBAL_Y_MIN_REAL, &
&GLOBAL_Y_MAX_REAL,GLOBAL_Z_MIN_REAL,GLOBAL_Z_MAX_REAL, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST, &
&LOCAL_X_NODES_GHOST,LOCAL_Y_NODES_GHOST,LOCAL_Z_NODES_GHOST, &
&LOCAL_X_NODES_REAL,LOCAL_Y_NODES_REAL,LOCAL_Z_NODES_REAL,DIVIDE_X,&
&DIVIDE_Y,DIVIDE_Z,IN,OUT,WEST,EAST,UP,DOWN,CARTESIAN_COMMUNICATOR,&
&LOCAL_WALLS_PLUS_FLUID_SURFACE)
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_REAL
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_X
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Y
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Z
INTEGER(KIND=4), INTENT(IN) :: IN
INTEGER(KIND=4), INTENT(IN) :: OUT
INTEGER(KIND=4), INTENT(IN) :: WEST
INTEGER(KIND=4), INTENT(IN) :: EAST
INTEGER(KIND=4), INTENT(IN) :: UP
INTEGER(KIND=4), INTENT(IN) :: DOWN
INTEGER(KIND=4), INTENT(IN) :: CARTESIAN_COMMUNICATOR
LOGICAL(KIND=4), INTENT(INOUT) :: &
&LOCAL_WALLS_PLUS_FLUID_SURFACE(LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST&
&,LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
END SUBROUTINE ADD_A_SURFACE_LAYER_TO_WALLS
END INTERFACE
END MODULE ADD_A_SURFACE_LAYER_TO_WALLS__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:53 2016
MODULE APPROX_2ND_DERIVATIVE_OF3D_FIELD__genmod
INTERFACE
SUBROUTINE APPROX_2ND_DERIVATIVE_OF3D_FIELD( &
&GLOBAL_X_MIN_REAL,GLOBAL_X_MAX_REAL,GLOBAL_Y_MIN_REAL, &
&GLOBAL_Y_MAX_REAL,GLOBAL_Z_MIN_REAL,GLOBAL_Z_MAX_REAL, &
&LOCAL_X_MIN_REAL,LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL, &
&LOCAL_Y_MAX_REAL,LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL,DIVIDE_X, &
&DIVIDE_Y,DIVIDE_Z,LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST, &
&LOCAL_Y_MIN_GHOST,LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST, &
&LOCAL_Z_MAX_GHOST,LST_LOCAL_WALLS,MY_MPI_PROCESS_ID, &
&MY_MPI_REAL_DATA_TYPE,EAST,WEST,UP,DOWN,IN,OUT, &
&CARTESIAN_COMMUNICATOR,LOCAL_X_NODES_GHOST,LOCAL_Y_NODES_GHOST, &
&LOCAL_Z_NODES_GHOST,LOCAL_X_NODES_REAL,LOCAL_Y_NODES_REAL, &
&LOCAL_Z_NODES_REAL,NUMBER_OF_PROCESSES,LATTICE_DIMENSION,ROOT, &
&LOCAL_FIELD_TO_BE_DIFFERENTIATED,LOCAL_2ND_DERIVATIVE_WRT_XX, &
&LOCAL_2ND_DERIVATIVE_WRT_YY,LOCAL_2ND_DERIVATIVE_WRT_ZZ)
INTEGER(KIND=4), INTENT(IN) :: LATTICE_DIMENSION
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_X
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Y
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Z
LOGICAL(KIND=4), INTENT(IN) :: LST_LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_REAL_DATA_TYPE
INTEGER(KIND=4), INTENT(IN) :: EAST
INTEGER(KIND=4), INTENT(IN) :: WEST
INTEGER(KIND=4), INTENT(IN) :: UP
INTEGER(KIND=4), INTENT(IN) :: DOWN
INTEGER(KIND=4), INTENT(IN) :: IN
INTEGER(KIND=4), INTENT(IN) :: OUT
INTEGER(KIND=4), INTENT(IN) :: CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: NUMBER_OF_PROCESSES
INTEGER(KIND=4), INTENT(IN) :: ROOT
REAL(KIND=8), INTENT(IN) :: &
&LOCAL_FIELD_TO_BE_DIFFERENTIATED(LATTICE_DIMENSION, &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(OUT) :: LOCAL_2ND_DERIVATIVE_WRT_XX( &
&LATTICE_DIMENSION,LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST, &
&LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(OUT) :: LOCAL_2ND_DERIVATIVE_WRT_YY( &
&LATTICE_DIMENSION,LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST, &
&LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(OUT) :: LOCAL_2ND_DERIVATIVE_WRT_ZZ( &
&LATTICE_DIMENSION,LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST, &
&LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
END SUBROUTINE APPROX_2ND_DERIVATIVE_OF3D_FIELD
END INTERFACE
END MODULE APPROX_2ND_DERIVATIVE_OF3D_FIELD__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:55 2016
MODULE DETERMINE_LOCAL_FLUID_SURFACE__genmod
INTERFACE
SUBROUTINE DETERMINE_LOCAL_FLUID_SURFACE(LOCAL_X_MIN_REAL, &
&LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL,LOCAL_Y_MAX_REAL, &
&LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL,MY_MPI_PROCESS_ID, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST,Q,IN,OUT, &
&WEST,EAST,UP,DOWN,CARTESIAN_COMMUNICATOR,LOCAL_X_NODES_GHOST, &
&LOCAL_Y_NODES_GHOST,LOCAL_Z_NODES_GHOST,LOCAL_X_NODES_REAL, &
&LOCAL_Y_NODES_REAL,LOCAL_Z_NODES_REAL,DIVIDE_X,DIVIDE_Y,DIVIDE_Z, &
&GLOBAL_X_MIN_REAL,GLOBAL_X_MAX_REAL,GLOBAL_Y_MIN_REAL, &
&GLOBAL_Y_MAX_REAL,GLOBAL_Z_MIN_REAL,GLOBAL_Z_MAX_REAL,LOCAL_WALLS,&
&ROOT,NUMBER_OF_PROCESSES,LATTICE_DIMENSION, &
&LOCAL_PRODUCT_XYZ_NODES_REAL,ALL_PROCESSES_LOCAL_BOUNDS)
INTEGER(KIND=4), INTENT(IN) :: NUMBER_OF_PROCESSES
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: Q
INTEGER(KIND=4), INTENT(IN) :: IN
INTEGER(KIND=4), INTENT(IN) :: OUT
INTEGER(KIND=4), INTENT(IN) :: WEST
INTEGER(KIND=4), INTENT(IN) :: EAST
INTEGER(KIND=4), INTENT(IN) :: UP
INTEGER(KIND=4), INTENT(IN) :: DOWN
INTEGER(KIND=4), INTENT(IN) :: CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_REAL
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_X
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Y
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Z
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MAX_REAL
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
INTEGER(KIND=4), INTENT(IN) :: ROOT
INTEGER(KIND=4), INTENT(IN) :: LATTICE_DIMENSION
INTEGER(KIND=4), INTENT(IN) :: &
&LOCAL_PRODUCT_XYZ_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: ALL_PROCESSES_LOCAL_BOUNDS(&
&6*NUMBER_OF_PROCESSES)
END SUBROUTINE DETERMINE_LOCAL_FLUID_SURFACE
END INTERFACE
END MODULE DETERMINE_LOCAL_FLUID_SURFACE__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:55 2016
MODULE DETERMINE_LOCAL_SOLID_SURF_VMSK__genmod
INTERFACE
SUBROUTINE DETERMINE_LOCAL_SOLID_SURF_VMSK(LOCAL_X_MIN_REAL,&
&LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL,LOCAL_Y_MAX_REAL, &
&LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL,MY_MPI_PROCESS_ID, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST,Q,IN,OUT, &
&WEST,EAST,UP,DOWN,CARTESIAN_COMMUNICATOR,LOCAL_X_NODES_GHOST, &
&LOCAL_Y_NODES_GHOST,LOCAL_Z_NODES_GHOST,LOCAL_X_NODES_REAL, &
&LOCAL_Y_NODES_REAL,LOCAL_Z_NODES_REAL,DIVIDE_X,DIVIDE_Y,DIVIDE_Z, &
&GLOBAL_X_MIN_REAL,GLOBAL_X_MAX_REAL,GLOBAL_Y_MIN_REAL, &
&GLOBAL_Y_MAX_REAL,GLOBAL_Z_MIN_REAL,GLOBAL_Z_MAX_REAL,LOCAL_WALLS,&
&ROOT,NUMBER_OF_PROCESSES,LATTICE_DIMENSION, &
&LOCAL_PRODUCT_XYZ_NODES_REAL,ALL_PROCESSES_LOCAL_BOUNDS, &
&GLOBAL_NUM_OF_INT_FACES_VOL_MASK,GLOBAL_SOLID_SURF_COUNT_VOL_MASK,&
&GLOBAL_SURFACE_AREA_VOL_MASK,GLOBAL_CURV_EXTRA_VOL_VOL_MASK, &
&LST_LOCAL_VOL_OF_INT_USER_MASK)
INTEGER(KIND=4), INTENT(IN) :: NUMBER_OF_PROCESSES
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: Q
INTEGER(KIND=4), INTENT(IN) :: IN
INTEGER(KIND=4), INTENT(IN) :: OUT
INTEGER(KIND=4), INTENT(IN) :: WEST
INTEGER(KIND=4), INTENT(IN) :: EAST
INTEGER(KIND=4), INTENT(IN) :: UP
INTEGER(KIND=4), INTENT(IN) :: DOWN
INTEGER(KIND=4), INTENT(IN) :: CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_REAL
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_X
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Y
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Z
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MAX_REAL
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
INTEGER(KIND=4), INTENT(IN) :: ROOT
INTEGER(KIND=4), INTENT(IN) :: LATTICE_DIMENSION
INTEGER(KIND=4), INTENT(IN) :: &
&LOCAL_PRODUCT_XYZ_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: ALL_PROCESSES_LOCAL_BOUNDS(&
&6*NUMBER_OF_PROCESSES)
INTEGER(KIND=4), INTENT(OUT) :: &
&GLOBAL_NUM_OF_INT_FACES_VOL_MASK
INTEGER(KIND=4), INTENT(OUT) :: &
&GLOBAL_SOLID_SURF_COUNT_VOL_MASK
REAL(KIND=8), INTENT(OUT) :: GLOBAL_SURFACE_AREA_VOL_MASK
REAL(KIND=8), INTENT(OUT) :: &
&GLOBAL_CURV_EXTRA_VOL_VOL_MASK
LOGICAL(KIND=4), INTENT(IN) :: &
&LST_LOCAL_VOL_OF_INT_USER_MASK(LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST&
&,LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
END SUBROUTINE DETERMINE_LOCAL_SOLID_SURF_VMSK
END INTERFACE
END MODULE DETERMINE_LOCAL_SOLID_SURF_VMSK__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:57 2016
MODULE DETERMINE_LOCAL_SOLID_SURFACE__genmod
INTERFACE
SUBROUTINE DETERMINE_LOCAL_SOLID_SURFACE(LOCAL_X_MIN_REAL, &
&LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL,LOCAL_Y_MAX_REAL, &
&LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL,MY_MPI_PROCESS_ID, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST,Q,IN,OUT, &
&WEST,EAST,UP,DOWN,CARTESIAN_COMMUNICATOR,LOCAL_X_NODES_GHOST, &
&LOCAL_Y_NODES_GHOST,LOCAL_Z_NODES_GHOST,LOCAL_X_NODES_REAL, &
&LOCAL_Y_NODES_REAL,LOCAL_Z_NODES_REAL,DIVIDE_X,DIVIDE_Y,DIVIDE_Z, &
&GLOBAL_X_MIN_REAL,GLOBAL_X_MAX_REAL,GLOBAL_Y_MIN_REAL, &
&GLOBAL_Y_MAX_REAL,GLOBAL_Z_MIN_REAL,GLOBAL_Z_MAX_REAL,LOCAL_WALLS,&
&ROOT,NUMBER_OF_PROCESSES,LATTICE_DIMENSION, &
&LOCAL_PRODUCT_XYZ_NODES_REAL,ALL_PROCESSES_LOCAL_BOUNDS, &
&GLOBAL_NUMBR_OF_INTERFACE_FACES,GLOBAL_SOLID_SURFACE_COUNT_REAL, &
&GLOBAL_SURFACE_AREA,GLOBAL_CURVATURE_EXTRA_VOLUME)
INTEGER(KIND=4), INTENT(IN) :: NUMBER_OF_PROCESSES
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: Q
INTEGER(KIND=4), INTENT(IN) :: IN
INTEGER(KIND=4), INTENT(IN) :: OUT
INTEGER(KIND=4), INTENT(IN) :: WEST
INTEGER(KIND=4), INTENT(IN) :: EAST
INTEGER(KIND=4), INTENT(IN) :: UP
INTEGER(KIND=4), INTENT(IN) :: DOWN
INTEGER(KIND=4), INTENT(IN) :: CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_NODES_REAL
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_X
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Y
LOGICAL(KIND=4), INTENT(IN) :: DIVIDE_Z
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: GLOBAL_Z_MAX_REAL
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
INTEGER(KIND=4), INTENT(IN) :: ROOT
INTEGER(KIND=4), INTENT(IN) :: LATTICE_DIMENSION
INTEGER(KIND=4), INTENT(IN) :: &
&LOCAL_PRODUCT_XYZ_NODES_REAL
INTEGER(KIND=4), INTENT(IN) :: ALL_PROCESSES_LOCAL_BOUNDS(&
&6*NUMBER_OF_PROCESSES)
INTEGER(KIND=4), INTENT(OUT) :: &
&GLOBAL_NUMBR_OF_INTERFACE_FACES
INTEGER(KIND=4), INTENT(OUT) :: &
&GLOBAL_SOLID_SURFACE_COUNT_REAL
REAL(KIND=8), INTENT(OUT) :: GLOBAL_SURFACE_AREA
REAL(KIND=8), INTENT(OUT) :: GLOBAL_CURVATURE_EXTRA_VOLUME
END SUBROUTINE DETERMINE_LOCAL_SOLID_SURFACE
END INTERFACE
END MODULE DETERMINE_LOCAL_SOLID_SURFACE__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:58 2016
MODULE FIND_GLOBAL_MIN_AND_MAX__genmod
INTERFACE
SUBROUTINE FIND_GLOBAL_MIN_AND_MAX(MY_MPI_PROCESS_ID, &
&LOCAL_X_MIN_REAL,LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL, &
&LOCAL_Y_MAX_REAL,LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST, &
&ORIGINAL_CARTESIAN_COMMUNICATOR,MY_MPI_REAL_DATA_TYPE, &
&LOCAL_LOGICAL_MASK,LOCAL_3D_ARRAY,GLOBAL_3D_ARRAY_MIN, &
&GLOBAL_3D_ARRAY_MAX)
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: &
&ORIGINAL_CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_REAL_DATA_TYPE
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_LOGICAL_MASK( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(IN) :: LOCAL_3D_ARRAY( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_MIN
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_MAX
END SUBROUTINE FIND_GLOBAL_MIN_AND_MAX
END INTERFACE
END MODULE FIND_GLOBAL_MIN_AND_MAX__genmod
!COMPILER-GENERATED INTERFACE MODULE: Thu Oct 27 21:01:58 2016
MODULE FIND_GLOBAL_MIN_AND_MAX_MASKED__genmod
INTERFACE
SUBROUTINE FIND_GLOBAL_MIN_AND_MAX_MASKED(MY_MPI_PROCESS_ID,&
&LOCAL_X_MIN_REAL,LOCAL_X_MAX_REAL,LOCAL_Y_MIN_REAL, &
&LOCAL_Y_MAX_REAL,LOCAL_Z_MIN_REAL,LOCAL_Z_MAX_REAL, &
&LOCAL_X_MIN_GHOST,LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST, &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST,LOCAL_Z_MAX_GHOST, &
&ORIGINAL_CARTESIAN_COMMUNICATOR,MY_MPI_REAL_DATA_TYPE,LOCAL_WALLS,&
&LST_LOCAL_VOL_OF_INT_USER_MASK,LOCAL_3D_ARRAY, &
&GLOBAL_3D_ARRAY_IN_MASK_MIN,GLOBAL_3D_ARRAY_IN_MASK_MAX, &
&GLOBAL_3D_ARRAY_OUT_MASK_MIN,GLOBAL_3D_ARRAY_OUT_MASK_MAX)
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_GHOST
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_GHOST
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_PROCESS_ID
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_X_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Y_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MIN_REAL
INTEGER(KIND=4), INTENT(IN) :: LOCAL_Z_MAX_REAL
INTEGER(KIND=4), INTENT(IN) :: &
&ORIGINAL_CARTESIAN_COMMUNICATOR
INTEGER(KIND=4), INTENT(IN) :: MY_MPI_REAL_DATA_TYPE
LOGICAL(KIND=4), INTENT(IN) :: LOCAL_WALLS( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
LOGICAL(KIND=4), INTENT(IN) :: &
&LST_LOCAL_VOL_OF_INT_USER_MASK(LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST&
&,LOCAL_Y_MIN_GHOST:LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST: &
&LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(IN) :: LOCAL_3D_ARRAY( &
&LOCAL_X_MIN_GHOST:LOCAL_X_MAX_GHOST,LOCAL_Y_MIN_GHOST: &
&LOCAL_Y_MAX_GHOST,LOCAL_Z_MIN_GHOST:LOCAL_Z_MAX_GHOST)
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_IN_MASK_MIN
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_IN_MASK_MAX
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_OUT_MASK_MIN
REAL(KIND=8), INTENT(OUT) :: GLOBAL_3D_ARRAY_OUT_MASK_MAX