Parallel Operating Environment Overview

1. What is the Parallel Operating Environment?

2. Understanding the Parallel Operating Environment
   1. Partition
   2. Partition Manager
   3. Resource Manager

3. Executing Parallel Programs Using the POE
   1. Set your Path
   2. Compiling and Linking a Parallel Program
      1. Fortran (mpxlf)
      2. C (mpcc)
   3. Setting up the Execution Environment
      1. Environment Variables
      2. Some Preset POE Environments
      3. Creating a .rhosts File
      4. Creating a host.list File
   4. Invoking the Executable

4. Recommendations for Running on the SP2

5. Miscellaneous Environment Variables

6. Acknowledgements and References

What is the Parallel Operating Environment?

• IBM's environment for developing and running distributed memory, parallel Fortran, C or C++ programs. Runs on the IBM RS/6000 platform using AIX.

• The POE consists of components for developing, executing, debugging, profiling, and tuning parallel programs.

  • Parallel Compiler Scripts

  • PE Environment Variables

  • Program Marker Array

  • System Status Array

  • Parallel Debugging Facility

  • Parallel Profiling Capability

Understanding the Parallel Operating Environment

Partition:

The group of processor nodes on which you run your program is called your partition.

Partition Manager:

The Partition Manager establishes and controls your partition. It consists of a set of subroutines that are linked into your program and an Internet daemon process called pmd. The Partition Manager is responsible for:

• Requesting the nodes for your parallel job
• Acquiring the nodes needed for your parallel job (if the Resource Manager is not used)
• Copying the executables from the initiating node to each node in your partition
• Loading the executable on each node in your partition
• Setting up stdout and stdin

Resource Manager:

The Resource Manager keeps track of the nodes that are currently processing a parallel task. When requested by the Partition Manager, the Resource Manager will allocate nodes for your use. It attempts to enforce a "one parallel task per node" rule.

Processor Pool

The local systems administrator will divide the processor nodes into disjoint pools of processors for management purposes. You may request that your parallel tasks run on specific pools. The processor pools at the MHPCC are listed here. You may determine the available pools by typing:

     jm_status -P 

User Space Protocol

A fast method of communication between nodes. Can be used only with the high performance switch.

Internet Protocol

A slower method of communication between nodes. Can be used with ethernet or the high performance switch.

Communication SubSystem (CSS)

The Communication SubSystem (CSS) is the set of library routines which implement either of the above communication protocols.

Executing Parallel Programs Using POE

1. Compile and link the program using shell scripts which call the C or FORTRAN compilers while linking in the Partition Manager and MPL subroutines.

2. Set up your execution environment. This includes setting the number of tasks and specifying the method of node allocation.

3. Optional: start either of the POE X-Windows analysis tools:
   1. Program Marker Array (Optional: click here).
   2. System Status Array (Optional: click here).

4. Load and execute the parallel program on the processor nodes of your partition.
   1. Load the same executable on all nodes; normal procedure for SPMD programs.
   2. Individual load separate executables onto the nodes; normal procedure for MPMD programs.

Set Your Path

     set path = ($path /usr/lpp/poe/bin)

Compiling and Linking a Parallel Program

• mpxlf and mpcc invoke the corresponding compilers (xlf, and cc) for compilation and link in the Partition Manager initialization routines. See the mpxlf and mpcc man pages for more detailed information.

• Fortran:

  
       mpxlf  options  {-us,-ip} filename
    

• C:

  
       mpcc options {-us,-ip} filename
    

• -ip causes the IP CSS to be statically bound with the executable. Communication during execution will use the Internet Protocol.

• -us causes the US CSS library to be statically bound with the executable. This CSS library uses the US protocol for dedicated use of the high-performance switch. It allows you to drive the switch adaptor directly from your parallel tasks.

• If neither flag is set, then a CSS library will be dynamically linked with the executable at run time. The library which will be linked is determined by the MP_EUILIB environment variable.

Other mpxlf and mpcc Compiler Options

• any of the options normally accepted by the xlf or c compiler

• -v causes a "verbose" output listing of the mpxlf or mpcc shell script

• -g compiles the program suitable for debugging with pdbx or xpdbx. This option is also necessary to use the Source Code view in the Visualization Tool .

• -c compiles only (does not link the object files)

• -o exename names the executable to exename

• -l (note: this is lower case L) names additional libraries to be searched. Several libraries are automatically included. See the mpxlf and mpcc man pages for more detailed information.

• -L<pathname> Places <pathname> into the library search path. Directories will be searched in the order of their occurrance on the command line.

• -p permits profiling with prof or gprof

• -I (note: this is upper case i) names directories for additional include files. /usr/lpp/poe/include is automatically included.

• -O optimize the output code.

Environment Variables

• MP_PROCS The number of nodes to allocate for your partition

• MP_RESD specifies whether or not the Partition Manager should connect to the POWERparallel system Resource Manager to allocate nodes. Can be either:
  1. yes
  2. no

• MP_HOSTFILE specifies the name of a host file for node allocation.
  • Can be any file name, NULL or ""
  • The default host list file is host.list in the current directory. You do not need to set MP_HOSTFILE if your host list file is the default host.list.
  • You must specify a host list file if you:
    • need specific node allocation
    • request non-specific node allocation from a number of system pools.
    • use a host list file named something other than the default host.list.

• MP_EUILIB Specifies which CSS library implementation to use for communication. Can be either:
  1. ip The Internet Protocol CSS.
  2. us The User Space CSS; lets you drive the high-performance switch directly from your parallel tasks without going through the kernel or operating system; use this library on the MHPCC SP2.

• MP_EUIDEVICE The adaptor set to use for message passing; either Ethernet, FDDI, token-ring, or the high-performance switch adaptor (note: this variable is ignored if the US CSS library is used).
  1. en0 Ethernet
  2. fi0 FDDI
  3. tr0 token-ring
  4. css0 high-performance switch; use this on the MHPCC SP2.

• MP_RMPOOL The number of the POWERparallel system pool that should be used by the Resource Manager for non-specific node allocation. This is only valid if you are using the POWERparallel Resource Manager for non-specific node allocation (from a single pool) without a host list file. You may obtain information about available pools by typing "jm_status -P" at the Unix prompt.

• SP_NAME Specifies the name of the Control Workstation. This variable is used by the System Status Array tool.

Some Preset POE Environments

The following four sets of POE environment variables should cover most users' needs and are provided to help simplify getting started:

• User Space (US) protocol and non-specific node allocation from a single pool.

• User Space (US) protocol and specific node allocation or non-specific node-allocation from more than one pool

• Internet protocol (IP) and non-specific node allocation from a single pool

• Internet protocol (IP) and specific node allocation or non-specific node-allocation from more than one pool

The .rhosts File

• The Unix .rhosts file is required by the POE. The format of the .rhosts file is:

  
       hostname1  userid
       hostname2  userid
       hostname3  userid
       hostname4  userid
         ...
  

• You must create the .rhosts file in your home directory

• It is recommended that you include all SP node names in your .rhosts file.

The Host List File (specific node allocation)

• The host list file specifies the actual (specific) nodes on which you wish to run your job.

• You need to create a host list file if:
  • You want specific node allocation (you choose the nodes - not the Resource Manager).
  • You want non-specific node allocation from more than one pool (optional, click here for details).

• You do not need to create a host list file if you want non-specific node allocation from a single pool.

• How to use the host list file
  • Create a file called host.list in your current directory. This is the default name/location.
  • If you want to use a filename other than host.list , then you will need to set the MP_HOSTFILE environment variable to that file's name
  • If you wish to use ip communications, set MP_EUILIB to ip. MP_RESD may be either yes or no.
  • If you wish to use us communications, set MP_EUILIB to us. MP_RESD must be yes.
  • Execute your program. The Partition Manager will then attempt to run your program on the nodes you specify (task 0 will be run on the first listed host, task 1 on the second listed host, etc.).

• The number of nodes used depends upon the MP_PROCS variable.

• If any of the listed hosts are unavailable, the Partition Manager returns an error and the program is not executed - even if there are more than the number of nodes needed in your host list file.

• If a host is listed more than once in the host list file, then multiple tasks will be allocated to that particular host.

• Additionally, you can specify whether or not multiple tasks (from different users) can share the same CPU or share the same network adaptor (optional click here for details).

• File Format:

  
       hostname1
       hostname2 
       hostname3
       hostname4
    

• Example (actual names will depend upon your system):

  
       ! Selected node names
       fr3n05.mhpcc.edu
       fr3n01.mhpcc.edu
       fr3n03.mhpcc.edu
       fr3n07.mhpcc.edu
    

• Upon successfully running a parallel program, you can generate an output host list file containing the names of the nodes which were used by setting the environment variable MP_SAVEHOSTFILE to the name of the file in which the node names will be stored.

Invoking the Executable

• Start any X-Windows analysis tools (ie. pmarray) before the executable is invoked.

• For SPMD programs, simply issue the name of the executable, specifying any command line flags that may be required. Command line flags may be used to temporarily override any MP environment variables that have been set.

• For MPMD programs, execution begins once you have loaded the partition using poe.
  • setenv MP_PGMMODEL mpmd
  • enter poe at the unix prompt.
  • load each program on the processor nodes

    
         % poe
         0:hostname> master [options]
         1:hostname> slave [options]
         2:hostname> slave [options]
         3:hostname> slave [options]
         4:hostname> slave [options]
        

  • Execution begins after the last node has been loaded. Alternatively, the program names may be listed in a file and poe will load the nodes based on that file. Set the environment variable MP_CMDFILE to that file name.

Recommendations for Running on the SP2

• Use dynamic linking of the communication libraries instead of static linking.

• During program development mode, use IP communication libraries over the switch.

• During production mode, use US communication over the switch.

• If you are using User Space communications and enough nodes are not available, use IP communications instead.

• Write scratch files to local scratch space whenever possible. At the MHPCC, each node has a /localscratch directory - it is faster than writing scratch files to your home directory.

• Use non-specific node allocation (currently just one pool).

• Inform the Resource Manager about your job.

• Set MP_PROCS to the number of nodes you wish to use.

• Start the executable from an SP2 node

• During program development it may be helpful to set MP_INFOLEVEL to a higher value (5) and to run with MP_EUILIB set to ip .

Miscellaneous Environment Variables

• MP_RETRY The period of time between processor node allocation retries if there are not enough processor nodes immediately available.

  
       C Shell:    setenv MP_RETRY 10
       Korn Shell: export MP_RETRY=10
    

• MP_RETRYCOUNT The number of times that the Partition Manager should attempt to allocate processor nodes before returning without running your program.

  
       C Shell:     setenv MP_RETRYCOUNT 15
       Korn Shell:  export MP_RETRYCOUNT=15
    

• MP_SAVEHOSTFILE The name of an output host list file to be generated by the Partition Manager.

  
       C Shell:     setenv MP_SAVEHOSTFILE progname.hosts.used
       Korn Shell:  export MP_SAVEHOSTFILE=progname.hosts.used
    

• MP_EUIDEVELOP Causes MPL to do more detailed checking during program execution.

  
       C Shell:     setenv MP_EUIDEVELOP yes
       Korn Shell:  export MP_EUIDEVELOP=yes
    

• MP_STDOUTMODE Enables you to manage the STDOUT from your parallel tasks.

  
       C Shell:     setenv MP_STDOUTMODE ordered
       Korn Shell:  export MP_STDOUTMODE=ordered
                 -or-
       C Shell:     setenv MP_STDOUTMODE unordered
       Korn Shell:  export MP_STDOUTMODE=unordered
                 -or-
       C Shell:     setenv MP_STDOUTMODE 6
       Korn Shell:  export MP_STDOUTMODE=6
    

• MP_INFOLEVEL The amount of diagnostic information displayed as your program runs; set to an integer between 0 and 5 (lower value implies less diagnostic information).

  
       C Shell:     setenv MP_INFOLEVEL 2
       Korn Shell:  export MP_INFOLEVEL=2
    

Acknowledgements and References

Last revised: 2/13/95 Blaise Barney