MK Dynamics

Programming - Parallel Fortran Code


Description

Introduction

The Open MPI project is an open source implementation of the Message Passing Interface. The interface is a suite of tools that allow for interprocess communications which facilitate executing portions of a program in parallel. Detailed discussions on the the Message Passing Interface (MPI) can be found here:
http://condor.cc.ku.edu/~grobe/docs/intro-MPI.shtml

Details on OpenMPI can be found here:
https://www.open-mpi.org

Installing OpenMPI as a plugin to the Eclips IDE + Photran + Parallel Tools Platform Framework

We start by installing OpenMPI, which can be downloaded here:
https://www.open-mpi.org/software/ompi/v1.8

At the time of this writing, version 2.0.0 is NOT supported by our IDE, called Eclipse NEON. Rather, version 1.8.8 needs to be downloaded and installed. Our targer system is an Intel quad-core I7 processor, running Fedora 23 Linux (64 bit version). This detail will become important as we delve deeper into the actual mechanisms of execution. This type of processor should be general enough for most cases.

Once the tarball has been decompressed, it is important to adjust the LD_LIBRARY_PATH environment variable to point to the proper location, otherwise, there will be much grief during linking and running the generated executables. This variable must be properly set prior to installing OpenMPI because directories are actually hardcoded into installation at build time. In my system, I found the following to work properly:

Then from the install directory that contains the decompressed installation files, it is required to do a ./configure --with-devel-headers. The --with-devel-headers switch will install additional files required by the Eclipse IDE to properly work with the OpenMPI modules.

The configuration script will perform a series of systems tests and setup the make files for a system specific build. Generally, the configuration takes a few minutes and completes with no issues. Once the configuration is complete, we are now ready to perfom the install, which will build the system specific executables and libraries.
This can be accomplished by entering: sudo make -j 4 install all

The sudo is required because the libraries and executables need to be installed as root. The -j 4 switch causes make to use all four cores to person the build, greatly speeding up the process. It takes a about 5 minutes on this Intel I7 3GHz processor.

Once the installation has completed, it's status can be viewed by typing ompi_info at the command prompt.

A full listing of the output of ompi_info is shown below:

Notice that the version of Open MPI is 1.8.8. At the time of this writing, this version is known to work with the Eclipse IDE + Photran + Parallel Tools Platform. In the author's understanding, version 2.0.0 is known to NOT work.

It was found that if, after compilation and execution of the source code, that error messages are displayed pertaining to MPI or MCA, a working remedy is to ensure that the LD_LIBRARY_PATH environment variable is properly set, and then to reconfigure and rebuild the OpenMPI installation files as described above. If problems persist, another thing to try is to rename the folder /usr/local/lib/openmpi to /usr/local/lib/openmpi_backup, and then perform a ./configure --with-devel-headers and then a sudo make -j 4 install all. This will ensure that fresh files are installed into /usr/local/lib/openmpi.

More information on installation issues can be found here:
https://eclipse.org/ptp/docs/releases/release-1.1.php
https://www.open-mpi.org/faq/?category=building#easy-build

Installing the Eclipse IDE + Photran + Parallel Tools Platform

At the time of this writing, the latest version of the Eclipse Integrated Development Environment (IDE) is NEON.
It can be downloaded here:
https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/neon/R/eclipse-parallel-neon-R-linux-gtk-x86_64.tar.gz
Other tools are also available for the Eclipse Platform and may be obtained by following the links on their homepage.

I recommend decompressing the downloaded Eclipse installation files into a temporary directory, such as /home/Installs/eclipse-installer

To install, simply run the file eclipse_inst by entering ./eclipse_inst at the command prompt. Eclipse requires Java to run; therefore, if there are any installation issues, it is recommended that you ensure that the latest version of Java is installed on your machine.

After Eclipse is installed, click "Launch" to start it up. It "may" be necessary to install some additional add-on tools to access the parallel development tools. I can't be certain since I have tried the installation a number of times before I was able to get the system to operate properly. This is more a reflection of my learning curve and my particular system setup, and not a criticism, per se, of the Eclipse developers.

Once the Eclipse IDE is up and running, go to >help>Install New Software, which will bring up the following screen:

Ensure that all "Linux Tools", "Parallel Tools Platform", and "Performance, Profiling and Tracing Tools" are selected, and click "Next" to update the Eclipse IDE.

Setting Up The Eclipse IDE For Compiling A Simple Hello World Type Program With Parallel Execution

Now that the OpenMPI binaries are installed, a new project can be started within the Eclipse IDE. Since I am using GNU+Linux, and wish to use GNU as much as possible, I will use the GNU Fortran compiler, versus the Intel Fortran Compiler. This is a personal preference. The discussions that follow will assume use of the GNU Fortran compiler.

To start a new project, go to >File>New Project> and drill down to "Others" in "Project type:". Select Executable (Gnu Fortran on Linux/*nix). The "Toolchains:" should have GCC Fortran highlighted.

It is a good idea to have "use default location" and "show project types and toolchains only if they are supported on the platform" checkmarked, to minimize having to go in to the configurations and be forced to mess around with paths and directory settings. Give the project a meaningful name and click "Next".

Under "Configurations:", ensure that Debug and Release are checked. This is more of a check to verify that the toolchain is working properly. It is also useful to have two versions of a project once it becomes sufficiently complex. If all is well click "Advanced Settings".

The next step gave me allot of grief until I figured it out, and I am happy to be able to save you some trouble... Since we are going to be using OpenMPI, it states on the OpenMPI website that it is HIGHLY recommended to use their specialized compiler and execution binaries. We must tell the Eclipse IDE to use "mpifort" rather than "gfortran" to compile and link. "mpifort" has special wrappers embedded that handle accessing all of the required includes and libraries. Without this, setting up all of paths and environment variables can be a daunting task. "mpifort" makes it a snap to compile OpenMPI based programs.

To set this up, under "Advanced Settings", go to "Fortran Build" and select the drop-down arrow. Select "Settings" and then GNU Fortran Compiler. In the box that says "gfortran", replace it with "mpifort".

Once "mpifort" is used as the compiler, you don't have to worry about setting up the includes or libraries. The next step is not critically necessary but I like to have verbose messages when compiling, so I went ahead and set the "Debug Level" to "Maximum" and checked "Generate gprof information (-pg)" and "Generate prof information (-p)" under "Debugging". Under "Miscellaneous" I checkmarked "Verbose (-v)".

Once the compiler is setup, the linker is next. Basically, the exact same things are done for the linker. Click "GNU Fortran Linker" and change the "command" from gfortran to "mpifort".

As before, it is not critically necessary but you can go to the "Debugging" item and checkmark the production of additional debugging information. Once the command is changed to "mpifort", click "OK" and then "Finish". You should now be in the Eclipse IDE Photran screen. This is perspective that allows for actually entering, building and running the code. I have been messing around with Photran for a while, so details on your screen will be slightly different.

Right click on the name of your newly created project in the "Project Explorer" and select "New" and then "Fortran Source Code".

Give the new source file a meaningful name, MAKING SURE TO GIVE THE FILENAME A .f08 EXTENSION. The extension must be a .f08, with a non-capital "f". Keep "Template" set to "Default free-form Fortran source file template". Verify and click "Finish". The Eclipse IDE differentiates between filenames that end in .F08 and .f08. A filename ending in .F08 will not build properly.

I don't recommend changing the source folder because it may become difficult to build the project due to PATH issues. Click "Finish" when done. A new file will appear in the "Project Explorer" with the name given. Double click this file and the editing screen will appear. A default tempplate program will appear, but we will not be using it. Go ahead and delete all of the lines in the editor. As a first program, I found the one named "Hello World" at:
http://condor.cc.ku.edu/~grobe/docs/intro-MPI.shtml
to be useful. Copy and paste it into the editor.

We are now ready to compile, link and execute the "Hello World" type program to verify that all is setup correctly. In a subsequent section, we will using the debug functions of the Eclipse IDE to step through the disassembly of the executable to gain valuable insights into the physical implementation of parallel exectution code.

Compiling, Linking and Executing A Simple Hello World Type Program With Parallel Execution

Assuming everything was setup as described in the previous sections, to compile the program should be as simple as clicking on >Project>Build Project. There will be some activity with a progress bar, and allot of verbose debugging information will appear in the console as shown below:

I am an information junkie, and looking at the verbose debugging output on the console is very pleasing. Of important note is that underneath the line which says "Invoking: GNU Fortran Compiler", there is a command line that starts with "mpifort", and under "Invoking: GNU Fortran Linker", there is another line which starts with "mpifort". As explained in the previous sections, "mpifort" is a special wrapper executable which enables greatly facilitated compilation and linking of Fortran programs using MPI, without having to worry about PATHs or setting up libraries.

As can be seen at the bottom of the console output, the program compiled and linked properly, with no error messages. We are now ready to run the program. First we will run the program using a single core and single thread to make sure everything works so far. Then we will step up to multiple cores/threads. To run a program, a "Run Configuration" needs to be created first.

To create a run configuration, go to >Run>Run Configurations. Select C/C++ Application (even though we are writing in Fortran). Click the "New Launch Configuration" button and give the run configuration an appropriate name. The rest of the fields should have been already filled in by Eclipse. The window is shown below:

Once everything is set, click "Run", and you should see the following output:

The status message in the console states that the program terminated with exit value 0, which means no errors. The output of the program is shown directly below the status line. Next, we will setup a debug configuration where we can disassemble the executable and step through the assembly instructions. Doing this will give us valuable insight into how MPI programs actually work. This will become particularly important for multithreaded/multi-core execution programs.

Setting Up A Debug Configuration in the Eclipse IDE + Photran + Parallel Tools Platform

To set up a debug configuration, click on >Run>Debug Configurations. Since a run configuration was already created, the debug configuration should be all set up and ready to go. Click "Debug" to get started.

Once "Debug" is clicked, the Eclipse IDE will start the debug/disassembly process and the following perspective screen will appear:

Debugging can now commence. It is very instructive to activate instruction stepping mode and step through each assembly language instruction because you can see the function calls being made to set up the Message Passing Interface. In the next section, we will show a video of stepping through the assembly language instructions and talk about what is happening at each step.

Debugging an OpenMPI Fortran Executable in Eclipse IDE + Photran + Parallel Tools Platform

The following video shows the process of debugging a simple "Hello World" type Open MPI Fortran program.

In the video, a number of observations can be made. At 0:55, we can see that many modules are loaded upon executing the assembly instruction that directs the instruction pointer to the location of the main function. Many of these modules are related to setting up the MPI. It can also be observed that at one point, exectution is actually taking place over two cores in two separate threads, even though this was not explicitly stated in the run/debug command. This gives an insight into the inner workings of how OpenMPI actually works.

It appears that the additional thread on the second core is running housekeeping routines pertaining to the Message Passing Interface. The thread starts and stops at the same time that the MPI initialization instructions are being executed. A number of MPI routines are also executed upon finalization of the program.

In the next sections, we will setup and perform a debug/run of this program using parallel exectution.

Setting Up and Running an OpenMPI Fortran Executable on Multiple Cores/Threads in Eclipse IDE + Photran + Parallel Tools Platform

In this section, we will setup the simple "Hello World" type program for running using multiple cores/threads.

Go to >Run>Run Configurations, and create a new run configuration under "Parallel Application". Just to make it interesting, increase the number of processes to 4. Click "Apply" and then "Run".

The program should execute with no issues and "Hello World" should appear in the console 4 times.

Setting Up a Parallel Debug of an OpenMPI Fortran Executable on Multiple Cores/Threads in Eclipse IDE + Photran + Parallel Tools Platform

In this section, we will setup the simple "Hello World" type program for debugging using multiple cores/threads.

Go to >Run>Debug Configurations, and create a new debug configuration under "Parallel Application". Just to make it interesting, increase the number of processes to 4. Click "Apply" and then "Debug".

After the progress bar finishes indicating that debugging is being set up, the following screen should come up:

There can be a number of step-through scenarios at this point. My personal favorite is to activate all images and step into the main() code. This will call the OpenMPI functions and display the details of how OpenMPI actually works to setup and shutdown itself.

A quick look at the process list by typing "ps -FH --User mark" reveals the process structure of what actually happens during debugging. It appears that four instances of gdb are started, under the auspices of the sdm debugger.

Interestingly enough, disassembly during a parallel debug does not occur.

Running a Parallel Debug of an OpenMPI Fortran Executable on Multiple Cores/Threads in Eclipse IDE + Photran + Parallel Tools Platform

In this section, we will perform a parallel debug of the simple "Hello World" type program using multiple cores/threads.

The most informative source of in this mode of the Parallel Debugger is the console. It displays all of the steps necessary to setup the program for debugging, as well as the inter-image messaged that are passed during setup. The console output for this program is shown below:

In the debugging view, there will be a set of diamond symbols, signifying an individual image of the running program. These can be activated by double-clicking on the diamond. A single click designates that particular image as the lead process. After activating an image, corresponding message information will appear in the console view.

For this program, to get an informative step-through, click step into the main process and then step over the OMPI particular processes. On occasion, it may be necessary to click group suspend since one image may be waiting for another, and the interprocess message may never be sent due to being held back by the debugger. The end result is that the debugging may appear to hang, but actually it is waiting for an interprocess message. Clicking on group suspend fixes this issue.

Also, it can become very complicated to debug a single image out of the group, because this will result in that image becoming unsynchronized with the group. Usually, this will require a terminate and restart of the debugging process. The goal is to step into the main process, step over the OMPI related processes, and step over the function call to writing "Hello World". If done correctly, this will produce an output of "Hello World" repeated four times.

The structure of the images and loaded libraries can be seen in this snapshot of the debug view shown below.

A video of performing a step-through of a parallel execution program is shown below: