1. Environment Variables

An environment variable is a dynamic-named value containing information about your login session, set configuration options, and customize the shell environment in Linux-based operating systems. In simple terms, an environment variable is used to define a location for storing a value, you can also refer to its symbolic name. These values are stored within the system and are used by command-line applications. The stored value can be deleted, displayed, edited, and re-saved.

2.1 Use printenv to print all environment variables.

2.2 You can set a temporary environment variable by:

MY_NAME=TAIYI

Then call this variable as $MY_NAME, for example:

echo $MY_NAME

Notice that the above environment variables are available only for the current session. If you open a new shell, or if you log out, all variables will be lost.

Please verify this.

2.3 Setting a Persistent Environment Variable

One way to set a persistent environment variable is to use the user-based configuration file .bash_profile in your home directory. With Bash, you can declare the variables in the following format:

export MY_NAME=TAIYI

Please add the above line to your .bash_profile, then save .bash_profile.

To update current environment variables, run:

source .bash_profile

Now, we have made $MY_NAME a persistent environment variable, and it will be loaded every time you log in.

2. Python MPI job submission

This demo provides an example of submitting a simple Message Passing Interface (MPI) job using Python, and in particular, the mpi4py module. MPI is a standard protocol that defines how messages are passed between processes, including one-to-one and broadcast communications. Here we will use the mpi4py module that allows point-to-point (sends, receives) and collective (broadcasts, scatters, gathers) communications of any picklable Python object, as well as optimized communications of Python object exposing the single-segment buffer interface (NumPy arrays, builtin bytes/string/array objects). Check MPI for Python for more.

2.1 Installing mpi4py on your home folder

By far the easiest way to install mpi4py is using Anaconda, which will also install dependencies (including MPI). To do so, let’s first load Anaconda on Taiyi, type the following line in your terminal:

module load python/anaconda3/5.2.0

Now you should be able to use the conda command. We then create a new environment (e.g., my_mpi) and install mpi4py and other modules under my_mpi:

conda create --name my_mpi python=3.8 mpi4py numpy scipy

Please refer to Python environments for questions about the conda environment and how to create one.

On Taiyi, the above command will take a few minutes to complete. Once finished, you will see (using ll -a) a hidden folder .conda where modules got installed. To activate the environment:

source activate my_mpi

Check mpi4py is properly installed by running:

python -c "import mpi4py"

If the above command gives no error, mpi4py is successfully installed on your home folder.

2.2 The Python script

Copy the demo files in /work/ese-ouycc/mpi_demo/ to your home folder:

cp -r /work/ese-ouycc/mpi_demo ~/mpi_demo

You will find a demo file mpi_demo.py inside, take a look at the file.

2.3 The submission script

You will also find a job submission file submit_taiyi.sh inside, take a look at the file.

2.4 Submit the job

To submit the MPI job to Taiyi, go to folder mpi_demo, type:

bsub < submit_taiyi.sh

Using bqueues to check the availability of nodes, and bjobs to check the status of the submitted job(s).

2.5 Check the results

Once your job is completed, take a look at sys.log and job.log. What do you get? Make sure the MPI jobs run successfully.