問題描述
來自一個文件的多個 mpirun 與多個文件運行 (Multiple mpiruns from one file vs multiple file runs)
I am working on an older cluster that I am not the administrator of and is in locked configuration which is causing me some issues. The system uses the original mpich and the cluster script is written in Perl using parallel::mpi for the runs. This Monte Carlo script generates 5000 test cases to be run that are then launched on the cluster. I looked at the original code and it was taking around 500 (not 5000) of the test and putting them in three files. The files then passed them to the cluster around 260 max at a time. I asked the system administrator if he knew why the programmer had done this and he said it was because mpich(1) would not allow more than 260 jobs being sent at a time. I am not sure if that was an mpich1 thing or a parallel::mpi thing.
So I rewrote the Perl program to generate 19 files with around 250 cases for each of the files to get all 5000 cases to run. My question is I usually have one file which I run and launch with a pbs_mpirun command. The original program had three separate launch pbs files. So now I have 19. Can I launch them all from the same file? Do I have to put some type of sleep between the mpirun commands? The way the cluster queues are set up only one user can be running one job on the same queue at a time. So if I launched to launch multiple runs to queue n64 only one would run at a time which is fine but I don't want to have to submit 19 runs and fill up the qstat list to complete one monte carlo if I don't have to.
This may be something common but I just have never dealt with so any advice would be appreciated. Below is my PBS file which launches the first Perl cluster files. The Perl cluster files are mpi_wokernode_1.pl ‑ mpi_workernode_19.pl.
#!/bin/sh
###Lines starting with "# " are comments, batch system configuration
###commands starting with "#PBS"
#PBS ‑l walltime= 12:00:00
#PBS ‑N MONTE
### Declare job non‑rerunable
#PBS ‑r n
### Output files (overwritten in successive jobs)
#PBS ‑e system1:/filearea
#PBS ‑o system1:/filearea
### Return error and output on output stream
#PBS ‑j oe
### Queue name (small, medium, long, verylong)
#PBS ‑q n64@old_cluster
#PBS ‑l select=64:ncpus=1
#PBS ‑l place=free
##PBS ‑m e
#PBS ‑W group_list=groupa
cd /filearea
# Count all available processors
NPROCS=`grep ‑v "\#" $PBS_NODEFILE | wc ‑l`
pbs_mpirun mpi_workernode_1.pl
pbs_mpirun mpi_workernode_2.pl
參考解法
方法 1:
This sounds like an issue that's pretty specific to your system, so it might be hard to get useful advice here. However, if you have a home directory on the machine, you can usually install your own MPI in there and launch that. You'll just add ‑‑prefix=$HOME/<path to install> to your ./configure line and you should be ready to go. You'll probably need to modify your PBS script so it uses your MPI instead of the default one. That's probably just combining the last two lines to look like:
/path/to/mpiexe ‑n <num_procs> /path/to/mpi_program
This assumes a couple of this.
- You have some sort of NFS sharing set up for your home directory. Without this, you'll have to copy the MPI executables to all of the nodes in your system, which is a pain.
- You have access to the original MPI program and can execute it directly without your wrapper script. This will make the entire process easier if you do.
- Your system isn't doing some nastiness that prevents you from running your own MPI. I have used systems in the past that made it difficult / impossible to replace the default MPI library with your own. Your system probably isn't like that, but you'll have to experiment to find out.
(by Carole、Wesley Bland)