How to submit GPAW jobs on mills

Example below shows how to submit GPAW parallel or serial job on mills in order to obtain the electronic band structure of Ni.

Input files

• Save two files below as *.py files.

band.py

from gpaw import GPAW, FermiDirac
from ase import Atoms
from ase.io import read, write
from gpaw import GPAW, PoissonSolver, MixerSum
from ase.structure import bulk

# -------------------------------------------------------------
# Bulk configuration
# -------------------------------------------------------------

a = 3.5249 
atoms = bulk('Ni', 'fcc', a=a)
atoms.center()

write('system.traj', atoms)

for a in atoms:
    if a.symbol == 'Ni':
        a.magmom = 0.6

# Make self-consistent calculation and save results
calc = GPAW(h=0.18,
            mode='lcao',
            xc='PBE',
            basis='dzp',
            kpts=(8,8,8),
            occupations=FermiDirac(width=0.05, maxiter=2000),
            mixer=MixerSum(beta=0.010, nmaxold=8, weight=100.0),
            poissonsolver=PoissonSolver(eps=1e-12),
            txt='band_sc.txt')

atoms.set_calculator(calc)
atoms.get_potential_energy()
calc.write('band_sc.gpw')


# Calculate band structure along Gamma-X 

from ase.dft.kpoints import ibz_points, get_bandpath
points = ibz_points['fcc']
G = points['Gamma']
X = points['X']
kpts, x, X = get_bandpath([G, X], atoms.cell, 80)

calc = GPAW('band_sc.gpw',
            mode='lcao',
            xc='PBE',
            basis='dzp',
            kpts=kpts,
            txt='band_harris.txt',
            fixdensity=True,
            parallel={'domain': 1},
            usesymm=None,
            convergence={'bands': 'all'})

if calc.input_parameters['mode'] == 'lcao':
    calc.scf.reset()

calc.get_potential_energy()
calc.write('band_harris.gpw')

find_band_as_dat.py

from ase.lattice import bulk
from gpaw import GPAW

calc = GPAW('band_harris', txt=None)
import numpy as np
#   eps_skn = np.zeros((2,80))
eps_skn = np.array([[calc.get_eigenvalues(k,s)
                     for k in range(80)]
                    for s in range(2)]) - 10.75103

#eigs -= GPAW('band_sc.gpw', txt=None).get_fermi_level()

for n in range(10):
    for k in range(80):
        print k, eps_skn[0, k, n], eps_skn[1, k, n]
    print

Job submission script

• Save the following in a file jobscript

 #$ -N band
 #$ -o out.log
 #$ -e band.err
 #$ -pe openmpi 16
 #$ -l h_rt=3:00:00
 #
 source /opt/shared/valet/docs/valet.sh
 vpkg_rollback all
 vpkg_require numpy/1.6.1-2.7
 vpkg_require mkl/10.3.8-64bit
 vpkg_require openmpi/1.6.1-intel64
 #
 export PYTHONPATH=/lustre/scratch/ksaha/gpaw/ase:$PYTHONPATH
 export PATH=/lustre/scratch/ksaha/gpaw/ase/tools:$PATH
 export GPAW_SETUP_PATH=/lustre/scratch/ksaha/gpaw/gpaw-setups-0.8.7929
 export PYTHONPATH=/lustre/scratch/ksaha/gpaw/gpaw:${PYTHONPATH}
 export PYTHONPATH=/lustre/scratch/ksaha/gpaw/gpaw/build/lib.linux-x86_64-2.7:${PYTHONPATH}
 export PATH=/lustre/scratch/ksaha/gpaw/gpaw/build/bin.linux-x86_64-2.7:/lustre/scratch/ksaha/gpaw/gpaw/tools:${PATH}
 #
 #
 cd /home/1076/FCC-Ni_bandstructure/Wiki
 #
 mpirun /lustre/scratch/ksaha/gpaw/gpaw/build/bin.linux-x86_64-2.7/gpaw-python band.py
 #mpirun /lustre/scratch/ksaha/gpaw/gpaw/build/bin.linux-x86_64-2.7/gpaw-python find_band_as_dat.py

Running the job script

• To run band.py execute:

qsub jobscript

• To run find_band_as_dat.py execute:

1. modify the 4th line of the jobscript "#$ -pe openmpi 16" to "#$ -pe openmpi 1" -- this means we will be running in a single core.

2. comment the second last line and uncomment the very last line of the jobscript

3. then submit job: qsub jobscript

4. bands should be generated into "out.log" file

• One can also submit jobs using 'qrsh' following the steps:

1. execute

 qrsh

2. load the following libraries executing:

   
 source /opt/shared/valet/docs/valet.sh
 vpkg_rollback all
 vpkg_require numpy/1.6.1-2.7
 vpkg_require mkl/10.3.8-64bit
 vpkg_require openmpi/1.6.1-intel64

3. run jobs:

mpirun -np #cores /lustre/scratch/ksaha/gpaw/gpaw/build/bin.linux-x86_64-2.7/gpaw-python inputfile > output.log &

Test results

• Once you successfully finish running both jobs, you should be able to reproduce this figure:

gnuplot> plot "result.dat" using 1:2 with lines title "Spin up", "result.dat" using 1:3 with lines title "Spin down"

FIG. 1: Electronic band structure of fcc-Ni.