SESA6061 Coursework 2 (Hand in via e-assignments by Wed 4th Dec 23:00)

1 Wave equation (Week 7 Computer Laboratory Session)

Problem specification
You should first work through the 1D wave equation tutorial ‘OpenSBLI on Virtual machines’, starting
on page 6 of this handout. In the computer rooms the virtual machine file is located in
’C:\Apps\OVAs\RITM0222382\OpenSBLI2019.ova’ or
’S:\VirtualBox OVAs\RITM0222382’ (a network drive)
and you should start from item 2 in the tutorial. In the virtual box you should check that the number
of cores is set to 4 and the display scale is 100%.
In this exercise you will compare the solution of the 1D wave equation for different orders of central
differencing, and demonstrate the order of convergence. The script is set up to advance the wave
equation until a time of t = 1, with 200 grid points in the x0 direction. The analytical solution is given
for a time t as φ(x0 , t) = sin(2π(x0 + c0 t)), where the wave-speed is c0 = 0.5. All simulations in this
section should be advanced until a time of t = 1.
At the bottom of the wave.py file the simulation parameters are selected. For the 1D-wave equation
these refer to:
c0: The wave-speed in the governing equation
dt: The time-step
niter: The number of iterations to perform
block0np0: The number of grid points to use in the x direction
Delta0Block0: The spacing between each grid point

Exercises
1. Change the wave.py and plot.py files to calculate and print out the L1 and L∞ error norms for
4th order central differencing and Nx0 = 100 grid points. The L1 and L∞ error norms are defined
for exact and reconstructed solutions φ (xi , tn ) and φni in 1D as

N
1 X
L1 = |φ (xi , tn ) − φni | , (1)
N
i=1
L∞ = max1≤i≤N |φ (xi , tn ) − φni | , (2)

One way of doing this is to add a print command:

print sum(phi_error)/nx,max(phi_error)
2. Modify the wave.py problem file to use (a) a second order and (b) a sixth order numerical scheme,
and re-run the simulation. Compare the L1 and L∞ error to the previous 4th order result.
3. Complete the table below for an 4th order scheme with different grid resolutions (remember to
change the value of N in plot.py each time). For error  and grid spacing h = N1x , the order of
0
convergence of a scheme N is given by  = hN . Produce a log-log plot of the error as a function
of the grid spacing h, the gradient of the line is equal to the order of convergence N . Discuss
any deviations from the expected behaviour.

1
 Grid points L1 error L∞ error
25
50
100
200
400

Table 1: Convergence analysis for the 1D-wave equation.

2 3D Taylor-Green vortex turbulent simulation (Week 8)

Problem specification
In this section you will perform a compressible Taylor-Green vortex simulation on a 323 uniform grid.
Example scripts are given to read in the data and loop over each of the files.
In OpenSBLI the standard Cartesian base (x, y, z) is denoted (x0 , x1 , x2 ), with velocity components
(u, v, w) given by (u0 , u1 , u2 ). The computational domain is a cube of size 0 ≤ x0 ≤ 2π, 0 ≤ x1 ≤ 2π,
and 0 ≤ x2 ≤ 2π, with periodic boundary conditions applied in all directions. The initial condition at
t = 0 is given by

u0 (x0 , x1 , x2 , t = 0) = sin (x0 ) cos (x1 ) cos (x2 ) , (3)

u1 (x0 , x1 , x2 , t = 0) = − cos (x0 ) sin (x1 ) cos (x2 ) , (4)

u2 (x0 , x1 , x2 , t = 0) = 0, (5)

1 1
p(x0 , x1 , x2 , t = 0) = 2
+ (cos (2x0 ) + cos (2x1 )) (2 + cos (2x2 )) . (6)
γM∞ 16
In this normalisation the initial density is given by ρ = M∞ 2 γp, with constant initial temperature

T0 = 1. Reynolds, Prandtl and Mach numbers are set to Re = 800, Pr = 0.71, and M = 0.1
respectively; the ratio of specific heat capacities is taken to be γ = 1.4. The simulation is to be run
until a non-dimensional time of t = 20, with a time-step of dt = 0.005.

1. Change to the taylor green vortex directory.

cd ../taylor_green_vortex

2. Open the taylor green vortex.py problem script and change the simulation parameters at the end
of the file to be consistent with the problem description above. The constants are:

Re: Reynolds number

gama: Ratio of heat capacities
Minf: Freestream Mach number
Pr: Prandtl number
dt: The time-step
niter: The number of iterations to perform
block0np0: Number of grid points in the x0 direction
block0np1: Number of grid points in the x1 direction

2
 block0np2: Number of grid points in the x2 direction
Delta0block0: Grid spacing in x0
Delta1block0: Grid spacing in x1
Delta2block0: Grid spacing in x2

3. Set the desired frequency for writing output files by modifying the save every input. The input is
based on the iteration number, by default the simulation is writing output every 400 iterations.
For a total simulation of 4000 iterations, 10 files would be written out.

h5 = iohdf5(save_every=400, **kwargs)

4. Save the script and run the problem for a N = 323 grid (same steps as for the wave equation).

5. Create an output directory and move the temporary output files into it. Check the number of
‘.h5’ files in the new directory is consistent with the selected file writing frequency.

mkdir output_data/32
mv opensbli_output_* ./output_data/32

6. Run the plot.py program that will be distributed. This file loops through all the output files
and computes the kinetic energy integrated over the domain according to
Z
1 1
Ek = ρuj uj dΩ, (7)
ρref Ω Ω 2

where the repeated index j implies the summation convention and ρref is the mean density over
the domain. Check that you are getting sensible results - the values should range from 0.12286
at t = 4 to 0.02010 at t = 20.

Exercises
1. For homogeneous turbulence in the incompressible limit, the viscous dissipation rate  is equal
to the enstrophy (the square of the vorticity) integrated over the domain, such that
Z
1
ε= ωi ωi dΩ, (8)
ReΩ Ω
Extend the plot.py program to calculate all 9 possible velocity derivatives. Hence calculate the
vorticity components and the time-evolution of the average dissipation rate. Plot the kinetic
energy and dissipation rate as functions of time.

2. For this triply-periodic problem, write down the kinetic energy transport equation. Hence esti-
mate the peak dissipation rate and compare this to your value on the 323 grid.

3. Repeat for a 483 simulation (note down how long this takes to run compared to 323 ).

3
3 Post-processing flow field datasets (Week 9)
This part doesn’t require the Virtual Box software. Flowfields for Taylor-Green simulations at 643 and
1283 will be distributed by dropoff/safesend. You can then work in either Matlab or Python (your
choice) to post-process the data.

Matlab:
N=64
file={’opensbli_output_000400.h5’};
dum=h5read(char(file),’/opensbliblock00/rhou0_B0’);
u=dum(4:N+7,4:N+7,4:N+7);
dum=h5read(char(file),’/opensbliblock00/rhou1_B0’);
v=dum(4:N+7,4:N+7,4:N+7);
dum=h5read(char(file),’/opensbliblock00/rhou2_B0’);
w=dum(4:N+7,4:N+7,4:N+7);

Note that u, v and w are now 3D arrays e.g. u(i, j, k) (i is the index in the x direction, j in y, k in z)
including two halo elements on all sides.

Python:
You will first need to install HDF5 reading modules using ’pip install h5py’
or ’conda install h5py’

Example of commands to read the data:

import numpy as np
import h5py
def read_dataset(file, dataset):
group = file["opensbliblock00"]
d_m = group["%s" % (dataset)].attrs[’d_m’]
size = group["%s" % (dataset)].shape
start = [abs(d+2) for d in d_m]
end = [s-abs(d+2) for d, s in zip(d_m, size)]
read_data = group["%s" % (dataset)].value[start[0]:end[0],
start[1]:end[1], start[2]:end[2]]
return read_data
# Read an HDF5 file using the h5py library
fname = ’opensbli_output_000800.h5’
f = h5py.File(fname, ’r’)
# Extract one of the datasets from the file ’f’: [rho_B0, rhou0_B0,
rhou1_B0, rhou2_B0, rhoE_B0]
rhou = read_dataset(f, ’rhou0_B0’)
# The data is of size (N+4, N+4, N+4) as there are two halos on each side
print(rhou.shape)

Note that python stores arrays with a different ordering compared to Matlab e.g. u(k, j, i) (where k is
the index in the z direction, j in y, i in x i.e. row versus column ordering).

Exercises
1. Read in a dataset and compute the integrated kinetic energy. Check that this is consistent with
the results from last week.

4
 2. Using fourth order central differencing compute the z vorticity component
∂v ∂u
ωz = − (9)
∂x ∂y

Find the maximum vorticity at z = π/2 for a 643 simulation at t=10.0

3. Make a contour plot of the u velocity and z vorticity at t = 10 in the x, y plane at z = π/2 for
323 , 643 and 1283 simulations and comment on the changes as you refine the grid.

4. Optional (bonus marks): try plotting Q slices, or 3D surfaces of vorticity or Q, or (more advanced)
making a movie of the flow development

5. Submit a concise document with your results and answers (in sentences) to the various questions
from each week’s exercises. Include a listing of your post-processing program that calculates the
vorticity. Marks distribution will be roughly 30% for the wave equation work, 40% for the Taylor
Green KE and  results and 30% for flow visualisation.

N.D.Sandham & D.Lusher 9/11/2019

5
 OpenSBLI on Virtual Machines
David J. Lusher: d.lusher@soton.ac.uk

November 9, 2019

These instructions are for running OpenSBLI on a pre-configured Linux virtual machine using the
VirtualBox software. To install the software on a different machine, please follow the getting started
guide included in the version 2 repository. The virtual machine is a fully functioning Linux operating
system, the Firefox browser can be used to browse the internet assuming your base machine has an
active internet connection.

1 Setting up the virtual machine and code

1. Install the VirtualBox software from:

https://www.virtualbox.org/wiki/Downloads

2. Import the OpenSBLI.ova virtual machine file

File->Import Appliance->Select the file->Continue->Import

Leave all of the settings as they are

3. The machine will now be visible in the main VirtualBox window. Assign the amount of resources
to give to the virtual machine. By default it is assigned to use 4 CPU cores and 4GB of memory.

Right click the machine->Settings->System->Motherboard

Move the slider to adjust the memory assigned to the VM
Change to the Processor tab, change the number of CPUs to use
Click OK to exit

4. Double click the machine to power it on. Click away the messages about mouse integration.

5. Choose the default ’user’ account, and enter the password.

Username: user
Password: opensbli (all lowercase)

6. Open the software folder on the Desktop and navigate to the OpenSBLI applications folder.

software->opensbli->apps->wave

7. Open the wave.py problem script with a text editor

Right click wave.py->Open with text editor

6
 This is the problem file to solve a one-dimensional wave equation. The equation is specified
at the top, followed by periodic boundary conditions, an initial condition and selection of the
numerical scheme to use. By default this is using 4th order central differencing and 3rd order
explicit Runge-Kutta time-advancement.

8. Open a terminal and call the Python script to generate a simulation code.

Right click inside the wave folder->Open in Terminal

In the new window type: python wave.py and press return
Check the directory now contains additional files: ls

You should now have four new files:

opensbli.cpp, opensbliblock00_kernels.h, defdec_data_set.h, bc_exchanges.h and some
LaTeX files. These contain C code to perform the simulation, you can inspect each file using
a text editor. To be able to run the code we must translate it using the OPS library, which
generates parallel versions to run on multiple processors.

9. Translate the code using OPS to generate parallel versions:

python $OPS_TRANSLATOR opensbli.cpp

You should now have folders for CUDA, MPI and so on, plus an opensbli_ops.cpp file. You
need to apply the translation step each time you make changes to the opensbli.cpp code generated
by OpenSBLI.

10. Compile an MPI version of the code:

make opensbli_mpi

11. Run the executable for a selected number of processes ’np’:

mpirun -np 4 ./opensbli_mpi

The simulation will now run, when finished the output data is written to an opensbli_output.h5
HDF5 file for post-processing. If you wish to run the simulation again remove the HDF5 file from
the directory before repeating the above steps. Steps 9-11 can be performed in one command by
typing: bash run.sh

12. Plot the output with the provided plot script

python plot.py
cd simulation_plots/
evince phi.pdf

Each time a simulation is run, the above steps must be performed. The opensbli_output.h5
file should be moved or deleted prior to running another simulation. If the output file already
exists, any additional simulations will not overwrite it with new output. To save the output for
later use:

mv opensbli_output.h5 your_file_name.h5

7
 To clean up the directory before additional runs, use make clean

13. To save your current work, power off the machine

Power options on the top right of the screen->Power logo->Power off

Once the machine has turned off, your current state will be saved. When you turn the machine on
again the state will resume as you left it. If you wish to move the machine to another computer
that has VirtualBox, select File->Export Appliance in the main VirtualBox window.

Basic Linux usage

The virtual machine is set-up with a copy of Ubuntu 18.04 LTS and the libraries needed to run
OpenSBLI. Common Python libraries used for post-processing are also installed, including NumPy,
H5py, and Matplotlib. As the coursework requires basic proficiency in using a Linux environment and
the command-line, below is a list of useful commands that you will need.

Useful commands
ctrl+shift+t : Open a new terminal (On Ubuntu)
ls : List the contents of the current directory
pwd : Print the current working directory
cd : Change into a directory (For example, cd my_directory/)
cd .. : Go back one directory
cd ~/ : Change to your /home/user/ directory
mv : Move a file (mv file1 file2)
rm : Remove a file (rm my_file.txt)
rm -rf : Remove a directory (rm -rf my_directory/)
mkdir : Create a directory (mkdir my_directory)
cp : Copy a file (cp file1 file2)
cp -r : Copy a directory (cp -r directory1 directory2)
nano : Command-line text editor to edit a file (ctrl+X to exit)
gedit : GUI-based text editor (others can be installed)
man : Show the help manual for a command (man ls, man cd,..)
cat : Print the contents of a file to screen (cat my_file.txt)
tail : Show the end of a file (tail my_file.txt)
./ : Execute a file (./run.sh)

Tab-completion and wildcards

Tab-completion is a useful time saving feature to avoid having to type full file names. Type the first
few letters of the file or directory you are trying to access, press tab and if it exists in the directory the
full name will be completed for you. Wildcards (*) can be used to apply commands on multiple files
at once. For example mv *.py ./my_directory would move all Python files in the current directory
into my_directory.
If you require additional Python libraries, they can be installed using the pip package manager
with sudo pip install <package-name>.