client.tex

\subsection{Introduction}
The computation is client is used to compute the jobs of experiments. Usually there have to be a lot of jobs computed to evaluate experiments
and since they are independent from each other, this task can be parallelized across many CPU cores. The computation client can be started on arbitrarily many
machines and will manage the available CPUs and start jobs from the available experiments. It connects to the central database and downloads all required resources
such as instances and solver binaries and writes back the results to the database.

\subsection{System requirements}
\label{clientSR}
\index{System requirements client}
The client is written in C/C++ and should be able to run on most Linux distributions where a MySQL C connector library is available.

Because the central storage location for all required computational ressources, experiment metadata and results is a MySQL database, the client has to be able to establish a connection to the machine that hosts the database. 
\marginlabel{TCP/IP Connections:}
This means that the machines where the client runs on have to be able to establish a TCP/IP connection
to the database machine. 

The client was mainly tested on the bwGRID\footnote{{http://www.bw-grid.de/}}, a distributed computer cluster that consists of several hundred
nodes at several physical locations at universities of Baden-Württemberg, Germany. Even though the machine hardware is homogenous, the network topology of bwGRID is not.
\marginlabel{Connection alternatives:}
In cases where direct network access from the computation nodes back to the database server is not possible it is usually possible to tunnel a connection
over the cluster's login node back to the database via SSH. For example running
\marginlabel{\Eexample}\begin{verbatim}
ssh -f -N -L 0.0.0.0:1234:databasehost:3306 user@databasehost
\end{verbatim}
on the login node sets up a tunnel for connections at port 1234 and forwards them to your database machine at port 3306, where MySQL listens.
The options \texttt{-f} and \texttt{-N} will let the tunnel continue to run in the background, even after logging out from the login node.
To terminate the tunnel, simply run e.g. \texttt{killall ssh} on the login node. In the client configuration (see below), you would
then specify the IP/hostname of the login node and port 1234 as the database hostname and port.

Other than that, the client has to be able to write temporary files to some location on the filesystem. This can be configured (\ref{client_command_line_arguments}) if it differs from the client binary location.

Because the client will download missing solver binaries and instances and upload results it also needs a reasonably fast network connection to the database. 
\marginlabel{Shared filesystems:}
\index{Shared filesystems}
Shared filesystems can considerably reduce the required bandwidth since every file is only downloaded once. Alternatively you can create
a package from within the GUI that contains all solver binaries and instances of an experiment. However, if you modify experiments while the client is running it will still download missing files.

\newpage
\subsection{Usage}
\subsubsection{Configuration}
\marginlabel{Configuration file:}
\index{Client configuration}
Configuration is done by some command line arguments and a simple configuration file, called ''config''. This file has to be in the \textbf{working directory} of the client at runtime.
In the configuration file you have to specify the database connection details and which hardware the client runs on. This is done by configuring so called ''grid queues'' in the GUI application.
They contain some basic information about the computation hardware such as number of CPUs per machine. The client will then use this information to run as many parallel jobs as
the grid queue information allows it on each machine where it is launched. Here is a sample configuration file:
\marginlabel{\Eexample}
\begin{verbatim}
host = database.host.foo.com
port = 3306
username = dbusername
password = dbpassword
database = dbname
gridqueue = 3
jobserver_host = jobserver.host.foo.com
jobserver_port = 3307
\end{verbatim}
\index{Grid queue}
Note that the gridqueue value is simply the ID of the grid queue.

Beside the configuration file there are several command line options the client accepts, please also see ''./client --help'':
\marginlabel{Command line arguments:}
\label{client_command_line_arguments}
\begin{verbatim}
-v <verbosity>:
  Integer value between 0 and 4 (from lowest to
  highest verbosity)
-l:
  If flag is set, the log output is written to a file
  instead of stdout.
-w <wait for jobs time (s)>:
  How long the client should wait for jobs after
  it didn't get any new jobs before exiting.
-i <handle workers interval ms>:
  How long the client should wait after handling
  workers and before looking for a new job.
-k:
  Whether to keep the solver and watcher output files after
  uploading to the DB. Default behaviour is to delete them.
-b <path>:
  Base path for creating temporary directories and files.
-d <path>:
  Download path for resources. If the download path is a shared filesystem,
  all clients with access to it will only need to download files once and then
  copy them to their own base path.
-h:
  Toggles whether the client should continue to run even
  though the CPU hardware of the grid queue is not homogenous.
-s:
  Enables simulation mode where the client will fetch and run
  jobs but won't write any results back to the database.
-t:
  Expects walltime in the format [[[d:]h:]m:]s, you can let the client
  know how long it will be able to run on the system. The expected time left
  will then also show up in the GUI.
\end{verbatim}

Verbosity controls the amount of log output the client generates. A value of 4 is only useful for debugging purposes, a value of 0 will make the client log important messages and all errors.

If the ''l'' flag is set, log output goes to a file whose name includes the hostname and IP address of the machine the client runs on. This is done to avoid name clashes in shared filesystems typically found in computer clusters.
Otherwise log output goes to standard output.

With the ''-w'' option you can tell the client how long to wait before exiting after it didn't start any jobs. This can be useful to keep the clients running and ready to process new jobs
while you evaluate preliminary results and add new jobs or whole experiments. The wait option is also used to determine how long attempts should be made to reconnect to the database after
connection losses. The default value is 10 seconds.

The ''-i'' option controls how long the client should wait between its main processing loop iterations. If this value is low, it will look for new jobs when there are unused CPUs more frequently.
For maximum job throughput this value should be lower than the average job processing time but lower values will also put more strain on the database and increase the client's CPU usage. The default
value is 100ms which should work fine in most cases. The client will also adapt to situations where there are free CPUs but no more jobs and increase the interval internally and fall back to the configured
value once it got another job.

The ''-k'' flag tells the client that it should keep temporary job output files after a job is finished. The default behaviour is to delete them.

The ''-b'' base path option can be used to specify a directory the client can use to write temporary files to. The default value is ''.'', i.e.
the working directory at runtime.

\marginlabel{inhomogenous machines}
\attention The first client to start with a particular grid queue will write the information about the machine it runs on to the grid queue entry in the database. All following clients will then
compare their machines to the information in the grid queue and exit, unless the number of cores and the CPU model name match. With the ''-h'' option you
can override this behaviour.

\subsubsection{Launching}
After configuration you can simply run the client on your computation machines. On computer clusters there are often queuing systems that you have to use to gain access to the nodes.
On bwGRID for example, we could use the following short PBS (portable batch script) and submit (\textit{qsub scriptname}) our client to a node with 8 cores:
\begin{verbatim}
#!/bin/sh
#PBS -l walltime=10:00:00
#PBS -l nodes=1:ppn=8
cd /path/to/shared/fs/with/client/executable
./client -v0 -l -i200 -w120
\end{verbatim}
\attention You should always run the client from within its directory (i.e. cd to the directory) to avoid problems with relative paths such as the verifier path from the example configuration above.

As soon as clients start you should be able to see jobs changing their status from ''not started'' to ''running'' in the GUI's or Web frontend's job browsers.

\subsubsection{Troubleshooting}
If errors or failures occur the client will always attempt to shut down cleanly, that is stop all running jobs and set their status to ''client crashed'' and write
the last lines of its log output as ''launcher output'' to each job. This can fail when network connections fail or the client receives a SIGKILL signal causing it to exit immediately.
In case of network failures you should still be able to find useful information in the client's logfile on the local filesystem.

\subsection{Verifiers}
\label{sec:verifiers}
Verifiers are programs that the client runs after a job finishes. Verifiers are getting passed the instance of the job and the solver output as arguments and are supposed to
write a newline character followed by a (textual/ASCII) integer result code at the end of their output. The result code should convey some information about the result of
the job, for example whether the output of the solver is correct given the problem instance.
This code will be written to the database as ''result code'' while the verifier's exit code will be written as ''verifier exit code''. Any output the verifier writes to standard out will
be written as ''verifier output''. The call specification for a verifier binary looks like this:
\begin{verbatim}
./verifier_binary <path_to_instance> <path_to_solver_output>
\end{verbatim}
We provide a verifier for the SAT problem that works on CNF instances in DIMACS format and solvers that adhere to a certain output format (see the source code).
If you want to write an own verifier specific to your problem you can also use the source code as implementation example.
\attention Note that you have to make sure
that your possible result codes are specified in the \textit{ResultCodes} table in the database before running clients or there will be errors when the client tries to write results. By convention,
the web frontend and GUI application consider result codes that begin with a decimal ''1'' as correct answers.

\subsection{Experiment priorization}
\label{sec:experiment_priorization}
Sometimes it can be useful to compute several experiments in parallel but give some a higher priority than others. In order to accomplish that, experiments can be marked as inactive and individual jobs can be prioritized. Only jobs of active experiments with priority equal to or greater than 0 are considered for processing
by the client. Futhermore, experiments can be assigned a priority. The clients will then try to match the relative number of CPUs working on an experiment with its relative priority to
all other experiments that are assigned to the same grid queue. For example, if you have three experiments with priorities 100, 200 and 300 respectively the running clients will try to have 16\% of CPUs working on the first, 33\% of CPUs working on the second and 50\% of CPUs working on the third experiment.