experimentmode.tex

% !TeX root = user_guide.tex
\subsection{Experiment Mode}
\subsubsection{Experiments}\label{sec:experiments}
\marginlabel{Experiment}\index{Experiment}\index{create/remove/edit experiments}
An experiment consists of solver configurations, instances and the number of runs for each solver configuration and instance. In the experiment tab the user can create/remove/edit experiments.

\marginlabel{Create}
By using the create-button in the first tab of the experiment mode an experiment can be created. This will open a dialog where you have to provide some data.
\begin{enumerate}
\item Name: the name for the new experiment
\item Description: a description for the experiment. Provide some useful information about the experiment to quickly identify experiments in the experiments table.
\begin{item}
Default Cost: this will be the default cost for this experiment. This will affect some default behaviour in the GUI and the WF, e.g. the appropriate column in the job browser will be visible by default and the others will not be visible. The user can choose between three types of costs:
\begin{enumerate}
\item resultTime: the CPU time needed for a run will be used as cost.
\item wallTime: the real time needed for a run will be used as cost.
\item cost: if a verifier is used which outputs cost, then this will be used as cost.
\end{enumerate}
\end{item}
\item Limits: the user can specify if the outputs should be limited. Outputs that can be limited are solver output, watcher output and verifier output. This might save disk space on the DB server. It is possible to preserve the first and/or the last lines or bytes.
\item Configuration experiment: if set, this will be a configuration experiment and the Configuration Scenario tab will be enabled for this experiment, see section \ref{sec:configuration_scenario} for more information.
\end{enumerate}
After pressing the create-button the newly created experiment will be loaded automatically.

\marginlabel{Remove}
To remove an experiment use the appropriate button.

\marginlabel{Edit}
To edit an experiment use the appropriate button. There you can edit the data you provided by creating the experiment. If you want to change the priority of an experiment you can do this by directly editing this property in the experiment table. The same applies to activating and deactivating experiments. For more details about the effect of the priority property, see section \ref{sec:experiment_prioritization}. Deactivated experiments won't be computed by clients.

\marginlabel{Discard}
To discard an experiment use the appropriate button. This button is only available if an experiment is loaded.

\marginlabel{Load}
To load an experiment use the appropriate button or double click the experiment you want to load in the experiment table.

\marginlabel{Import}\label{sec:import_data_from_experiments}
% todo: überarbeiten.
It is possible to import data from other experiments. To import data from other experiments the following steps have to be applied:
\begin{enumerate}
\item Load the experiment you want to import data to
\item Press the import button in the experiment tab. This will open a new window with three tables for experiments, solver configurations and instances.
\item Select the experiments you want to import data from. This will update the solver configuration and instance tables to show all solver configurations and instances for the selected experiments. Orange rows mean that the solver configuration or instance in that row exists in the currently loaded experiment. Two solver configurations are considered as equal if they have the same solver binary associated and have the same launch parameters.
\item Select the solver configurations and instances to import
\item Select \textit{import finished jobs} if you also want to import jobs
\item Press \textit{Import} 

\attention Note that this action might generate new jobs. This \textit{might} happen if you import solver configurations and instances with their jobs to an experiment where some of the solver configurations and instances actually exist and they are in the \textit{same seed group}.
\end{enumerate}
\marginlabel{Filter}
% todo: ...
\subsubsection{Client Browser}
The client browser represents all clients currently connected to the database. Red rows denote dead clients. \marginlabel{Dead clients}A client is considered to be dead if the client didn't communicate with the database for a period of time.

The client browser also deals as the only way to directly communicate with clients. 

\marginlabel{Kill clients}After selecting the clients you can open the context menu with the right mouse button and select \textit{Kill Clients Hard} or \textit{Kill Clients Soft}. Hard means that the clients will terminate all currently computing jobs and sign off. Soft means that the clients won't start new jobs and will wait for the currently computing jobs to finish.

\marginlabel{Client details}To view the jobs which a client has computed in his lifetime you can double click a client entry in the client table. This will show a dialog with a table containing all jobs the client calculated and is currently calculating. You can also send messages to the clients in this dialog.

\subsubsection{Configuration Scenario (Optional)}\index{Configuration Experiment}\index{Configuration Scenario}\label{sec:configuration_scenario}
Configuration Scenarios are used to define a solver with its parameters to be configured with a configurator. This tab is only enabled for configuration experiments, see section~\ref{sec:experiments}. To define a configuration scenario the DB must contain at least one solver with a \marginlabel{Parameter Graph}parameter graph specified (see section~\ref{sec:parameter_graph}). There are two ways to specify a configuration scenario. Either the user imports the configuration scenario from another configuration scenario in the DB by using the \marginlabel{Import Scenario}\textbf{Import Scenario} button or the user specifies the configuration scenario manually. The following steps have to be applied to specify a configuration scenario manually:
\begin{enumerate}
\item Select the solver to be used for this configuration scenario in the \textbf{Solver} combo box. This combo box will only contain solvers which have parameter graphs specified. If it contains no solvers then you might want to head to the DB mode and specify a parameter graph for one or more solvers.
\item Select the solver binary for the selected solver that will be used to execute the solver on the grid in the \textbf{Solver Binary} combo box.
\item Select the parameters that should be used for the configurator. First the user has to select the parameters that should be used for the solver configurations created by the configurator. Then the user must specify which of these parameters should be configured and which have fixed values. If a parameter should have a fixed value and is not a boolean parameter then the value should also be specified by the user.
\end{enumerate}
After the steps above the configuration scenario can be saved. It is now possible to generate some random solver configurations to test the configuration scenario. To do this the user has to use the \index{Generate Solver Configurations}\marginlabel{Generate Solver Configurations}\textbf{Generate Solver Configurations} button. An input dialog will pop up and the user can choose the number of solver configurations that should be created. After applying the dialog the solver configurations will be created and saved to the DB. The user can see the result in the \textbf{Solvers} tab.

The last task which can be done in this tab is generating an \index{Instance, Seed-Course}\marginlabel{Instance, Seed-Course}Instance, Seed-Course for the configuration experiment. After selecting the instances for the solver to be trained on in the \textbf{Instances} tab (see section~\ref{sec:instances}) the user can generate an Instance, Seed-Course by using the appropriate button. This will open a dialog with a table containing the course. The user can move/sort the instances to change that course. By selecting the columns with some instance properties by using the \textbf{Columns} button the user has the ability to sort the instances after an instance property.

\subsubsection{Solvers}
\index{Solver configuration}Creating solver configurations is done in the solvers tab. This tab contains a table on the left side and a panel with all solver configurations currently associated with this experiment.
\marginlabel{Choosing solvers}To create solver configurations you have to choose solvers for which you want create solver configurations. This can be done in the left table, the solvers table. By selecting some solvers and finally pressing the \textit{choose}-button, solver configuration prototypes will be created for the solvers. You can see the newly created solver configurations in the panel on the right side. This panel is organized as follows. For each solver exists one layer. Each layer contains all solver configuration for the associated solver. A solver configuration is titled with a name. The name can be changed and is used in the other areas of the GUI to identify a solver configuration. So it might be good practice to choose different names for the solver configurations in an experiment.

\marginlabel{Modifying solver configurations}A solver configuration consists of a solver binary, parameters and a seed group. The solver binary is chosen in the first combo box. The parameters can be specified in the parameters table. Just select the parameters you want for this solver configuration and specify their values if they have some. Finally you have to specify the seed group. The default seed group is \textit{0}. You might want to change that. See section \ref{section:seed_groups} for more information about seed groups.

\marginlabel{Importing solver configurations}To import solver configurations from other experiments you can import them in the experiments tab (see section \ref{sec:import_data_from_experiments}) or if you just want to import solver configurations without their jobs, you can use the \textit{Import}-button. This will open a dialog where you can import solver configurations where you have two options. Either you want to import solver configurations from one or more experiments or you want to import solver configurations from one or more solvers. Choose the tab in this dialog accordingly. After selecting the solver configurations to be imported, use the \textit{Import}-button of this dialog to import the selected solver configurations.

\attention The imported solver configurations will not be saved to the DB until the user uses the \textit{Save}-button. 

\marginlabel{Tabular view for solver configurations}To change the view of the solver configuration panel to a tabular view, press the \textit{Change View}-button. This will change the panel into a table. Here you can remove multiple solver configurations by selecting them and opening the context menu by pressing the right mouse button and choosing \textit{Remove}. It is also possible to edit solver configurations in that view by double clicking a solver configuration or by using the context menu. If you select a solver configuration in the tabular view and change back to the normal view then the view will automatically be scrolled to the previously selected solver configuration.

\attention All modifications to solver configurations are not directly saved to the DB. You can always use the \textit{Undo}-button the undo all changes and load the last saved state. By pressing the \textit{Save}-button all modified and new solver configurations will be saved to the DB and deleted solver configurations will be removed from the DB.

\attention Modifying and saving solver configurations which have calculated runs might be not a good idea. Therefore the GUI supplies a possibility to reset the affected jobs. This might not be needed if the changed parameters have no effects to the results.
\subsubsection{Instances}\index{select instances}\label{sec:instances}

Instances are associated with an experiment in the \index{Instances tab}\textit{Instances} tab. This tab consists of two tables. On the left side are the instance classes and on the right side are the instances which are in the selected instance classes.

To associate instances to the currently loaded experiment, you can use the buttons below the instances table or select the instances manually. Additionally it is possible to filter instances by using the \marginlabel{Filter}\textit{Filter}-button and sort the instances by clicking on the appropriate column in the table header. With the \marginlabel{Columns}\textit{Columns}-button other columns can be made visible, e.g. columns for instance properties. The \marginlabel{Undo}\textit{Undo}-button can be used at any time to revert changes to the last saved state. To make changes permanent the \marginlabel{Save}\textit{Save}-button must be used.

\marginlabel{Import}It is also possible to import the selection from other experiments by using the import functionality in the experiments tab, see section~\ref{sec:experiments}.
\subsubsection{Generate Jobs}\index{Generate Jobs}
After choosing solver configurations and selecting instances for the experiment, jobs can be generated in the \textit{Generate Jobs} tab. In this tab there is a table representing a matrix with the instances and solver configurations in the experiment. Each cell in the table represents the number of jobs for the instances in that row and the solver configuration in that column.

To set the number of jobs for all or the selected cells, you can use the \textit{Set Number of Runs}-button. This will open a dialog where you can choose if you want to set the number of runs for all or only the selected cells. With the \textit{Number of Runs}-text field you can choose the number of runs and finish this process by using the \textit{Apply}-button. Now you can determine which cells have be changed. \marginlabel{Colors}White cells means no change, red cells means that the value is below the actually saved value and green cells means that the value is above the actually saved value. \marginlabel{Generate Jobs}By using the \textit{Generate Jobs}-button, those changes can be made permanent, and jobs will be created and/or deleted accordingly. This will open a dialog where you have to provide some data:
\begin{enumerate}
\item Timeout: the CPU time limit for the newly created jobs in seconds.
\item Max memory: the maximum amount of memory the newly created jobs can use in megabytes.
\item Wall clock time limit: the maximum real time the newly created jobs can use in seconds.
\item Stack size limit: the maximum stack size the newly created jobs can use in megabytes.
\item Max Seed: if seeds have to be generated (i.e. there are solver configurations which have a seed parameter) then this will be the maximum seed.
\end{enumerate}
If the value \textit{-1} is given for the limits, it means that there is no limit. After using the \textit{Generate}-button the jobs will be generated. The results of this process can be revised in the \textit{Job Browser} tab, see section~\ref{sec:job_browser}.

\index{Queue Selection}\marginlabel{Queue Selection}By using the \textit{Select Queue}-button it is possible to select one or more grid queues for computation. This will open a dialog where you can select grid queues for this experiment. For more information about creating grid queues see section~\ref{sec:grid_queues}.

\index{Generate Cluster Package}\marginlabel{Generate Cluster Package}..
\subsubsection{Grid Queues}\index{Grid Queue}\label{sec:grid_queues}

\subsubsection{Job Browser}\label{sec:job_browser}

\subsubsection{Analysis}