====== System validation observations ======


Once every week, usually on Thursday morning, we run a suite of
interferometric system validation observations to see if the system is
ready for the science period, which runs from Thursday evening until
Monday morning. This entails a total of 16 observations, covering all
RCU modes and the most common antenna configurations.

The observations are defined in an XML file generated by the
''genvalobs'' script. This script is available on the frontend node,
''lhn001''. Alternatively, the software distribution can be downloaded
from this page.

{{:operator:genvalobs-0.9.tar.gz|}}


=== Command line options ===

The online help of ''genvalobs'' is listed here:

<code>
Genvalobs version 0.6 (c) 2010 Michiel Brentjens <brentjens@astron.nl>
Current LST: 23:19:12.72

usage: genvalobs [options] "source name"

The source name must be enclosed in single or double quotes if it
contains spaces. The following sources are supported:

- "Cas A"  / cas: LST 21:00--03:00
- "3C 123" / 123: LST 01:30--07:30
- "3C 196" / 196: LST 05:00--11:00
- "Vir A"  / vir: LST 09:30--15:30
- "Her A"  / her: LST 14:00--20:00
- "Cyg A"  / cyg: LST 16:00--24:00

If no source is specified, the program chooses the source that is
closest to the meridian at transit.

Options:
-o / --output    Name of the output file. Default is
                 lofar-validation-yyyymmdd-hhmmss.xml

-d / --duration  Duration of individual observations in
                 seconds. Default is 120.

-g / --gap       Gap between observations in seconds. Default
                 is 180.

-s / --stations  Station set. One of: superterp, core, remote, nl,
                 europe, all. Default is nl.

-i / --include   Comma separated list of station names to add to
                 station set.

-e / --exclude   Comma separated list of stations to exlude from
                 station set.
                 
-w / --wait      Number of seconds to wait before the first observation
                 must begin. Default is 540 seconds.

-h / --help      Display manual.
</code>


=== Typical usage ===

If one wants to generate observations that start in 9 minutes from
now, using all LOFAR-NL stations, and a calibrator source close to the
meridian, a simple
<code>
user@lhn001:~ $ genvalobs
</code> 

suffices. If, for example CS021 is not available due to maintenance,
and one would like to test some international stations too, because
they are going to be used over the weekend, then one can use
<code>
user@lhn001:~ $ genvalobs -e CS021 -i UK608,FR606,DE603
</code>

By default, the script generates an XML file in the current directory
with a name like ''lofar-validation-20110428-131701.xml'', which can
be uploaded to MoM using the ''Tools -> XML upload'' command.

The observations are uploaded to the LOFAROPS project, and will appear
at the bottom of the project page. Quickly select all of them, set
them to ''described'', and notify the observer on duty. The
observations should then, in OTB, be set to ''prepared'', and
subsequently set to ''approved'' in MoM. Make sure you move the
observations to the ''System validation'' folder in order to minimize
the clutter in the already full LOFAROPS main page. Again, notify the
observer, who will assign storage nodes and will schedule the
observations using [[software:the_lofar_scheduler|the lofar scheduler]].


=== Data inspection ===

Now carefully note the SAS IDs of the observations. If they are, for
example, 29415 to 29400, then one can generate the inspection plots
after the observations have completed, using the command

<code>
user@lhn001:~ $ launch-msplots.sh `seq 29415 -1 29400`
</code>

Make sure, though, that you have the packages ''LofIm'', ''Pythonlibs'',
and ''Pyautoplot'' loaded, preferably through your ''.mypackages'' file.

This starts inspection processes on all ''locus'' machines that
contain date from these measurement sets. The resulting plots are
written to ''/globaldata/inspect/<sas id>''. I usually rsync that
directory to my own laptop to look at the plots with my local web
browser:
<code>
user@laptop:~ $ cd /data/inspect
user@laptop:/data/inspect $ rsync -avz -e 'ssh -A user@portal.lofar.eu ssh' user@lhn001:/globaldata/inspect/ ./
</code>

Each directory contains hundreds of systematically named ''.png''
files with statistics for each subband and baseline. Pick a subband
somewhere in the middle of the frequency range, and look at several
statistics. The standard deviation and fringe snr are particularly
useful. Any station that looks different in these plots is suspicious
and requires further inspection of log files or individual
visibilities.



MoM XML specs

specifications:  http://www.lofar.org/operations/doku.php?id=mom3:msss_observation_pipelines
documentation: http://www.lofar.org/operations/doku.php?id=mom3:howto_templates
