.. highlight:: bash Quality Assurance 2 =================== Allegro also performs QA2 (Quality Assurance 2) work before data delivery to the PI. These projects have a dedicated category, and a dedicated selection of scripts simplifies the work for the QA2 reducers. Starting a QA2 project ---------------------- * As user **alma**: #. Create a QA2 project (in this example we use project ID ``2013.1.12345.S``) with the aid of the :ref:`lab-pm-wizard` and link yourself to the project (not alma, you as you :D ). Fill in the JIRA QA2 tickets (the numbers can be found in the email from ESO; we only need the ID that follows the ``SCOPS-`` part). #. Go into the project directory. These are all in ``/allegro/data/projects``, where ``FS`` can currently be one of ``/lustre1``, ``/lustre2`` or ``/allegro1``. The project subdirectories have (deliberately) obscure names. There are 3 ways to identify your project: a. There will be a soft link to it in ``$ALLEGRO/home/``. b. If you select the project in the wizard, then open a new terminal by clicking on the terminal icon in the wizard, the terminal will open in that directory. c. An alias has been generated that allows you to immediately jump into the project root directory. The alias consists of a prefix ``P_`` (for project), the category (``qa2_`` in that case), the short ID (``12345.`` here, but it is generally at least 3 digits long with leading ``0``), and the cycle (``2`` in this example). So typing ``P_qa2_12345.2`` will jump directly into this project. This alias only works for Allegro members. #. Go into the archive directory of the newly created project and create a directory with the schedule block (SB) name, e.g.: :: $ cd archive $ mkdir B6_IRAS12345 #. Go into that directory and download the raw ASDMs (`ALMA Science Data Model `_) and the md5sum. A helper tool ``qa2_download`` can help you. It needs as a command line argument the path of the webpage where the raw data is provided (that link can be found in the email from ESO). This script will search for ASDMs, download them and check the md5sum. E.g: :: $ cd B6_IRAS12345 $ qa2_download http://almascience.eso.org/arcdistribution/qa2-for-nodes/2a599e929r9d9w9dlsls9f0g0d0020sd $ ls md5sums.txt uid___A002_X4sda4d_Xa12 uid___A002_X4sda4d_Xa12.tar uid___A002_Xdd9c61_X66d uid___A002_Xdd9c61_X66d.tar In this example, the project contains two execution blocks (EBs), which were downloaded as a ``tar``-file, and then extracted by the ``qa2_download`` script. The raw data is a so-called *ASDM*. #. Update the analysisUtils. :: $ update_jao_mirror If you get a message that the update of the analysisUtils is locked, just contact the person in question, and check if an update is necessary/possible. #. Log out as user alma. * As **yourself**: #. Go into your analysis directory: :: $ cd /analysis/ #. Establish a QA2 lock, which prevents others from accidentally updating the analysisUtils. Optionally, you can check the status. :: $ qa2_lock True Locking analysisUtils for QA2 $ qa2_lock * 27-Aug-2014 17:01 (you) #. Set up the directory structure for the calibration and imaging. Again, there is a little helper tool, ``qa2_start``, that assists you with that. As a command line argument it needs the SB name (see point 2.). This script will create a subdirectory with the SB name, which contains more subdirectories (more about that in the next step). :: $ qa2_start B6_IRAS12345 If you run ``qa2_start`` without an SB as argument, the script will a. list the SBs if there is more than one; b. if there is only a single SB, the script will make the correct subdirectory structure for that. The newly-created SB directory contains two sub-directories: :: $ cd B6_IRAS12345 $ ls B6_IRAS12345 packaging ... (Note that there are **two** directories with the SB name, at different layers of nesting.) You will also see a few files, e.g: :: ... Checklist_SB_B6_IRAS12345.txt Checklist_SB_B6_IRAS12345_uid___A002_X4sda4d_Xa12.txt Checklist_SB_B6_IRAS12345_uid___A002_Xdd9c61_X66d.txt README.header.txt scriptForPI.py We will see the purpose of these directories and files later. Now, everything is set up, and you are ready to jump into the calibration. Doing QA2 --------- The general instructions for doing QA2 are found at the `ESO TWiki Cycle2DataReduction `_. Broadly the procedure breaks into 2 steps: #. Calibrate, and report on (via checklists etc), each execution block (EB) separately. #. Flux equalization of the EBs; combining the EBs to create images; and final SB-level reporting. Note that we have some local differences here (different directory structure, availability of allegro-specific scripts) from the ESO setup. In more detail: #. Move into the SB directory. :: $ cd B6_IRAS12345 $ ls calibrated X66d Xa12 You find one directory for each EB, and one (``calibrated``) where the two EBs are then merged. In the example, there are 2 EBs: ``X66d`` and ``Xa12``. #. The EB-level calibration is outlined as follows. * Each EB subdirectory contains just a single file, which is the ASDM: :: $ cd X66d $ ls uid___A002_Xdd9c61_X66d * Start up casapy: :: $ casapy If you have set up your ``${HOME}/.casa/init.py`` correctly, you should see, in the casapy initialization messages, the import of analysisUtils, with a version number (which is needed later for the checklist). * Generate the EB calibration script via the command :: >>> es.generateReducScript() This can take from 15 to 30 minutes to run. The calibration script is placed in the PWD (which should be ``/analysis////`` - note the double occurrence of the SB name) and will have a name which ends in ``scriptForCalibration.py``. It is useful to have a look in it: many checklist items can be answered from information in the script. * Decide on the calibration steps you want to run, and pre-set them via :: >>> mysteps = [] The steps are described right at the beginning of the calibration script. **NOTE** that the EB checklist requires observation dates and the total integration time, which are output by the CASA task ``listobs``, called at an early step in the calibration script. This info is printed at the casapy prompt, but if you don't want to risk it scrolling off the screen, it may be advisable to run the cal script in two stages: first to the step which runs ``listobs``, and then to do the remaining steps. * Now you are ready to run the calibration script: :: >>> execfile() This will generate a lot of diagnostic plots which are placed in subdirectories off the PWD which have names ending in ``.plots``. A large part of QA2 is checking these plots for 'sanity'. * Once you have imported the ASDM into a measurement set (the first step in the ``scriptForCalibration.py``), you can use the ``qa2`` module of the ``allegroUtils`` to get a first overview of the dataset. :: >>> from allegroUtils import qa2 >>> qa2.listobs(vis='uid___A002_Xdd9c61_X66d.ms') Note that this ``listobs`` is not the same as the CASA tool of that name. * Fill in the EB checklist. + All the checklists are located 2 levels higher in the directory structure, i.e. at ``/analysis///``. The EB ones have the EB identifier as part of their names; the SB-level checklist has no EB identifier. For the present example, the EB checklist is named ``Checklist_SB_B6_IRAS12345_uid___A002_Xdd9c61_X66d.txt``. The present example SB will have fields as follows: :: Project code : 2013.1.12345.S Contact Scientist: You can get this from the SB JIRA ticket (linked from the Wizard). :: PI : Listed on the Wizard. :: SB Name : B6_IRAS12345 ASDM UID : uid___A002_Xdd9c61_X66d Observation Dates and Total Integration Time (from listobs): See above. :: PWV : A really precise value is not important - you can estimate it by eye from the graphs produced by ``qa2.listobs``. This task however also prints a median value to the casapy prompt. :: Number of good antennas: ? :: Band(s) : ? :: TDM/FDM/both : TDM = 'Time Domain Mode', which always has 128 channels and 2 GHz bandwidth. FDM = 'Frequency Domain Mode' may have varying channel schemes. :: bandwidth switching (different spectral setup for phasecal and target): Get it from listobs (qa2 or casa??). (Note that 'phasecal' means the same as 'gaincal'.) :: Fields by intent (paste from reduc script): DATA IMPORT -----------------------------------------------------[ ] Check that ``importasdm`` was invoked in the cal script. :: Comments: Yes No - asis='Antenna Station CalAtmosphere Receiver Source' [ ] [ ] Normally answer 'no' to this unless it is specified in the script. :: A) INTERFEROMETRIC CALIBRATION ----------------------------------[ ] 1) Apriori calibration ----------------------------------------[ ] Comments: Yes No - fixplanets necessary? [ ] [ ] See if this was called as one of the 'steps'. :: - fix of SYSCAL table time necessary? [ ] [ ] You need to catch this in the casapy output! :: - other es.fix necessary? name: [ ] [ ] Check in the cal script (expect 'no'). :: - checked WVR tables? [ ] [ ](ACA) - checked Tsys tables? [ ] Check the relevant plots. Look for excursions or other anomalies in the spectral windows. :: - Antenna position table necessary? [ ] [ ] Check in the cal script - specifically, in the call to ``gencal``, if the parameters are all zero, then answer 'no'. :: - checked Weights? [ ] Check the relevant plots. At this point in the script, you can expect that the 'split' command is invoked, which will also discard unnecessary spectral windows (for example, all the even-numbered ones, which are averaged of the odd-numbered ones). :: 2) Other flagging and set cal models --------------------------[ ] Comments: - flagged edge channels (TDM mode)? [ ] [ ] Check in the cal script to see if the relevant command was present. :: - flagged atmospheric lines? [ ] [ ] SPW and channel ranges: Ditto. :: - spatially resolved flux calibrator [ ] [ ] - flagged channels with spectral features of the calibrator? (if not in Butler-JPL-Horizons 2012) At the moment we always just flag the Butler JPL channels. :: - checked model amp vs uvdist? [ ] Casapy produces a plot of this. Check it for sanity. :: uvrange (m) for later gaincal: ? :: - Quasar flux calibrator [ ] [ ] - reference value from ASDM Source tbl? [ ] - other reference: [ ] ? :: 3) bandpass ---------------------------------------------------[ ] The 'bandpass plots' are actually graphs against time of the spectrally-averaged phases of the bandpass calibrator. Some of the plots will be empty: these were flagged antennas. :: Comments : - Is REFANT chosen by script gen. good? [ ] [ ] Name of REFANT: Should be among the plots? :: - checked .ap_pre_bandpass table? [ ] The overall trends should be smooth. :: - mixed: used solint = 'inf,XMhz'? [ ] The two solint fields refer to time and frequency averaging respectively. The QA2 person may need to (or always must???) hack the script. ?? :: - TDM and mixed: checked .bandpass table? [ ] - FDM: checked .bandpass_smooth20ch table? [ ] The overall trends should be smooth. :: 4) amp cal of all calibrators (gaincal) -----------------------[ ] Comments: - For resolved flux cal: limited uvrange? [ ] [ ] Check the script, copy affected ants to comments (??) :: - For resolved flux cal: paste .fluxscale file here This file is in the PWD. In the present example it is named ``uid___A002_Xdd9c61_X66d.ms.split.fluxscale``. Cut-and-paste all the 'Flux' rows. (Note there is a script ``qa2_fluxscale.sh`` which will echo the necessary information. Just run it, with no arguments, in the PWD on the linux command line.) ('See also qa2_flux.py' - why, what does this do??) :: - checked .phase_int table? [ ] See relevant plots. The overall trends should be smooth. :: - checked .ampli_inf table? [ ] Note that, if the primary flux calibrator is spatially resolved, there will be ``.ampli_inf_short`` files. You don't want these: the plot files you want are in directory ``.flux_inf`` in this instance. The checking criterion is that the flux *ratios* between the various calibrators should stay the same from antenna to antenna. :: - checked .phase_inf table? [ ] Note that 'int' implies a fine time resolution, i.e. no binning of time samples; whereas for 'inf' all the time samples are averaged. Thus if the 'int' phases were ok (see item above), it is hard to see how the 'inf' ones can be wrong. :: - checked uv data on calibrators? [ ] This is checked by running ``plotms_summary `` at the casapy prompt. (This task is part of the qa2 utils.) :: 5) applycal and split out of .split.cal dataset ---------------[ ] Check the last 'split' performed in the script. :: 6) generated qa2 report ---------------------------------------[ ] See the ESO guidelines. :: 7) In case of good flux calibration: upload of flux info ------[ ] There is a script within CASA: ``qa2_upload_flux_scale.sh``. * Fill in the SB checklist. Blah... * If you have made the first run, which lead to some modifications of the ``scriptForCalibration.py``, you can start with a cleared directory by running the little helper ``qa2_clear`` in the EB directory. This script will delete all files except the ``scriptForCalibration.py`` and the ``ASDM``. :: $ ls -1 applycal.last bandpass.last ... uid___A002_Xdd9c61_X66d uid___A002_Xdd9c61_X66d.ms uid___A002_Xdd9c61_X66d.ms.antpos uid___A002_Xdd9c61_X66d.ms.flagcmd.png uid___A002_Xdd9c61_X66d.ms.flagversions uid___A002_Xdd9c61_X66d.ms.listobs uid___A002_Xdd9c61_X66d.ms.scriptForCalibration.py uid___A002_Xdd9c61_X66d.ms.split ... $ qa2_clear Keeping the following files: - uid___A002_Xdd9c61_X66d - uid___A002_Xdd9c61_X66d.ms.scriptForCalibration.py Continue? (y/[N]) y Deleting files ... done $ ls -1 uid___A002_Xdd9c61_X66d uid___A002_Xdd9c61_X66d.ms.scriptForCalibration.py .. _`fig-fluxscale`: .. figure:: images/fluxscale.png :figwidth: 470 :scale: 70% :align: right :alt: fluxscale Comparison of flux density scales derived from two different execution blocks (X66d, Xa12) * You can check the flux density of the calibrators by running ``qa2_fluxscale.sh`` in the console: :: $ qa2_fluxscale.sh Flux density for J1751+0939 in SpW=0 (freq=2.16276e+11 Hz) is: 2.22776 +/- 0.00677892 (SNR = 328.631, N = 9) Flux density for J1751+0939 in SpW=1 (freq=2.17226e+11 Hz) is: 2.22678 +/- 0.00226855 (SNR = 981.587, N = 9) Flux density for J1751+0939 in SpW=2 (freq=2.1995e+11 Hz) is: 2.2078 +/- 0.00680433 (SNR = 324.471, N = 9) Flux density for J1751+0939 in SpW=3 (freq=2.18298e+11 Hz) is: 2.22428 +/- 0.00555449 (SNR = 400.447, N = 9) ... Flux density for J1851+0035 in SpW=7 (freq=2.32452e+11 Hz) is: 0.26856 +/- 0.00338729 (SNR = 79.2846, N = 9) Fitted spectrum for J1751+0939 with fitorder=1: Flux density = 2.19474 +/- 0.00107564 (freq=223.031 GHz) spidx=-0.549826 +/- 0.016824 Fitted spectrum for J1851+0035 with fitorder=1: Flux density = 0.285309 +/- 0.00565662 (freq=223.031 GHz) spidx=-1.4004 +/- 0.573228 You can also get graphical output (:num:`Figure #fig-fluxscale`) when you use the ``qa2`` module within CASA: :: >>> qa2.fluxscale(sb=True) Check the helpfile of this function, since you have many more options, e.g., also comparing the derived flux density scales from different execution blocks. To upload the flux density information (if necessary; check the ESO webpage if you have to), simply run ``qa2_upload_fluxscale.sh``. The script will automatically upload all the necessary files to the JAO server. * Generate the QA2 report. #. Combine the EBs for the flux density calibration and imaging. A helper tool ``qa2_fluxequal_start.sh`` will link the necessary files (``*.split.cal``, ``*.fluxscale``) from the EB directories. :: $ cd ../calibrated $ ls $ qa2_fluxequal_start.sh $ ls uid___A002_X4sda4d_Xa12.ms.split.cal uid___A002_X4sda4d_Xa12.ms.split.cal.fluxscale uid___A002_Xdd9c61_X66d.ms.split.cal uid___A002_Xdd9c61_X66d.ms.split.cal.fluxscale $ casapy You can then start to generate the ``scriptForFluxCalibration.py`` and the ``scriptForImaging.py``. Also here, if you think that you want a clean start, the ``qa2_clear`` will help you to remove all files except the ``*fluxscale``, ``*split.cal``, ``scriptForFluxCalibration.py`` and ``scriptForImaging.py``. Do not forget to fill out the checklist for the SB ``Checklist_SB_B6_IRAS12345.txt``, and the ``README.header.txt`` file for the PI. After QA2 --------- If you think that you are done with QA2, you have to inform the Data Reduction Manager (DRM) at ESO through the JIRA QA2 ticket. #. Go into the packaging directory to generate the files that have to be appended to the JIRA ticket :: $ cd packaging/ticket $ ls run_packaging.py $ casapy >>> execfile("run_packaging.py") #. This script will generate a directory ``JIRA``. All the files in that directory must be uploaded to the JIRA ticket. Moreover, a short description of the calibration and imaging should be added to the ticket. :: $ ls JIRA 2013.1.12345.S.B6_IRAS12345.ticket.zip Checklist_SB_B6_IRAS12345.txt Checklist_SB_B6_IRAS12345_uid___A002_X4sda4d_Xa12.txt Checklist_SB_B6_IRAS12345_uid___A002_Xdd9c61_X66d.txt #. After project approval, generate the hard package, and deliver it to ESO. :: $ cd packaging/hard $ casapy >>> execfile("run_packaging.py") >>> exit $ qa2_deliver That script will create a directory in your ``public_html/earlyscience``, which can be accessed through the web, and link all the data files and md5sum into it. Inform the DRM where he/she can find the data. Once the DRM has the data you can delete the directory from your personal webspace. Note that ``qa2_deliver`` will also copy a bunch of stuff to ``/net/chaxa/data1/QA2_delivery/``. The space on this disk is not infinite, so you'll need to watch that QA2 projects are deleted in timely fashion, so the buildup in ``chaxa:/data1`` does not get too large. #. Unset the qa2-lock :: $ qa2_lock False