#### Previous topic

Tutorial for the MASCARA reduction pipeline

Observer

# funcs – general usage functions¶

## Introduction¶

The funcs module contains classes and functions of general utility of astronomical purpose. Most of these functions are used all over the place throughout mascara, but can also be used as stand-alone functions.

The mascara.funcs module is composed of three submodules to make the organisation and loading simpler. The submodules are mostly independant from each other and relate to different topics:

## timing – Calendar conversion and timing functions¶

mascara.funcs.timing contains various timing routine which are commonly used in astronomy. The calendar conversion functions concentrates mostly on converting calendar dates into Julian dates back and forth using mascara.funcs.timing.calendar2jd() and mascara.funcs.timing.jd2calendar() For instance, for converting a calendar date to Julian Date:

>>> from mascara.funcs.timing import calendar2jd
>>> calendar2jd(2015, 02, 23, 11, 23, 43)
>>> 2457076.974803241


It is also possible convert a time difference in Julian date:

>>> from datetime import timedelta
>>> calendar2jd(timedelta(seconds=20))
0.00023148


Moreover, it is possible to check for the time continuity inside a sequence. mascara.funcs.timing.checkdiscontinuity() verifies that given the observing time and exposure time of two consecutive images, these are taken one after the other.

>>> from mascara.funcs.timing import checkdiscontinuity
>>> checkdiscontinuity((2457076.9748, 2457076.9948), 10.)
True


## mio – I/O functions¶

mascara.funcs.mio contains a large number of I/O functions, going from writing files to disks, finding files or creating directories. Some basics examples are below:

### Saving functions¶

For saving an image and header into a compressed fits file with mascara.funcs.mio.saveCompFits() :

>>> from mascara.funcs.mio import saveCompFits,  savepng
>>> data = numpy.random.rand(1000,1000)


For saving an image in .png format and binning it beforehand:

>>> image = numpy.random.rand(1000, 1000)
>>> savepng(image, 'mydirectory', 'myimagename', binfactor=4)


### Find files or directory¶

For finding all the files ending with ‘LPS’ in mydirectory with a given format use mascara.funcs.mio.find_files() :

>>> from mascara.funcs.mio import find_files, list_directories, pick_directories
>>> fn = find_files('mydirectory', '*LPS', format='.fits')


It works also for finding files starting with LPS in another format:

>>> fn = find_files('mydirectory', 'LPS*', format='.h5')


Similarly, to list all the subdirectories inside a given directory which have a given common root with mascara.funcs.mio.list_directories()

>>> directories = list_directories('mydirectory', 'LP')
>>> print directories
['mydirectory/201410LPS',
'mydirectory/201410LPW',
'mydirectory/201522LPS']


And if inside that list, an additional selection along another root is required with mascara.funcs.pick_directories()

>>> pick_directories(directories, 'S')
['mydirectory/201410LPS',
'mydirectory/201522LPS']


### Disk operations¶

mascara.funcs.mio.check_free_space() calculates the available disk space on a give hard drive. The function is valid both on Linux and Windows platforms. The output is in MB.

>>> from mascara.funcs.mio import check_free_space, make_disk_space, moveData
>>> check_free_space(folder='C:\\')
450273L


It is possible to then free some disk space by deleting files inside a folder. mascara.funcs.mio.make_disk_space() will by default remove the oldest files / subdirectory inside the given folder

>>> make_disk_space(path='E:\\', file = 'thisone.txt')


To copy a directory from one location to another:

>>> moveData(frompath, topath)


## utils – General usage functions¶

mascara.funcs.utils includes some general functions for statistics:

Let’s demonstrate how they work:

>>> distr = numpy.random.rand(1000)
>>> distr = numpy.hstack((distr, np.ones(10)+1.5))
>>> print distr.std(), mascara.funcs.utils.robuststd(distr)
0.348824171156 0.288635778078
>>> mascara.funcs.utils.percentile(distr, 0.75)
0.75398454483993305


But also for reading in and working with stallar catalogues:

>>> bla = mascara.funcs.utils.loadStellarCatalog(filename='StarCat.fits', Vmag=(4., 7.))
>>> ra, dec, vmag, bmag, sID, sptype = bla


### The Stacker class¶

The Stacker class is for easy stacking of images taken from the same camera at close observing time. The images are ‘rotated’, practically shifted, from their observing time to a reference time. In order to work best, it is recommended that the observing time and reference time do not differ by more than 10min. The Stacker class needs as inputs a mascara.observer.Site and mascara.observer.Camera instance.

>>> from mascara.funcs.utils import Stacker
>>> stacker = Stacker(site, camera)
>>> stacker.image_rotate(image, obstime, reftime, exposuretime=6.4)
>>> binnedimage, nimages = stacker.binImage(datetime.now())


### The multiprocessing logging classes¶

Python 3.2 and higher up versions enable easy logging with multiprocessing. The two following classes are used to reproduce this feature on Python 2.7:

>>> import mascara
>>> import multiprocessing
>>> ### Start a queue which allows communication between the processes
>>> qlog = multiprocessing.Queue()
>>> ### Convert the queue to a handling function
>>> qh = mascara.funcs.utils.QueueHandler(qlog)
>>> ### Get the logger
>>> logger = logging.getLogger()
>>> ### Give the logger the handler
>>> ### And now create the function which will read from the queue
>>> ql = Mmascara.funcs.utils.QueueListener(qlog, handler)
>>> ql.start()


## References/APIs¶

### Stacking of images¶

class mascara.funcs.utils.Stacker(site, camera, **kwargs)[source]

Stacker creates a stacker instance which is used to stack images on top of each other in order to bin them temporally. The stacker first align the images before stacking them. In order to do so, it requires valid site and camera instances as inputs.

__init__(site, camera, **kwargs)[source]

Initialise the Stacker Class.

Inputs:
• site, a Site instance
• camera, a CamCoord instance
Keywords:
• imageshape, provide here a tuple with the image size to use if not provided, the stacker will use the image size definition from the camera instance.
• sidex, sidey, two integer values to define the size of the boxes which will be moved around to realign the images. Large sidex-sidey make the stacking fast, but sensible to camera aberrations.
• oversize, an integer defining the oversize factor to use while aligning the image. If the factor is too small, there is a risk of aligning outside the image deifinition frame.
binImage(endtimestamp)[source]

Bin the actual image

Input:
• endtimestamp, a datetime object or JD date, which is used to marked the end of this binning sequence
Outputs:
• binimage, the binned image
• nimages, the number of images used for binning
image_rotate(image, currenttime, TimeReference, exposuretime=False)[source]

“rotate” image to new reference frame. The image is actually not rotated, but shifted in pieces to the new location for the given TimeReference.

Inputs:
• image, the image to ‘rotate’
• currenttime, when this image was taken. Currenttime can be either a datetime object or JD date
• TimeReference, the new time reference for ‘rotating’ the image
Keywords:
• exposureTime, default false, else precise the exposure time

of the image

Outputs:
• cxp, cyp, the grid coordinates at the new ReferenceTime
makeGrid()[source]

makes the grid used for rotating the images

### Logging with multiprocessing on Python 2.7:¶

class mascara.funcs.utils.QueueHandler(queue)[source]

This class is new in python 3.2 and later but has been copied here for python 2.7. It sends a logging handler to a Queue. It would be used together with a multiprocessing queue to centralise logging to file in a multi-processing configuration and to avoid write contention between processes.

__init__(queue)[source]

Initialise the instance using the passed queue.

emit(record)[source]

Emits a record. Writes the Logrecord to queue, preparing it first.

enqueue(record)[source]

Enqueue a record. This implementation uses the no_wait : event are put to queue as soon as possible .

prepare(record)[source]

Prepares a record for queueing. The object returned by this method is enqueued.

class mascara.funcs.utils.QueueListener(queue, *handlers)[source]

Queue listener works in pair with QueueHandler for writing/printing log files using logging in the case of multi-processes. This class implements an internal thread listener which watches for LogRecords being added to Queue, removes them and passes them a list of handler for processing.

__init__(queue, *handlers)[source]

initialises an instance with the specified queue and handlers

_monitor()[source]

Monitors the queue for records and ask the handler to deal with them. This method run on a separate internal thread. The thread will terminate once it sees the sentinel object in the queue.

dequeue(block)[source]

dequeue a record, and optionally block it

handle(record)[source]

Handle a record. This just loops through the handlers offering them the record to handle.

prepare(record)[source]

prepares a record for handling.

start()[source]

Start the listener... This starts up a background thread to monitor the queue for LogRecords to process.

stop()[source]

stop the listener. This asks the thread to terminate and wait till it does so. Note that if we don’t call it before the application exits, there may still be some records inside the queue.

### timing – Calendar conversion and timing functions:¶

timing functions used for MASCARA

Includes:
• Timing routines:
• check_discontinuity
• astrotiming
• calendar2jd
• jd2calendar
• jd2epoch
mascara.funcs.timing.checkdiscontinuity(obstime, texp)[source]

Given an exposure time -texp- and an array containing at least the last 2 dates of observation, checktimescale verifies that the latest exposure is made in the continuity of the previous one. This is relevant in cases of a serie of observations during one night. If the 2 exposures diverge by more than twice the exposure time, than a discontinuity is found, and the function returns True.

Input:
• obstime, an array containing the dates of observation.
• texp, a scalar for the exposure time, and including the read- out time of the detector.
Output:
• a boolean value, True if a discontinuity is found, else False
mascara.funcs.timing.jd2calendar(jd, tz=None)[source]

Convert a Julian day date (JD) into a calendar date, using a datetime format.

Input:
• JD, a scalar or a vector, in float64 precision
Output:
• date, a datetime object in UTC timezone
mascara.funcs.timing.calendar2jd(*args, **kwargs)[source]

Calendar to Jd converts a datetime object into a Julian date object

Input:
• date, a datetime object or a timedelta object or an array of datetime objects
Output:
• JD, the Julian date in np.float64
mascara.funcs.timing.jd2epoch(jd, Julian=True)[source]

Conversion of a given Julian date to an epoch expressed in decimal years.

Input: jd, the Julian date to convert, if jd is None, it takes the
current date

Keyword: Julian. If not set, the reference epoch is 1900

Output: epoch. The date converted in decimal years.

mascara.funcs.timing.epoch2jd(epoch)[source]

Convert an epoch into a Julian Date

### mio – I/O functions:¶

I/O functions used for MASCARA

Includes:
• CheckFreeSpace() check how much disk space is free on computer
• MakeDiskSpace() delete the oldest folder found in a given path
• find_files(), find files in path according to root
• list_directories(), list all directories
• pick_directories(), select from a list those who satisfy condition
• makedir(), wrapper around os.makedirs to create several directories at once
• cleanup() removes directories forcibly (even if data remains inside!)
• moveData(), which is more a copy data than a move data
• savepng()
• savecompfits(), to save a compressed fits file
mascara.funcs.mio.find_files(path, strm, format='.fits')[source]

List of the files in the given format in the given path directory. It works very similarly to : ls mydirectory/test*.fits

Inputs:
• path, the complete path of the directory to look at
• strm, the string to find (using * to get the files)
Keyword:
• format, default ‘.fits’
Output:
• a list of all files satisfyig the condition
mascara.funcs.mio.list_directories(path, root)[source]
list_directories works a bit list the os.listdir function, except that you

can feed it a root and it’ll list all directories having that root in their names in the given path.

Inputs:
• path, a string, giving the location where to look
• root, a string, giving the root to use for search
Output:
• a list of the directories
Example:
>>>  dires = list_directories('D:\LaPalma\', '20141114')

mascara.funcs.mio.pick_directories(dirlist, root)[source]

picks the right directories for a list using the root as reference

Inputs:
• dirlist, a list of directories
• root, a reference to search for in the list
Output:
• a list of the directories
mascara.funcs.mio.check_free_space(folder='D:\\')[source]

CheckFreeSpace looks how much free space is in the given folder... preferably the hard disk. To be used to for overwriting/deleting old raw data at the beginning of the night, to avoid getting into memory troubles.

Input: folder, a string giving the path of the folder to check

Output: the amount of free space in MB

mascara.funcs.mio.make_disk_space(path='D:\\')[source]

Removes the oldest files in the path

Keyword:
• path, to define where to remove files
mascara.funcs.mio.moveData(frompath, topath, rawpath='D:\\Raw\\', keeprawfiles=False)[source]

Copy the Data from the directory frompath to the directory topath. and delete afterwards the frompath.

Inputs:
• frompath, the source directory
• topath, the destination directory
Keywords:
• keeprawfiles, if True, the program makes a copy of the files from the source to the destination
• rawpath, the directori
mascara.funcs.mio.savepng(image, path, name, binfactor=1)[source]

savepng saves the input image at the given path. The image can be rebinned using the binfactor keyword. The scaling is defined via ...

Inputs:
• image, an array
• path, the saving directory
• the name of the file (without the .png)
Keywords:
• binfactor, if 1, the saved image has the same dimension as the original image. Set to larger as 1 to reduce image size.

to save an image in a compressed fits.

Inputs:
• path, the path for saving the images
• name, the name of the image
• data, the data to be saved

The data is saved in a CompImageHDU, using a RICE compression type. If the data is not in UINT16 format, it will be converted before the compression.

### General usage functions:¶

Utilities functions used for MASCARA

Includes:
• rotmat, create a 3x3 (xN) rotation matrix along a specific axis
• pol2rect, convert polar to rectangular coordinates
• rect2pol, convert rectangular to polar coordinates
• percentile, compute the Nth percentile of a distribution
• makevortex, make Delauney vortices
• match_vortex, match vortices together
• cleanupStellarCatalog -> remove blended stars ...
• find_outliers,
• robuststd, computes the standard deviation of a distribution
• rebin, rebins a 1D or 2D array into a new shape
• statbin, rebins and provides statistics
• gaussian, make a guassian
• gfitting, fit a gaussian
mascara.funcs.utils.find_outliers(*args, **kwargs)[source]

find_outliers searches for outliers in Y as a function of X.

Inputs:
• X, array of points
• Y, array of points correlated to X
• bins, the number of bins to consider, default 5
• sig, a rejection criterion, default 4
Output:
• a list of the index of the good pixels
mascara.funcs.utils.robuststd(X)[source]

Calculates a robust standard deviation for the distribution X, by removing gradually the outliers. Usefull when a few outliers are skewing the calculation of the standard deviation.

Input:
• X, array-like
Output:
• the robust standard deviation of X
mascara.funcs.utils.percentile(im, value)[source]

Percentile computes the qth percentile of the data. It is based on the numpy version percentile, however without any interpolation of the numbers. And is faster than the scipy.scoreatpercentile() function.

Input:
• im, a 2d or 1d array
• value, a scalar for the percentile to calculate. Must be between 0 and 1.
Output:
• the value of the data at the percentile.
mascara.funcs.utils.gaussian(x, xo, w, A)[source]

Returns a gaussian function centered in xo, with a width w and an amplitude A.

Inputs:
• x, 1d array of the values used for computing the Gaussian
• xo, the centre of the gaussian
• w, the width of the gaussian
• A, the amplitude
Output:
• G(x) = A exp(-(x-xo)^2/(2W^2))
mascara.funcs.utils.rebin(image, binfactor)[source]

rebins an image to a new format. Condition: newformat is smaller than the original image format.

Inputs:
• image, an numpy array
• binfactor, the binning factor
Output:
• binimage
mascara.funcs.utils.statbin(image, binfactor)[source]

statbin returns the mean values and standard deviation of the values inside a bin.

Inputs:
• image, a numpy array, 1D
• binfactor, the binning factor. Has to be smaller than the image size...
Outputs:
• the mean values of the bins
• the std values of the bins

loadStellarCatalog load in memory a binary fits table catalog of stellar names and coordinates for further use (for instance predict the visible stars from an observing site).

Keywords:
• filename: the fits file containing the stellar catalog. By default it’s a vizier catalog containing only stellar coordinates. But any other fits.catalog could be loaded.
• Vmag, Rmag, Bmag or Jmag, the magnitude range of the stars to load. Default is set from Vmag= 2 to 8, to avoid using too much memory. But any fainter number is fine too.
• sID = ‘ASCC’, ‘TYC’, ‘HIP’ or ‘HD’, the stellar identification
• sIDonly, boelan to return only the stellar identifiers.
Outputs:
• ra, an array of right ascension coordinates, in degree, and in J2000
• dec, an array of declination
• vmag, the corresponding stellar visible magnitude
• bmag, the B magnitude
• sID, the stellar Identifiers
• stype, the spectral type
mascara.funcs.utils.cleanupStellarCatalog(ra, dec, mag, dist=2.5)[source]

Astrostars is a short routine for selecting stars within a catalog that are suited for astrometry. They are bright (limiting magnitude defined via mag1), and have no close neighbours (define close via dist) of magnitude down to mag2.

Inputs:
• ra, dec, the stellar coordinates
• mag, the corresponding stellar magnitudes. Used for rejected the fainter stars
Keyword:
• dist = 2.5, the distance in arcmin between 2 stars for considering them blended
Outputs:
• ra, dec mag of the selected stars

skyQuad does nothing with the image itself. Creates tiles for further sky quality analysis.

Inputs:
• the image itself, to draw out the image dimensions
Keywords
• tilesize, the size of the tiles.
• aligned = True. If True the tiles follow a classical grid pattern. Else the tiles are shifted by half a tile for each row.
Outputs:
• lx, ly, upx, upy, the corners of the sky Quads.
mascara.funcs.utils.pol2rect(ra, dec, degree=True)[source]

Convert any type of spherical coordinates in a matrix of cartesian rectangular coordinates to be used for coordinates conversion.

Example: To convert (ra-dec)(J2000) to (ra-dec)(to date):

>>> X = pol2rect(ra, dec)
>>> Xnew = N*P*X   # where N and P are the nutation and precession
matrices.
>>> ranew, decnew = rec2pol(Xnew)

Inputs:
• the 2 polar coordinates (ra,dec) - (alt-az) (phi theta) in DEGREE
Output:
-a matrix of cartesian coordinates
mascara.funcs.utils.rect2pol(X, Eq2AltAz=False)[source]

Conversion from rectangular coordinates to equatorial. Special care has to be taken around the celestial poles

Inputs:
• x, y, z, scalar or vector of rectangular coordinates
Outputs:
• ra, dec: the converted coordinates in degrees
mascara.funcs.utils.rotmat(angle, axis='x', degree=False)[source]

Rotmat generates a 3x3 rotation matrix for a given angle around the requested axis (x, y, or z)

Input:
• angle, the rotation angle
Keywords:
• the axis around which the rotation takes place.
• degree=True if the input angles are in degree
Output:
• the rotation matrix
mascara.funcs.utils.makevortex(x, y)[source]

Makevortex uses a Delauney griding method to create from the input coordinates a triangular grid. Each star is matched with its closest neighboor to form a triangle. The method is used in the astrometric solution for identifying the stars together.

Inputs:
• x, y, 2 vectors of equal size representing the coordinates of the star.
Output:
• dist, a vector of the possible distances.
• index, a array containing the indices of the 2 points forming the distance.
mascara.funcs.utils.match_vortex(rxobs, ryobs, rmobs, rxpre, rypre, rmpre, dtol=5.0, mtol=0.12)[source]

match_vortex matches together observed vortices made of the measured stellar positions to the vortices made from the calculated stellar positions.

Inputs:
• distobs, a vector containing all the distances computed from the 3 to 6 points.
• distpre, a vector containing all the computed predicted distances
• indices, an indice vector for distpre (which points makes which distance)
• tol, the tolerance for matching the distance together.