Loading Data
The Flexible Image Transport System (FITS) is the standard filetype used for storing data taken from a telescope. There are several Python packages that exist to help load FITS files into a Python data structure. However, the structure of a FITS file can vary significantly between telescopes, making it difficult to automate data preparation. Since we know what type of data we are processing, we can wrap a standard FITS loader with some additional logic to make this initial step slightly easier!
In this section, we show how specsuite can load / average several FITS files simultaneously. Unless you are processing data from one of the "supported" instruments (SBO, KOSMOS, or GMOS-N), you may also need to do a little extra work to prepare your data for reduction.
Note
We are working on creating specific data formatting functions for commonly-used telescopes with long-slit spectrographs. Although we have a "default" loader, this can fail to account for the niche data layout of some telescopes.
Warning
Currently, the following functions can only load FITS files. Some observatories will save data in a compressed format (such as GZIP compressed files). If you would like to use these loading functions, please unzip your data beforehand.
Basic Usage¶
Across all documentation, we will be using the sample data location in the "data/" directory on the specsuite repository. Although we recommend splitting up your exopsures into separate "calibration/" and "target/" folders, this is not strictly necessary for our loading functions to work. For reference, here is the structure of our data...
📂KOSMOS
|
└──📂calibrations
│ │ 📜bias.0001.fits
│ │ 📜bias.0002.fits
│ │ 📜bias.0003.fits
│ │ 📜flatfield.0004.fits
│ │ 📜flatfield.0005.fits
│ │ ⋮
│
└──📂target
│ 📜toi3884.0006.fits
│ 📜toi3884.0007.fits
│ 📜toi3884.0008.fits
│ 📜toi3884.0009.fits
│ 📜toi3884.0010.fits
│ ⋮
Loading Multiple Images¶
There are many types of exposures you may wish to load into Python as separate Numpy arrays. Using collect_images_array(), you can do just that! This function requires a directory to search for FITS files and a 'tag' to look for in the filenames contained within that directory. So, if we wanted to separately load in each "toi3884" exposure from our data directory...
import specsuite as ss
# Specifies where to look for files / how to load them
FILEPATH = "../data/KOSMOS/target"
TAG = "toi3884"
INSTRUMENT = "kosmos"
# Loads an array of data images
data = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = INSTRUMENT,
)
# Plots the first data image
ss.plot_image(data[0], norm='log', vmin=2e3, vmax=4e3)
This function returns a 3D Numpy array where the first dimension represents the file index. So, data[0] represents a 2D image corresponding to the first loaded FITS file. If you want to check which files were loaded, you can simply enable the debug option...
_ = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = INSTRUMENT,
debug = True,
)
Searching for files with 'toi3884' tag... ------------------------------------------ ✓ toi3884.0033.fits ✓ toi3884.0034.fits ✓ toi3884.0035.fits ✓ toi3884.0036.fits ✓ toi3884.0037.fits
You can see that a list of files were printed out with little checkmarks (✓) next to them. This shows a list of every FITS file in the specified directory containing the provided tag. This check means that specsuite was able to open the FITS file and that it was able to correct all minor formatting errors. If a file fails to load, an 'X' icon will be printed along with the associated error.
You may also want to exclude specific files in your directory. To skip specific filenames, use the ignore argument...
_ = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = INSTRUMENT,
ignore = ["toi3884.0034.fits", "toi3884.0036.fits"],
debug = True,
)
Searching for files with 'toi3884' tag... ------------------------------------------ ✓ toi3884.0033.fits ✓ toi3884.0035.fits ✓ toi3884.0037.fits
specsuite aligns file so the x- and y-axes represent the wavelength and spatial axes, respectively. If you only want to load a small portion of the image along the spatial axis, you can specify that using the crop_bds argument...
# Specifies which rows to keep from an image
DATA_REGION = (700, 800)
# Loads an array of data images
data = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = INSTRUMENT,
crop_bds = DATA_REGION,
)
# Plots the first data exposure
ss.plot_image(data[0], norm='log', vmin=2e3, vmax=4e3)
Averaging Multiple Images¶
A common practice for calibration images (i.e., biases, darks, flats, etc.) is the take the average of multiple exposures. This is done to minimize the impact of random, statistical fluctuations on your data calibration. To load the average of several files, you can use average_matching_files() with the same arguments used above...
FILEPATH = "../data/KOSMOS/calibrations"
bias = ss.average_matching_files(
path = FILEPATH,
tag = "bias",
instrument = INSTRUMENT,
)
arclamp = ss.average_matching_files(
path = FILEPATH,
tag = "neon",
instrument = INSTRUMENT,
)
flat = ss.average_matching_files(
path = FILEPATH,
tag = "flat",
instrument = INSTRUMENT,
)
Note that you can specify a sub-region to keep just like in collect_images_array().
Loading FITS Metadata¶
FITS headers often contain useful metadata (i.e., exposure time, telescope name, airmass, etc.). While we cannot predict which exact keywords are used in a given file, we create a dictionary filled with all of the key-value pairs found within the FITS header. This dictionary is automatically generated while loading data using collect_images_array() or average_matching_files() and can be returned using the return_metadata argument...
FILEPATH = "../data/KOSMOS/target"
data, metadata = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = INSTRUMENT,
return_metadata = True,
)
If the value is identical between all images, then the dictionary value will just contain a single value...
print("Exposure Time:", metadata["EXPTIME"])
print("Instrument Name:", metadata["INSTRUME"])
Exposure Time: 66.0 Instrument Name: kosmos
If each exposure contains unique data, though, the the metadata dictionary will contain a list of all values. The order of each entry should align with the order that files were loaded in...
print("Airmass:", metadata["AIRMASS"])
Airmass: [2.12510104 2.10341395 2.08106647 2.06040547 2.04021965]
Loading "Unsupported" Data¶
If you try to specify an instrument that does not have a pre-written formatting function, you will get a warning telling you that the 'default' loading procedure will be used. Let's take a look at what this means...
FILEPATH = "../data/KOSMOS/target"
data, metadata = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = "new instrument",
return_metadata = True,
)
UserWarning: 'new instrument' is not a supported instrument... - kosmos - gmos-hamamatsu - gmos-e2vDD Using the 'default' loading procedure!
The biggest difference is that data is a list of Numpy arrays and metadata is a list of dictionaries...
print(
f"'data' is a {type(data)} filled with {type(data[0])}\n"
)
print(
f"'metadata' is a {type(metadata)} filled with {type(metadata[0])}"
)
'data' is a <class 'list'> filled with <class 'numpy.ndarray'> 'metadata' is a <class 'list'> filled with <class 'dict'>
This is because observatories sometimes split data amonst multiple Header-Data Units (HDUs) in a single file. Each segment of data might have different sizes / metadata, so we cannot create a generalized function for stitching the data back together. Instead, data and metadata from each individual HDU is stored as an entry in the respective list. This means the length of the data and metadata lists is equal to the number of HDUs in your FITS files...
print(f"My FITS files have {len(data)} HDUs per file!")
My FITS files have 1 HDUs per file!
Since our example data only has one HDU, it is fairly easy to extract out the data / metadata...
new_data = data[0]
new_metadata = metadata[0]
However, if we load in some of our sample data from GMOS-N...
FILEPATH = "../data/GMOS/target"
data, metadata = ss.collect_images_array(
path = FILEPATH,
tag = TAG,
instrument = "new instrument",
return_metadata = True,
)
...and print out the length of data...
print(f"My FITS files have {len(data)} HDUs per file!")
My FITS files have 13 HDUs per file!
...we can see that there are a lot more entries to handle! In order for many of specsuite's tools to work, data must be a 3D Numpy array with the shape (N_files, wavelength pixels, spatial pixels) where wavelengths get longer from left to right. If you end up writing a function that can convert data and metadata into user-ready formats, feel free to reach out to us so we can include it in future releases of specsuite!
PATH = "../data/KOSMOS/calibrations"
bad_image = ss.average_matching_files(
path = PATH,
tag = "fl",
debug = True,
)
Searching for files with 'fl' tag... ------------------------------------------ ✓ flat.0029.fits ✓ flat.0030.fits ✓ flat.0031.fits ✓ flat.0032.fits ✓ flat.0033.fits ✓ fluxcal.0215.fits
Image statistics for average 'fl' image...
Min: 2035.1999999999998
Max: 156627.0
Mean: 71847.339
STD: 55256.913
If your files have unexpected features, using debug = True can help you quickly check which files are being loaded.
The Loaded Data is 'None'¶
If specsuite runs into an issue loading your data, it defaults to returning 'None'. These errors should be accompanied by a short diagnostic message, such as...
data = ss.average_matching_files(
path = "../data/KOSMOS/calibrations",
tag = "bad tag",
debug = True,
)
UserWarning: No files in '../data/KOSMOS/calibrations' with tag 'bad tag' were found...
As a first check, toggling the debug argument can help you verify whether the expected files are successfully loaded. If the correct files are listed but you are still getting 'None', there is likely an issue with a lower-level loading / formatting function, such as...
- FITS files have incompatible data shapes
- Header layouts change significantly between files
These types of issues are much harder to correct without adjusting specsuite's source code. If the printed error message does not identify a specific erroneous file, we recommend using the ignore argument to see if a small subset of files might be the culprit.