Project Description

The USGS Astrogeology Science Center develops and maintains a software toolkit for working with planetary images called the Integrated Software for Imagers and Spectrometers (ISIS). The goal of this project is to take the existing image processing pipeline (Pilot) and break down the separate processing steps into separate AirFlow DAG processes to allow users full control over the steps that are run in their image processing. The web application will provide a graphical drag-n-drop interface in which users can select desired processing steps, graphically arrange them into the desired processing workflow, then edit/configure each step with various parameters

The original proposal for this project can be found here.

Codebase

The code our project is based on can be found here.

High Level Requirements

Our project will be focused on improving the existing Pilot program by allowing users to do the following:

Specify the files to be processed either through a file path or from a CSV download of a Pilot image search result.
Select which of the processing steps in the POW pipeline to apply to the list of files.
Specify the processing options for each step such as what projection to use and what file format to convert the finished products to.
Specify either an output directory (for USGS Astrogeology Science Center team members) or to have the finished files to be packaged for download using the standard POW distribution.

Architecture

This image represents an early estimate of the general structure of the pipeline we we have constructed. A more detailed description of our pipeline's architecture can be found on our solution page.

Technologies

The new technologies implemented in our project have been decided on by our client

Airflow: a Python library that allow allows users to create and manage workflows in the form of directed acyclic graphs, or DAGs. These DAGs provide a programmatic approach to workflow design as they are layed out entirely in Python.
Docker: a containerization software that allows for a more efficient way to run tasks such as Airflow DAGs. The containers that Docker creates are akin to lightweight virtual machines, requiring far less resources than a normal vm. However, like other vms, Docker containers are based on pre-built images. These images provide our implementation a great deal or portability.