New Developer Onboarding
-
Developer access for OPERA PGE GitHub Repository: https://github.com/nasa/opera-sds-pge
- Developers need a pre-existing public github account. This account name will be provided in the NAMS request form for access to our repo.
- Developers need to submit a NAMS request for the “AGCY NASA Github Collaborator Form”
- On the NAMS form, for sponsor use Group Supervisor name
- For TEAMS & REPOSITORIES provide the following:
-
Developer access to OPERA PGE Jenkins: https://opera-pge-ci.jpl.nasa.gov/
- Must be behind JPL firewall to access
- Request for read/write access must be made with OPERA sysadmin to have JPL account added
- Once added, use JPL creds to log in
-
Developer access to OPERA SDS Google Drive: https://drive.google.com/drive/u/0/folders/0APxbcZdC1sajUk9PVA
- Contains SDS-wide documentation, meeting notes, schedules, etc…
- Meeting notes and sprint planning details for PGE work also live here
- Not super critical for developers to have access, but its a nice to have
- Talk to your Group Supervisor about setting up the subscription for the Google suite. Once set up, developers need to request access to this drive using their Caltech email address.
- Contains SDS-wide documentation, meeting notes, schedules, etc…
-
Developer access to Artifactory-FN: https://artifactory-fn.jpl.nasa.gov/ui/packages#/home
- This is a stand-alone instance of Artifactory which we use to share artifacts with members of the team that are foreign nationals
- The Algorithm Development Team (ADT) use this version of Artifactory to deliver their software container images and documentation
- We also use it to deliver PGE software container images and documentation for use with PCM
- Our Jenkins deployment pipeline is configured to automatically stage built images here
- Read/write permissions to the site can be granted by OPERA SDS team members
-
OPERA ADT Github Repo: https://github.com/opera-adt
- This is the public repository for the SAS source code developed by ADT.
- Read access should be sufficient for OPERA PGE developers
If Python is not available on your local workstation, or it is not at least v3.9, download and install Python for the host OS from the following link: https://www.python.org/downloads/
If developing on OSX, installing the Homebrew Package Manager is highly recommended. Instructions for installing Homebrew can be found here: https://brew.sh/
Some of the container build scripts expect the GNU style of common shell utilities, as opposed to the OSX versions.
If developing on OSX, make sure you install the coreutils package prior to attempting a container build:
brew install coreutils
The first step in when setting up a local development environment should be to clone a copy of the repository
cd <Local repository directory>
git clone https://github.com/nasa/opera-sds-pge.git
Next, we want to install a Python Virtual Environment (venv), which will ensure that we can install library dependencies without conflicting with the system installation of Python.
The following commands will create a Virtual Environment named venv, and then activate it (making it the active Python installation within the current shell session):
cd <Local repository directory>/opera-sds-pge/
python3 -m venv venv
source venv/bin/activate
The Virtual Environment will be installed to a venv directory within the opera-sds-pge directory. We will use this location when importing the code into PyCharm, which is described below.
Finally, install the package and its dependencies for development into the Virtual Environment:
cd <Local repository directory>/opera-sds-pge/
pip install --editable '.[dev]'
pip install -r requirements.txt
Note: If the above command fails, add <Local repository directory>/opera-sds-pge/src/ to your PYTHONPATH environment variable and rerun.
For development of PGE code on a local workstation, we strongly recommend use of the PyCharm Python IDE, as it is free (for the Community Edition), available for Windows/OSX/Linux, and has a wealth of built-in functionality to simplify/streamline tasks such as code refactoring and unit test development.
The latest version of PyCharm Community Edition is available here: https://www.jetbrains.com/pycharm/download/
Upon opening PyCharm, you should see a Welcome window similar to the screenshot below. To import the cloned OPERA PGE repository as a PyCharm project, click the "New Project" button in the upper right corner:
From the New Project menu, enter the path to the cloned copy of the OPERA PGE Repository in the Location field:
Next, ensure the "Previously configured interpreter" radio box is checked, then click the "Add Interpreter" link to the right of the field:
This will open the "Add Python Interpreter" menu. This is where we will instruct PyCharm to utilize the Python executable within the Virtual Environment which was installed above. Make sure "Virtualenv Environment" is selected from the sub-menu on the left side of the window, and then check the "Existing" radio button. Next, click the ellipses button (highlighted in red) to open a file system browser, and use it to browse to the location of the Python exe within the Virtual Environment. This should be <Local repository directory>/opera-sds-pge/venv/bin/python3
Click "OK" on the Add Python Interpreter menu, then click "Create" on the New Project window. You should see a pop-up dialog box stating that the selected directory is not empty. Click the "Create from Existing Sources" button to continue:
The main IDE window should now show the opera-sds-pge project displayed in the Project browser on the left side of the window. From here you can navigate the available source files and double click them to open in the editor window.
To ensure the project has been imported correctly, open the source file requirements.txt within the top-level of the Project. Once opened in an editor window, PyCharm will index the file and determine if there are any missing dependencies which should be installed. If you see any of the package names highlighted in yellow, it means that PyCharm could not find a local installation of the required package. If you hover your cursor over the name of the missing package, a tool window should pop up with an option to have PyCharm download and install the required package for you:
Repeat this for all missing package dependencies as needed.
After installing dependencies, the last step is to configure the Project such that PyCharm is aware of the directory containing our source code when it performs indexing. To do so, open the PyCharm settings menu (pictured for OSX):
From the Settings menu, on the left-hand side, expand the submenu named "Project: opera-sds-pge", then select "Project Structure" from the sublist. Within the main Project Structure menu, select the src directory of the Repository structure, and then click the light-blue "Sources" button above to mark the src directory as containing source code to be indexed. Next, highlight the venv directory, and then click the orange "Excluded" button to mark the directory as excluded from indexing. When finished, your menu should look like the following screenshot:
Click "OK" in the lower right corner to accept the changes. The project should now be ready for development.
The primary development machine for OPERA PGE work is "opera-dev-pge", which can be logged into using JPL credentials when on the JPL VPN.
The following will outline the steps to create a personal development environment suitable for performing tasks such as Acceptance Testing or running the Integration Test scripts outside of Jenkins.
The first step should be to install miniconda to provide a light-weight, isolated Python environment where developers can install their own packages without affecting the system installation of Python.
The following commands can be used to download the miniconda installer for Linux to a developer's home directory:
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
After downloading the installer script, install miniconda within your home directory:
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
With miniconda installed, adding packages to your local Python environment is as simple as running conda install. Install the following packages to your environment to get started:
conda install numpy matplotlib pandas gdal
See the documentation for the conda application for more info on updating the local environment. For packages that are unavailable from conda, pip can be used and will install automatically to the local miniconda environment.
The next step is to acquire a local clone of the OPERA PGE repository, from which PGE container images can be built and tested, or Integration Tests may be kicked off from outside of Jenkins.
Within the developers home directory, create a directory named repos and navigate into it:
mkdir ~/repos
cd ~/repos
Run the following command to clone a copy of the OPERA PGE repository into the repos directory:
git clone https://github.com/nasa/opera-sds-pge.git
Add the following line to your .bash_profile or .bashrc (or whatever init script is used by your preferred shell)
PYTHONPATH=/home/<username>/repos/opera-sds-pge/src:$PYTHONPATH
export PYTHONPATH
Reload your shell to incorporate the new changes. You should now be able to run the following from an interactive Python console to verify that the repo is installed as expected:
$ python
Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import opera
While we've been working so far within the developers' home directory under /home/<username>, this is not the best place to actually execute PGEs out of, due to the limited amount of disk space on the partition which mounts /home. Rather, developers are expected to work under the /data partition, which typically has much more disk space available.
Within the /data directory on opera-dev-pge, there should already be a directory named for the developer's user name available for use. If not, a sysadmin request needs to be made to have it created. This /data/<username> directory can then be tailored as the developer sees fit to execute SAS or PGE images.
During regular PGE development tasks, it is often necessary to either push or pull docker images or sample data to/from Artifactory-FN (described above). It is usually much much more efficient to download or upload artifacts directly between opera-dev-pge and the Artifactory server, rather than from a developer's local machine. To be able to use curl to interact with Artifactory, we first need to generate an API key that will used to authenticate requests.
After logging into the Artifactory-FN web console, click the "Welcome, " link in the top-right of the page, and then click on "Edit profile". At this point, you will need to enter your password again and hit the "Unlock" button to make changes to this page.
After unlocking the page, there should be an "API Key" section towards the middle of the page where a new API Key can be generated, or an existing key can be regenerated. After generating the API Key, copy the key value to a trusted location for future reference.
With the API Key now available, the following command can be used to download an artifact to opera-dev-pge:
curl -H "X-JFrog-Art-Api:<API_KEY>" -O "https://artifactory-fn.jpl.nasa.gov/artifactory/<TARGET_FILE_PATH>"
Where <API_KEY> is the key value copied earlier, and <TARGET_FILE_PATH> is the Artifactory path to the file to download, such as general/gov/nasa/jpl/opera/adt/dswx_s1/r1/interface/dockerimg_dswx_s1_interface_0.1.tar
To push an artifact to Artifactory for storage:
curl -H "X-JFrog-Art-Api:<API_KEY>" -T <PATH_TO_FILE> "https://artifactory-fn.jpl.nasa.gov/artifactory/<TARGET_FILE_PATH>"
Where <PATH_TO_FILE> is the local file path on opera-dev-pge, and <TARGET_FILE_PATH> is where the file should be stored on Artifactory. For PGE developers, the only location that should ever be uploaded will be somewhere under general/gov/nasa/jpl/opera/sds/pge/
- Article describing the "Mixin" design pattern leveraged heavily by the OPERA PGE architecture