Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requested Changes to Spectral Library Version 7 issue #141

Closed
wants to merge 7 commits into from
Closed

Requested Changes to Spectral Library Version 7 issue #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d9f451c
splib07
Mar 8, 2018
a9bb2d4
Merge pull request #1 from capstone-coal/master
Mar 15, 2018
840ff4b
spectral 7 converter start
Mar 15, 2018
b5505a6
Splib 07 test one file
Mar 17, 2018
7b10aa9
Finished Spectral Version 7 convert
Mar 21, 2018
0052322
Finish client changes to spectral convert function
Mar 28, 2018
0db1578
Condensed to one script
Mar 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions examples/convert_to_aster.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
#Copyright (C) 2017-2018 COAL Developers
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bdegley4789 there are a few major issues with the code in this file

  1. The documentation does not mention that you need to have the file ./ASCIIdata/ASCIIdata_splib07a/ChapterA_ArtificialMaterials/splib07a_Alizarin_crimson_(dk)_GDS780_ASDFRa_AREF.txt present in the current working directory in order for the conversion to work correctly... this would need to be corrected.
  2. The logic is intrinsically linked to the man made compound Alizarin_crimson... which is a paint pigment. There is absolutely no rational justification for us having an example of a conversion and spectral classification based upon a paint pigment as it has nothing to do with the COAL project. We are looking for (at a bare minimum) examples based on elements associated with COAL and surface mining activities e.g. Renyolds_Tnl_Sludge, Renyolds_TnlSldgWet and Schwertmannite. Make sure you ensure that the examples are related to the above. These proxy classes are extracted directly from mining.py. I've referenced these several times in the past so please take the guidance on board. Thanks.
  3. The code in this file is a complete copy of the proposed code you've implemented in pycoal/mineral.py... why are you duplicating the code? You should just be using the SpectralToAsterConversion.convert(...) function you created in pycoal/mineral.py.

#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; version 2.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty
# of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth
# Floor, Boston, MA 02110-1301, USA.
# encoding: utf-8

'''
convert_to_aster.py -- an example script to convert a USGS Spectral Library Version 7 file
to match the format of an ASTER Library Version 2.0 file

This class method converts a USGS Spectral version 7
<https://speclab.cr.usgs.gov/spectral-lib.html> .txt file into
an ASTER Library Version 2.0 <https://asterweb.jpl.nasa.gov/> .spectrum.txt file
ASTER Library Version 2.0 Spectral Library files are in .spectrum.txt file format

@author: COAL Developers

@copyright: 2017-2018 COAL Developers. All rights reserved.

@license: GNU General Public License version 2

@contact: [email protected]
'''

import os

#This will take the .txt files for Spectra in USGS Spectral Version 7 and convert
#their format to match that of ASTER .spectrum.txt files for spectra
library_filename = 'splib07a_Alizarin_crimson_(dk)_GDS780_ASDFRa_AREF.txt'
#Count amount of lines in library_filename
with open(library_filename,'r') as input_file:
for line_count, l in enumerate(input_file):
pass

input_file = open(library_filename,'r')
#Read Name of Spectra on first line of the file
spectra_line = input_file.readline()
spectra_name = spectra_line[23:]
k = 0
#Loop through file and store all wavelength values for the given Spectra
spectra_values_file = open('SpectraValues.txt','w')
while(k < line_count):
spectra_wave_length = float(input_file.readline()) * 100
spectra_wave_length = spectra_wave_length / 1000
spectra_y_value = spectra_wave_length * 10
line = str(spectra_wave_length) + ' ' + str(spectra_y_value)
spectra_values_file.write(line)
spectra_values_file.write('\n')
k = k+1
#Write new file in the form of an ASTER .spectrum.txt file while using stored
#Spectra Name and stored Spectra Wavelength values`
input_file = open(library_filename,'w')
input_file.write('Name:')
input_file.write(spectra_name)
input_file.write('Type:\n')
input_file.write('Class:\n')
input_file.write('Subclass:\n')
input_file.write('Particle Size: Solid\n')
input_file.write('Sample No.: 0095UUUASP\n')
input_file.write('Owner:\n')
input_file.write('Wavelength Range: ALL\n')
input_file.write('Origin: Spectra obtained from the Noncoventional Exploitation Factors\n')
input_file.write('Data System of the National Photographic Interpretation Center.\n')
input_file.write('Description: Gray and black construction asphalt. The sample was\n')
input_file.write('soiled and weathered, with some limestone and quartz aggregate\n')
input_file.write('showing.\n')
input_file.write('\n')
input_file.write('\n')
input_file.write('\n')
input_file.write('Measurement: Directional (10 Degree) Hemispherical Reflectance\n')
input_file.write('First Column: X\n')
input_file.write('Second Column: Y\n')
input_file.write('X Units: Wavelength (micrometers)\n')
input_file.write('Y Units: Reflectance (percent)\n')
input_file.write('First X Value:\n')
input_file.write('Last X Value:\n')
input_file.write('Number of X Values:\n')
input_file.write('Additional Information:\n')
input_file.write('\n')
j = 0
spectra_values_file.close()
#Read in values saved in SpectraValues.txt and output them to the library_filename
spectra_values_file = open('SpectraValues.txt','r')
while(j < line_count):
spectra_wave_length = spectra_values_file.readline()
input_file.write(spectra_wave_length)
j = j+1
#Close all open files
input_file.close()
spectra_values_file.close()
#Rename library_filename to match ASTER .spectrum.txt file format
os.rename(library_filename,library_filename + '.spectrum.txt')
#Remove temporary file for storing wavelength data
os.remove('SpectraValues.txt')
print("Successfully converted file " + library_filename)
102 changes: 102 additions & 0 deletions examples/splib07_convolved.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#Copyright (C) 2017-2018 COAL Developers
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; version 2.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty
# of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth
# Floor, Boston, MA 02110-1301, USA.
# encoding: utf-8

'''
splib07_convolved -- a script which will generate envi .sli and .hdr convolved library
files of USGS Spectral Library Version 7 <https://speclab.cr.usgs.gov/spectral-lib.html>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make this a hyperlink


Dependencies
USGS Spectral Library Version 7 <https://speclab.cr.usgs.gov/spectral-lib.html>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this a hyperink

must be downloaded to the examples directory

This code has three parts.
First, it loops through the USGS Spectral Library version 7
<https://speclab.cr.usgs.gov/spectral-lib.html> and moves all spectra files to a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this a hyperlink

modified directory.
Second, it converts all USGS Spectral Library Version 7 .txt files into
the ASTER Library Version 2.0 <https://asterweb.jpl.nasa.gov/> .spectrum.txt file format using
the SpectralToAsterConversion class in pycoal mineral.
Third, it creates a .db, .sli and .hdr envi files for the convolved spectral library using
the AsterConversion class in pycoal mineral.

All files generated will be located in the examples directory


@author: COAL Developers

@copyright: 2018 COAL Developers. All rights reserved.

@license: GNU General Public License version 2

@contact: [email protected]
'''

#!/usr/bin/python

import os
import sys
from sys import path
sys.path.insert(0, '../pycoal')
reload(sys)
sys.setdefaultencoding('utf8')
import fnmatch
import shutil
import mineral
import math
import numpy
import spectral

#This will take all the necessary .txt files for spectra in USGS
#Spectral Library Version 7 and put them in a new directory called
#"usgs_splib07_modified" in the examples directory
directory = 'usgs_splib07_modified'
if not os.path.exists(directory):
os.makedirs(directory)

exclude = set(['usgs_splib07_modified'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be looking only in the ./ASCIIdata/ASCIIdata_* wildcard directories. It is up to you how you wish to implement this.

for root, dir, files in os.walk("."):
dir[:] = [d for d in dir if d not in exclude]
for items in fnmatch.filter(files, "*.txt"):
if "Bandpass" not in items:
if "errorbars" not in items:
if "Wave" not in items:
if "SpectraValues" not in items:
shutil.copy2(os.path.join(root,items), directory)

#This will take the .txt files for Spectra in USGS Spectral Version 7 and
#convert their format to match that of ASTER .spectrum.txt files for spectra
library_filename = 'usgs_splib07_modified/'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've already defined the exact same string value on line 66 however this one has an appended /

# create a new mineral aster conversion instance
spectral_aster = mineral.SpectralToAsterConversion()
# Convert all files
files = os.listdir(library_filename)
for x in range(0, len(files)):
name = 'usgs_splib07_modified/' + files[x]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not reuse one of the string variables you've created above?

spectral_aster.convert(name)

#This will generate three files s07av95a_envi.hdr, s07av95a_envi.hdr.sli,splib.db and dataSplib07.db
#For a library in ASTER Library Version 2.0 <https://asterweb.jpl.nasa.gov/> format
library_filename = 'usgs_splib07_modified'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why create yet another sting variable with the same value...?

data_dir = "dataSplib07.db"
header_name = "s07av95a_envi"

# create a new mineral aster conversion instance
spectral_envi = mineral.AsterConversion()
# Generate .sli and .hdr
spectral_envi.convert(library_filename,data_dir,header_name)


105 changes: 102 additions & 3 deletions pycoal/mineral.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
# Software Foundation, Inc., 51 Franklin Street, Fifth
# Floor, Boston, MA 02110-1301, USA.

import sys
import os
import inspect
import logging
import math
import numpy
Expand All @@ -28,7 +31,7 @@ def __init__(self, library_file_name, class_names=None, threshold=0.0, in_memory
Construct a new ``MineralClassification`` object with a spectral library
in ENVI format such as the `USGS Digital Spectral Library 06
<https://speclab.cr.usgs.gov/spectral.lib06/>`_ or the `ASTER Spectral
Library Version 2.0 <https://asterweb.jpl.nasa.gov/`_ converted with
Library Version 2.0 <https://asterweb.jpl.nasa.gov/>`_ converted with
``pycoal.mineral.AsterConversion.convert()``.

If provided, the optional class name parameter will initialize the
Expand Down Expand Up @@ -317,8 +320,8 @@ class AsterConversion:

def __init__(self):
"""
This class provides a method for converting the `ASTER Spectral
Library Version 2.0 <https://asterweb.jpl.nasa.gov/>`_ into ENVI format.
This class provides a method for converting the ASTER Spectral
Library Version 2.0 <https://asterweb.jpl.nasa.gov/> into ENVI format.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the hyperlink? revert this...


Args:
None
Expand Down Expand Up @@ -362,3 +365,99 @@ def convert(cls, data_dir="", db_file="", hdr_file=""):
library = aster_database.create_envi_spectral_library(spectrum_ids, band_info)

library.save(hdr_file)

class SpectralToAsterConversion:

def __init__(self):
"""
This class provides a method for converting USGS Spectral Library Version 7
<https://speclab.cr.usgs.gov/spectral-lib.html> .txt files into ASTER Spectral
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include hyperlink

Library Version 2.0 <https://asterweb.jpl.nasa.gov/> .txt files

Args:
none
"""
pass

@classmethod
def convert(cls, library_filename=""):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is duplicate as that from examples/convert_to_aster.py. If also suffers from the same issues I've pointed out in my review of that file... correct and resubmit a PR.

"""
This class method converts a USGS Spectral Library Version 7
<https://speclab.cr.usgs.gov/spectral-lib.html> .txt file into
an ASTER Library Version 2.0 <https://asterweb.jpl.nasa.gov/> .spectrum.txt file
ASTER Library Version 2.0 Spectral Library files are in .spectrum.txt file format

Spectral Library Version 7 can be downloaded here <https://speclab.cr.usgs.gov/spectral-lib.html>

Args:
library_filename (str): path to Spectral File you wish to convert
"""
if not library_filename:
raise ValueError("Must provide path for Spectral File.")

line_count = 1
with open(library_filename,'r') as input_file:
for line_count, l in enumerate(input_file):
pass

input_file = open(library_filename,'r')
#Read Name of Spectra on first line of the file
spectra_line = input_file.readline()
spectra_name = spectra_line[23:]
k = 0
#Loop through file and store all wavelength values for the given Spectra
spectra_values_file = open('SpectraValues.txt','w')
while(k < line_count):
spectra_wave_length = float(input_file.readline()) * 100
spectra_wave_length = spectra_wave_length / 1000
spectra_y_value = spectra_wave_length * 10
line = str(spectra_wave_length) + ' ' + str(spectra_y_value)
spectra_values_file.write(line)
spectra_values_file.write('\n')
k = k+1
#Write new file in the form of an ASTER .spectrum.txt file while using stored
#Spectra Name and stored Spectra Wavelength values`
input_file = open(library_filename,'w')
input_file.write('Name:')
input_file.write(spectra_name)
input_file.write('Type:\n')
input_file.write('Class:\n')
input_file.write('Subclass:\n')
input_file.write('Particle Size: Solid\n')
input_file.write('Sample No.: 0095UUUASP\n')
input_file.write('Owner:\n')
input_file.write('Wavelength Range: ALL\n')
input_file.write('Origin: Spectra obtained from the Noncoventional Exploitation Factors\n')
input_file.write('Data System of the National Photographic Interpretation Center.\n')
input_file.write('Description: Gray and black construction asphalt. The sample was\n')
input_file.write('soiled and weathered, with some limestone and quartz aggregate\n')
input_file.write('showing.\n')
input_file.write('\n')
input_file.write('\n')
input_file.write('\n')
input_file.write('Measurement: Directional (10 Degree) Hemispherical Reflectance\n')
input_file.write('First Column: X\n')
input_file.write('Second Column: Y\n')
input_file.write('X Units: Wavelength (micrometers)\n')
input_file.write('Y Units: Reflectance (percent)\n')
input_file.write('First X Value:\n')
input_file.write('Last X Value:\n')
input_file.write('Number of X Values:\n')
input_file.write('Additional Information:\n')
input_file.write('\n')
j = 0
spectra_values_file.close()
#Read in values saved in SpectraValues.txt and output them to the library_filename
spectra_values_file = open('SpectraValues.txt','r')
while(j < line_count):
spectra_wave_length = spectra_values_file.readline()
input_file.write(spectra_wave_length)
j = j+1
#Close all open files
input_file.close()
spectra_values_file.close()
#Rename library_filename to match ASTER .spectrum.txt file format
os.rename(library_filename,library_filename + '.spectrum.txt')
#Remove temporary file for storing wavelength data
os.remove('SpectraValues.txt')
print("Successfully converted file " + library_filename)