Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Input data overhaul #184

Merged
merged 37 commits into from
Jul 16, 2020
Merged

Input data overhaul #184

merged 37 commits into from
Jul 16, 2020

Conversation

scotthavens
Copy link
Contributor

@scotthavens scotthavens commented Jul 15, 2020 •
edited
Loading

A big issue with SMRF and gridded data is that it must loaded all upfront which does not make sense for a year long run. Refactored the data loading to be more organized and have an easier call in SMRF.

  • The HRRR data can be loaded either upfront or each timestep through a new config option
  • When running threaded, the data is loaded into a data queue that the threads get from
  • All gridded data is used in the same way for thermal radiation, meaning that the clear sky thermal is modeled then adjusted for cloud factor and not distributing thermal radiation from the gridded dataset.
  • Remove MySQL for MySQL data deprecation? #127
  • Removed the last_storm_basin metric what calculated the time since last storm for the whole basin, this is not being used anymore

scotthavens added 30 commits July 6, 2020 10:01
@scotthavens
moved csv to it's own class
d28af3f
@scotthavens
added wrf loading
498fb1e
@scotthavens
netcdf loading
75f463a
@scotthavens
hrrr grib loading
086c37d
@scotthavens
some cleanup
d4089b9
@scotthavens
organized the tests to exapand testing of data
739ece2
@scotthavens
initial hrrr timestep working with the non threaded run
df20f44
@scotthavens
loading of hrrr working for non threaded run
fa91111
@scotthavens
data queue that loads all data for the threads to access
c249d72
@scotthavens
working for now with hrrr loading as a thread
a802564
@scotthavens
logging in tests
df7d460
@scotthavens
Merge branch '18_threading' into gridded_data
b845bb4
@scotthavens
needed the updated threading scheme for the hrrr data
1220ff2
@scotthavens
Topo class and some logging leftovers
394c3e4
@scotthavens
added a hrrr cloud factor interpolater for a single time step
c39f9dc
@scotthavens
flake8 and isort
0390815
@scotthavens
order import for data
ffd1ca8
@scotthavens
performing a test on tuolumne and had to make some fixes
851a2a7
@scotthavens
default clear sky if first is night time
23266f0
@scotthavens
added an argument for the create-distributed_threads for awsm
9090a48
@scotthavens
Merge remote-tracking branch 'upstream/master' into gridded_data
c611013
@scotthavens
messed up on the merge conflicts
750455c
@scotthavens
Merge remote-tracking branch 'upstream/master' into gridded_data
5b185d8
@scotthavens
some last cleanup from the merge
ed9616c
@scotthavens
renamed the data classes, started to consolidate the threads
f019a48
@scotthavens
streamlined the thread creation as all need the smrf and data queues,…
8c261f6 
… last_storm_day_basin wasn't used anywhere so removed all metion of it
@scotthavens
moved initializeDistribution to create_distribution (going to break A…
2152cd9 
…WSM), DRYing out code
@scotthavens
cleaned up wrf, all gridded datasets are treated the same
787c851
@scotthavens
Merge remote-tracking branch 'upstream/master' into gridded_data
b06eb00
@scotthavens
updating for new test structure
b63d531
scotthavens
scotthavens commented Jul 15, 2020
View reviewed changes
docs/getting_started/run_smrf.rst
@@ -30,7 +30,7 @@ below is the function :mod:`run_smrf <smrf.framework.model_framework.run_smrf>`.
s.loadTopo()

# initialize the distribution
s.initializeDistribution()
s.create_distribution()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will break AWSM. Should be included in AWSM #68

smrf/data/load_topo.py
@@ -104,18 +104,11 @@ def readImages(self, f):
# to double or int
for v_smrf in self.IMAGES:

# check to see if the user defined any variables
# e.g. veg_height = veg_length
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has never been used where one can specify different field names in the topo.nc. Most will be generated with basin_setup so it's just making it a little more rigid.

@scotthavens scotthavens requested a review from jomey July 15, 2020 16:41
@scotthavens
Copy link
Contributor Author

@etrujil and @Hedrick-ARS, here is a good overhaul to the input data for SMRF to make it more efficient. This also has removed the MySQL capability since we haven't used it in a few years.

jomey
jomey requested changes Jul 15, 2020
View reviewed changes
docs/user_guide/input_data.rst Outdated Show resolved Hide resolved
requirements.txt Show resolved Hide resolved
smrf/data/csv.py Outdated Show resolved Hide resolved
smrf/data/csv.py Show resolved Hide resolved
smrf/data/hrrr_grib.py Outdated Show resolved Hide resolved
smrf/distribute/air_temp.py Show resolved Hide resolved
smrf/distribute/solar.py Show resolved Hide resolved
smrf/tests/data/test_wrf.py Show resolved Hide resolved
smrf/tests/data/test_netcdf.py Show resolved Hide resolved
smrf/utils/utils.py Outdated Show resolved Hide resolved
@jomey
Copy link
Collaborator

jomey commented Jul 15, 2020

Some of my comments apply to multiple data loading classes and I did not repeat the message there. Easier to discuss suggestions in one place.

@scotthavens
Copy link
Contributor Author

Some of my comments apply to multiple data loading classes and I did not repeat the message there. Easier to discuss suggestions in one place.

Got it, a lot of the classes are similar so I've applied the comments to all classes that were applicable.

@scotthavens scotthavens requested a review from jomey July 15, 2020 22:12
@jomey jomey mentioned this pull request Jul 15, 2020
Open
jomey
jomey reviewed Jul 15, 2020
View reviewed changes
smrf/data/load_data.py Outdated Show resolved Hide resolved
@scotthavens scotthavens mentioned this pull request Jul 15, 2020
Closed
jomey
jomey reviewed Jul 15, 2020
View reviewed changes
smrf/data/load_data.py Outdated Show resolved Hide resolved
@scotthavens scotthavens mentioned this pull request Jul 15, 2020
Closed
@scotthavens scotthavens merged commit 8b65889 into USDA-ARS-NWRC:master Jul 16, 2020
@scotthavens scotthavens deleted the gridded_data branch July 16, 2020 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jomey jomey jomey approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@scotthavens @jomey