v0.4.3

v0.4.3 [DBR 13.3 LTS]

This release includes a number of enhancements and fixes, detailed below.

Raster checkpointing functions

Fuse-based checkpointing for raster operations is disabled by default but can be enabled and managed through:

• spark configs spark.databricks.labs.mosaic.raster.use.checkpoint and spark.databricks.labs.mosaic.raster.checkpoint.
• python: mos.enable_gdal(spark, with_checkpoint_path=path).
• scala: MosaicGDAL.enableGDALWithCheckpoint(spark, path).

This feature is designed to improve performance and reduce memory usage for raster operations by writing intermediate data to a fuse directory. This is particularly useful for large rasters or when working with many rasters in a single operation.

Enhancements and fixes to the raster processing APIs

• Added RST_Write, a function that permits writing each raster 'tile' in a DataFrame to a specified location (e.g. fuse directory) using the appropriate GDAL driver and tile data / path. This is useful for formalizing the path when writing a Lakehouse table and allows removal of interim checkpointed data.
• Python bindings added for RST_Avg, RST_Max, RST_Median, RST_Min, and RST_PixelCount.
• RST_PixelCount now supports optional 'countNoData' and 'countMask' parameters (defaults are false, can now be true) to optionally get full pixel counts where mask is 0.0 and noData is what is configured in the tile.
• RST_Clip now exposes the GDAL Warp option CUTLINE_ALL_TOUCHED which determines whether or not any given pixel is included whether the clipping geometry crosses the centre point of the pixel (false) or any part of the pixel (true). The default is true but this is now configurable.
• Within clipping operations such as RST_Clip we now correctly set the CRS in the generated Shapefile Feature Layer used for clipping. This means that the CRS of the input geometry will be respected when clipping rasters.
• Added two new functions for getting and upcasting the datatype of a raster band: RST_Type and RST_UpdateType. Use these for ensuring that the datatype of a raster is appropriate for the operations being performed, e.g. upcasting the types of integer-typed input rasters before performing raster algebra like NDVI calculations where the result needs to be a float.
• Added RST_AsFormat, a function that translates rasters between formats e.g. from NetCDF to GeoTIFF.
• The logic underpinning RST_MemSize (and related operations) has been updated to fall back to estimating based on the raster dimensions and data types of each band if the raster is held in-memory.
• RST_To_Overlapping_Tiles is renamed RST_ToOverlappingTiles. The original expression remains but is marked as deprecated.
• RST_WorldToRasterCoordY now returns the correct y value (was returning x)
• Docs added for expression RST_SetSRID.
• Docs updated for RST_FromContent to capture the optional 'driver' parameter.

Dependency management

Updates to and pinning of Python language and dependency versions:

• Pyspark requirement removed from python setup.cfg as it is supplied by DBR
• Python version limited to "<3.11,>=3.10" for DBR
• iPython dependency limited to "<8.11,>=7.4.2" for both DBR and keplergl-jupyter
• numpy now limited to "<2.0,>=1.21.5" to match DBR minimum

Surface mesh APIs

A set of experimental APIs for for creating and working with surface meshes (i.e. triangulated irregular networks) have been added to Mosaic. Users can now generate a conforming Delaunay triangulation over point data (optionally including 'break' lines as hard constraints), interpolate elevation over a regular grid and rasterize the results to produce terrain models.

• ST_Triangulate performs a conforming Delaunay triangulation using a set of mass points and break lines.
• ST_InterpolateElevation computes the interpolated elevations of a grid of points.
• RST_DTMFromGeoms burns the interpolated elevations into a raster.

British National Grid

Two fixes have been made to the British National Grid indexing system:

• Corrected a typo in the grid letter array used to perform lookups.
• Updated the logic used for identifying quadrants when these are specified in a grid reference

Documentation

A few updates to our documentation and examples library:

• An example walkthrough has been added for arbitrary GDAL Warp and Transform operations using a pyspark UDF (see the section "API Documentation / Rasterio + GDAL UDFs")
• The Python "Quickstart Notebook" has been updated to use the MosaicAnalyzer class (added after MosaicFrame was deprecated)

v0.4.2

v0.4.2 [DBR 13.3 LTS]

• Geopandas now set to "<0.14.4,>=0.14" due to conflict with minimum numpy version in geopandas 0.14.4.
• H3 python changed from "==3.7.0" to "<4.0,>=3.7" to pick up patches.
• Fixes an issue with fallback logic when deserializing subdatasets from a zip.
• Adjusts data used to speed up a long-running test.
• Streamlines setup_gdal and setup_fuse_install functions:
  • init script and resource copy logic fixed to repo "main" (.so) / "latest" (.jar).
  • added apt-get lock handling for native installs.
  • removed support for UbuntuGIS PPA as GDAL version no longer compatible with jammy default (3.4.x).

v0.4.1

v0.4.1 [DBR 13.3 LTS]

• Fixed python bindings for MosaicAnalyzer functions.
• Added tiller functions, ST_AsGeoJSONTile and ST_AsMVTTile, for creating GeoJSON and MVT tiles as aggregations of geometries.
• Added filter and convolve functions for raster data.
• Raster tile schema changed to be <tile:struct<index_id:bigint, tile:binary, metadata:map<string, string>>.
• Raster tile metadata will contain driver, parentPath and path.
• Raster tile metadata will contain warnings and errors in case of failures.
• All raster functions ensure rasters are TILED and not STRIPED when appropriate.
• GDAL cache memory has been decreased to 512MB to reduce memory usage and competition with Spark.
• Add RST_MakeTiles that allows for different raster creations.
• Rasters can now be passed as file pointers using checkpoint location.
• Added logic to handle zarr format for raster data.
• Added RST_SeparateBands to separate bands from a raster for NetCDF and Zarr formats.

v0.4.0

• First release for DBR 13.3 LTS which is Ubuntu Jammy and Spark 3.4.1. Not backwards compatible, meaning it will not run on prior DBRs; requires either a Photon DBR or a ML Runtime (Standard, non-Photon DBR no longer allowed).
• New setup_fuse_install function to meet various requirements arising with Unity Catalog + Shared Access clusters; removed the scala equivalent function, making artifact setup and install python-first for scala and Spark SQL.
• Removed OSS ESRI Geometry API for 0.4 series, JTS now the only vector provider.
• MosaicAnalyzer functions now accept Spark DataFrames instead of MosaicFrame, which has been removed.
• Docs for 0.3.x have been archived and linked from current docs; notebooks for 0.3.x have been separated from current notebooks.
• This release targets Assigned (vs Shared Access) clusters and offers python and scala language bindings; SQL expressions will not register in this release within Unity Catalog.

v0.3.14

• Fixes for Warning and Error messages on mosaic_enable call.
• Performance improvements for raster functions.
• Fix support for GDAL configuration via spark config (use 'spark.databricks.labs.mosaic.gdal.' prefix).

v0.3.13

R bindings generation fixed and improved.
Remove usage of /vsimem/ drivers for GDAL due to memory leaks.
Add support for MapAlgebra expressions via RST_MapAlgebra.
Add support for custom combine python functions via RST_DerivedBand.
Improve test coverage.
Allow for GDAL configuration via spark config (use 'spark.databricks.labs.mosaic.gdal.' prefix).

v0.3.12

• Make JTS default Geometry Provider
• Updated R bindings for sparkr and sparklyr
• Add raster tile functions.
• Expand the support for raster manipulation.
• Add abstractions for running distributed gdal_translate, gdalwarp, gdalcalc, etc.
• Add RST_BoundingBox, RST_Clip, RST_CombineAvg, RST_CombineAvgAgg, RST_FromBands, RST_FromFile, RST_GetNoData,
  RST_InitNoData, RST_Merge, RST_MergeAgg, RST_NDVI, RST_ReTile, RST_SetNoData, RST_Subdivide
• Add RST_Tessellate that generates H3 tiles from rasters.
• Add RST_ToOverlappingTiles that generates tiles with overlapping areas from rasters.
• Add GDAL generic format reader.

This release is for DBR < 13, mainly DBR 12.2 LTS. Stay tuned for a release very soon that targets DBR 13.3 LTS.

v0.3.11

• Update the CONTRIBUTING.md to follow the standard process.
• Fix for issue 383: grid_pointascellid fails with a Java type error when run on an already instantiated point.
• Bump maven-assembly-plugin from 3.5.0 to 3.6.0.
• Fix the cluster side init script generation.
• Fixed photon check for DBR warnings.
• Bump maven-surefire-plugin from 3.0.0 to 3.1.0.
• Fix the bug described in issue 360: incomplete coverage from grid_geometrykring and grid_tessellate.
• Add default value for script location path to init script.

v0.3.10

• Fixed k-ring logic for BNG grid close to the edge of the grid
• Fixed deprecated st_centroid2D expression
• Documentation improvements
• Fix handling null geometry fields for OGRFileFormat reader
• Added warning for future DBR environment support
• Added support for GeometryCollection
• Fix intersection operations with ESRI geometry APIs
• Fixed custom grid issues for grids not multiple of the root size resolution
• Fixed python binding for rst_georeference
• Fixed ESRI create polygon with correct path order with ESRI APIs
• Fixed automatic SQL registration with GDAL

v0.3.9

• Fixed k-ring interpolation on raster data read