add config settings for package versions (e.g. titiler-pgstac==0.5.1)

[hrodmn]

Right now the specific versions of the critical python packages are hard-coded in the runtime files

e.g.

eoapi-cdk/lib/titiler-pgstac-api/runtime/requirements.txt

Line 1 in 1e39ddb

titiler.pgstac==0.5.1

I have been following along over in developmentseed/eoAPI#144 and see pattern you follow for overriding the runtime. I suppose I could use this method to achieve my goal but it seems heavy-handed for my goal of staying up-to-date on dependency upgrades!

https://github.com/developmentseed/eoAPI/pull/144/files#diff-5e565e714fe20093460499e9958e38735b99628ac37b603c30e2c8af61aee726R169-R184

One of the things that is very convenient about eoapi-cdk is that it makes it possible to just import the cdk modules without maintaining separate runtime scripts, so adding some configurability for the deployment of the packages would be nice!

Would it make sense to be able to set the versions of packages like titiler-pgstac and tipg as settings in the config? Maybe it would be tricky since some of the handler code might be dependent on version-specific details, but it seems like a way to decouple versions of eoapi-cdk from versions of the family of dependencies.

[emileten]

I think it it would be a useful option, and you do make me realize that the current situation is undesirable in that it forces a user to either use the versions that are hard-coded here, or do a lot of work to inject 'custom' runtimes.

I did encounter this once in the past :

Maybe it would be tricky since some of the handler code might be dependent on version-specific details

Where updating the titiler version broke its handler, but I think since then we improved the way we wrote it, and in principle it should be as thin as possible and allow for the kind of feature you describe.

[emileten]

@hrodmn I just saw your PR. I have a quick question. Sorry that I had not thought of asking that question when writing my first reaction above.

We have to update the package versions in the default deployment anyways, regardless of the feature we describe here, because some defaults are getting outdated. I just filed an issue for this, I was hoping to get to it soon : #88.

If these were updated, and updated frequently in the future, would your issue be solved, or would the feature we described above still be helpful ?

[hrodmn]

If these were updated, and updated frequently in the future, would your issue be solved, or would the feature we described above still be helpful ?

@emileten I think it would still be useful because it would be good to be able to control the version of packages like titiler-pgstac which occasionally introduce breaking changes to the API. e.g. parameter name changes like coord-crs -> coord_crs in v0.7.0. With this feature in place, I could choose to update other packages while keeping titiler-pgstac<0.7. Without this feature I would need to stick with whichever dependency versions that got installed alongside titiler-pgstac<0.7 in an eoapi-cdk release.

The only downside that I can see about letting the user pick specific versions is the handler problem. If there are version-specific tweaks that need to be made to the handlers (like we probably need to do for pgstac>=0.8) then eoapi-cdk would need to have a way to declare a range of dependency versions that are compatible with a given release.

[vincentsarago]

We so many runtime version I'm worry that it will be confusing and that most user will not get that handlers are dependent to a specific version! I feel that having a default in eoapi-cdk the most safe way, as long as we make customization (letting user pass their own dockerfile and handler)

imagine that you don't have eoapi-cdk pinned to a specific version but you have an environment settings that overwrite the default titiler-pgstac, but in a new release we break the handler, you won't get any notification about why your application doesn't work anymore! It will be messy to debug!

[hrodmn]

imagine that you don't have eoapi-cdk pinned to a specific version but you have an environment settings that overwrite the default titiler-pgstac, but in a new release we break the handler, you won't get any notification about why your application doesn't work anymore! It will be messy to debug!

We could define a valid range of versions for each dependency in each release of eoapi-cdk. I don't know exactly how we would define the range, though. We would probably need to add some python unit testing to this repo to test the handlers against many versions of the dependencies.

Once we have a valid range of versions (or list of versions) defined, it should be pretty easy to validate in the typescript files so we can provide a useful error message:

import * as semver from 'semver';

function checkVersion(version: string): boolean {
    const validRange = '>=0.5 <0.7';
    
    if (!semver.satisfies(version, validRange)) {
        throw new Error(`Version ${version} is outside the valid range for this version of eoapi-cdk: ${validRange}`);
    }
}

@vincentsarago did you ever consider shipping handler scripts (e.g. https://github.com/developmentseed/eoapi-cdk/blob/main/lib/titiler-pgstac-api/runtime/src/handler.py) with titiler-pgstac? If the code in the handlers can be so tightly linked with the dependency versions, does it make sense to move them over to the dependency repos?

[vincentsarago]

@vincentsarago did you ever consider shipping handler scripts (e.g. https://github.com/developmentseed/eoapi-cdk/blob/main/lib/titiler-pgstac-api/runtime/src/handler.py) with titiler-pgstac? If the code in the handlers can be so tightly linked with the dependency versions, does it make sense to move them over to the dependency repos?

This could be a nice solution, but right now this handler is AWS Lambda oriented, if you do this for one cloud service then you open a 🕳️

Once we have a valid range of versions (or list of versions) defined, it should be pretty easy to validate in the typescript files so we can provide a useful error message:

This seems a lot to add here, It adds a burden to the maintainer of this library for something which (IMO) is better and safer to be handled as user level (via customization)

[hrodmn]

This seems a lot to add here, It adds a burden to the maintainer of this library for something which (IMO) is better and safer to be handled as user level (via customization)

@vincentsarago @emileten it's totally understandable that you would not want to keep track of compatibility with all possible versions of packages like titiler-pgstac in this package. If the custom Dockerfile/handler path is the best way forward I will do that!

[emileten]

This seems a lot to add here

I would agree with this statement from Vincent. It would be great but it's a bit risky because we don't have much time to maintain this library.