Collaboration question

[tloubrieu-jpl]

Hi Daniel,

I am using a ticket as an open letter to you.

I work at JPL, currently contributing to the Europa-Clipper Science Data System.

We inherited a CCSDS parsing python library from a student in the team and since we were very happy with it, and making extensions on it (such as reading packets with variable length), we were willing to open-source it.

But I realized you already have a successful development which does what we need minus maybe a few features, so we changed our strategy to:

• migrate the CCSDS parsing functions of our system to CCSDSpy
• create pull request on your repository to add the possibly missing features

We would like to hire a student to do this work this summer (June-August). Would you support this initiative ?

We don't have the details yet on the open-sourcing procedure on our side, but that might involve to move your repository from a personal organization to something more official (e.g. http://github.com/NASA-PDS/). Maybe you already have an opinion on that ?

Thanks,

[ddasilva]

Hi @tloubrieu-jpl,

First of all, thank you for your interest in the library. I'm happy to answer questions and support your migration to the library. I've done this for a few NASA missions in the past and am currently doing it for the Canadian Space Agency, so don't think too much of it.

I'd be happy to discuss adding additional features. In the past, I've found it's quicker for people to contribute requirements and then together implement them openly through github, rather than try to transfer code between organizations. Most of the time, the feature isn't there because we didn't prioritize it or weren't aware of its importance.... not because it requires outside knowledge or intellectual property to implement. It is a rather big production when code is transferred between organizations, because of legal concerns (from both sides), administrators need to get involved, and so on. I'd like to avoid red tape as much as possible! On the spectrum between "developing openly from scratch on github" and "transferring code from JPL" is you sharing code/tests to use as a reference for an implementation developed openly, and doing the necessary paperwork on your side to allow you to share it.

To determine what's most appropriate, could you give me an idea of what the missing features are? That will help me understand the situation better.

Regards moving the repo outside my personal account and into a community org: yes, this is on my radar! I just haven't gotten to it yet. I can certainly get this done in the future. My end goal is to make this a community run project with representation from major organizations (eg JPL, GSFC, JHU/APL, ESA, private space, and so on), but I haven't set up the necessary community governance policies or formed a team. I'm have my fingers crossed that one of our developers, @ehsteve (who organized the successful SunPy project years ago) will advise on that process in the future.

Best,
Daniel

[tloubrieu-jpl]

Thanks Daniel, that sounds perfect to me.

Regarding the missing features in CCSDSpy for Europa clipper there might not be any. We did not even start to analyse how we would migrate our code to ccsdspy so I don't really know what we will need yet.

I can try to quickly have a look at that.

However, since the migration to ccsdspy is not a short term priority but more a long term will to consolidate an open source resource for the community rather developing a 'short' lived and painfully maintained internal library used for a single mission, we will wait to have a student to do most of this work, this summer.

[ehsteve]

@ddasilva I see you created an org which is great. I'd be happy to volunteer to be a maintainer if you need some help.

[ddasilva]

@ddasilva I see you created an org which is great. I'd be happy to volunteer to be a maintainer if you need some help.

I invited you when I made the org but it looks like the invite expired! I invited you again

[ddasilva]

@tloubrieu-jpl Any update on this collaboration?

[tloubrieu-jpl]

Hi @ddasilva ,

We have selected a student to work on that subject this summer. I did not analyse the pull request that we would need for CCSDSpy to comply with our requirements.

Thanks,

[ddasilva]

Thanks for the update @tloubrieu-jpl . When you're ready, let me know if you want to jump on a call and talk further steps.

Best,
Daniel

[ehsteve]

@ddasilva could i have permission to assign labels to issues?

[ddasilva]

@ehsteve Can we move this to email or new issue so we don't unnecessarily ping @tloubrieu-jpl ?

Do you know what permission you need? You should have the highest privileges

[ehsteve]

@tloubrieu-jpl, i don't know how things work at JPL but the SMD Policy document (SPD-41a) does lay out guidance about open source contributions from NASA which, I assume, JPL has to be consistent with.

Here are some direct quotes from that document.

• Missions are encouraged to directly contribute to open source software projects to support the mission and benefit the community.
• Libraries developed using SMD funding shall be released no later than the publication it supports. Libraries should be developed openly in a version-controlled platform and SMD-funded researchers are encouraged to contribute to existing, open source libraries.
• SMD should encourage the adoption of open software policies with external partners, contractors, and grantees to ensure the usability of software to the scientific community. SMD shall foster and encourage contributions and engagement with the open
  source community from all Federal employees, contractors, and grantees. This includes leveraging existing open source software and communities.
• SMD should foster and encourage contributions and engagement with communities and organizations setting standards and best practices

[tloubrieu-jpl]

Hi @ddasilva, @ehsteve,

Thanks @ehsteve for pointing to this policy document which supports our initiative here.

We started to work with your package and currently we are testing the CCSDSpy library with one of our Europa-Clipper test datasets. We also attempted to plug CCSDSpy into our own code but we were stopped because of divergence between the packet field data_types supported by CCSDSpy and those supported by our native code which parses CCSDS packets.

Could we meet to discuss our work plan with you ? Would you be available from 2 to 3pm PDT this Friday (06/23) ?
We are having a list of features that we want to be supported, and I would like to have your opinions on them to be available or implemented in CCSDSpy library.

[ddasilva]

Hey @tloubrieu-jpl ,

Good to hear from you. I'm happy to meet this Friday at that time. Feel free to send a meeting invite to [email protected].

[tloubrieu-jpl]

Hi @ddasilva ,

We are now approved to open source the code we wrote to generically to parse the downlink files using the CCSDS standard (and using the CCSDSpy library).
Provisionally, I published the code there https://github.com/tloubrieu-jpl/pyDownLinkParser.

The packet definitions for Europa-Clipper are now in this repository but the intention is to externalize them in a different python package so that this one is really independent from the missions (eventually we can consider the CCSDS packet definition package as a plugin which will enable reading the defined packet whenever they are found in an input file).

I am thinking of 2 options to publish the repository:

1. your CCSDSpy organization, if you thinks that makes sense.
2. the nasa organization https://github.com/nasa/ , which would give some exposure to CCSDSpy from there.

Do you have thoughts on that ?

Thanks,

Thomas

[tloubrieu-jpl]

Hi @ddasilva ,

Did you have chance to see my question above ?

Thanks,

Thomas

[ddasilva]

Sorry, I totally missed this thread. I think it would be great to have this under https://github.com/nasa, because then we can link to it as a user success story when we promote the project. What do you think?

[tloubrieu-jpl]

I agree with that @ddasilva . Thanks !