GPII-4247: Added a script to update client credentials in CouchDB

[javihernandez]

This is a tiny script that updates the client credentials in CouchDB, see https://issues.gpii.net/browse/GPII-4247

Ideally, we should think a good way to automate this process and avoid as much as possible any manual intervention for the credentials to be updated in the production environment.

[gpii-bot]

CI job failed: https://ci.gpii.net/job/universal-tests/2026/

[gpii-bot]

CI job passed: https://ci.gpii.net/job/universal-tests/2028/

[klown]

Hi @javihernandez
This looks familiar, so you may already know this, but:

1. There is a utility library for accessing the database in the universal scripts folder, called dbRequestUtils.js. It uses the http module instead of request. It hides some of the details of making requests. Flushtokens, @cindyli's migration script, and the dataloader use the library.
2. The dataloader loads the credentials into the database, although it's not obvious. There is a step in the script for loading all the data from the "static data" folder. That static data folder is located at universal/testData/dbData/, and contains clientCredentials.json as one of its files. The dataloader is run whenever the credentials change in universal. However, you might be aiming to run it more frequently, or to use a different credentials file; hence, you may need a separate script for this.

[javihernandez]

Hey there @klown

as a follow up on our discussion on slack, yes, I can definitely use the utilities from dbRequestUtils.js.
Regarding the task of automating this process, I'm happy if we can find a way to achieve this (if possible).

First, let me clarify a few things.

We have the credentials for 3 different pilot sites (NOVA, Chickasaw, UMD) and one for us (the testers credentials). These credentials were generated locally and loaded into our production database by a devop.
Then, they were updated during Cindy's migration to the new schema. So, the credentials are only living in the database, nobody has a local copy of these credentials and I only know the ids just in case we need to update (or revoke) them. These updates are not going to happen very often, the last one was performed in August.

In terms of automating this, I guess that we should figure out a way to:

1.- Store the credentials ids in a decent and secure way, I'm not sure if having them in a private repository is an option, or if there are any good practices when dealing with this kind of situations.

2.- Find the best way, time (or event to trigger the update) and place from we can perform the task

I'm sure that our devops can help us to figure out the best way to put all the pieces together. However, given the complexity of the task, and that these small updates shouldn't happen very often, I talked to Cindy and we thought that a small script that can be manually used from time to time, would do the work. Then, I can send the updates to a devop in a file and the ids of the client credentials we want to update.

So, the question is, is it really worth to automate this process? Any ideas and opinions are welcome :)
cc @cindyli @amatas @stepanstipl

[klown]

Hey @javihernandez, thanks for the write up. And, it's good that you can make use of dbRequestUtils.js.

I have one question about:

I can send the updates to a devop in a file and the ids of the client credentials we want to update.

Why do you have to run the script to generate the updates? I'm not a security expert, but having to send the updated credentials from a developer to devOps is less secure than having devOps generate the updated credentials and upload them in one step, and store the ids in a secure way that allows access to those who need them, and have the proper privileges.

Not that I'm trying to give devOps more work :-)

[javihernandez]

Why do you have to run the script to generate the updates?

Because nobody has a copy of the current client credentials. The current script retrieves the client credentials, updates the data with the updates provided and saves the document back into couchdb. This way, neither the developer nor the devop are dealing with the full client credentials.

Let's say that we're going to deploy an update to one client credentials with what we have right now - this script.

First, the developer puts the update block in a JSON file, something like this:

{
    "allowedPrefsToWrite": [
        "http://registry.gpii.net/applications/com.microsoft.office",
        "http://registry.gpii.net/applications/com.microsoft.windows.colorFilters",
        "http://registry.gpii.net/applications/com.microsoft.windows.mouseSettings",
        "http://registry.gpii.net/common/cursorSize"
    ]
}

Then, the developer sends the following to the devop:

1. The JSON file
2. The id of the client credentials we want to update

The devop needs to run the script providing these 2 things, that's all. They don't need to know who's the owner of the credentials, they only know the id. If we want to store the ids for the devops to use them when needed, then they'll know which credentials belong to a particular site.

Regarding the update block, the script is implemented in a way that the block will end up with the content provided. This is, it'll overwrite any content in the updated block (or blocks).

[klown]

Thanks for the explanation @javihernandez. It makes sense to me now.

Given the free form nature of the update block for the allowedPrefsToWrite (and also the allowedIPBlocks array), it's doubtful that creation of the JSON file containing the updates can be automated.

Regarding using dbRequestUtils.js: there is no support for PUT requests. It does support POST with couchdb's \_bulk_docs end point, which could be used for a single document. That might be overkill, though, and I think I will add PUT capability to the library.

[javihernandez]

@klown thanks for taking a look at this. I'd say that, given that we've now decided to move to postgresql, we should not invest more time in improving this script - and we should then think about a good strategy for updating the sites credentials after our migration to postgresql.

In any case, there are still a few credentials in our production db that we need to update, which is becoming urgent at this point (my bad for not reminding about this :( ) . The pilot sites are updating to recent versions of Morphic that include new settings and they aren't able to save such settings. cc/ @cindyli @amb26

[klown]

@javihernandez

@klown thanks for taking a look at this.

No problem - you're welcome. And you're correct in that our focus should be the migration to postgresql. Hopefully something from the dbRequestUtils.js can be used at an abstract level, if not the details. We'll see.

[cindyli]

It's not wise to hide the body value. The CouchDB error message could also be reported via it. For example, when a unmatched revision id is provided at a update, body value will receive:

{
	error: "conflict",
	reason: "Document update conflict."
}

The error handling at line 120 should also take it into account.

[cindyli]

This doc should also explain that the JSON can only contain fields/values that need to be updated. It is not necessary to provide a complete structure of a client credential record.

In the script, verify:

1. If "_id" is given in the file, either make sure its value matches the value of the second parameter clientCredentialId, or don't allow "_id" to be given in the file at all;
2. JSON in this file should not contain "_revision" field.