Adding files that were missed in the previous release (Documentation)…

… and link fixing.
esnet · Nov 3, 2023 · 3dbb107 · 3dbb107
1 parent a85eb7b
commit 3dbb107
Show file tree

Hide file tree

Showing 5 changed files with 459 additions and 1 deletion.
diff --git a/website/content/en/docs/gdg/backup_guide.md b/website/content/en/docs/gdg/backup_guide.md
@@ -0,0 +1,187 @@
+---
+title: "Backup Commands Guide"
+weight: 16
+---
+
+Every namespace supporting CRUD operations has the functions: list, download, upload, clear operating on only the monitored folders.
+
+
+
+### Alert Notifications (DEPRECATED)
+
+This will stop working soon both as a concept in grafana and something that GDG will support.
+
+Allows you to manage alertnotifications (an) if you have any setup
+
+```sh
+./bin/gdg backup an list -- Lists all alert notifications
+./bin/gdg backup an download -- retrieve and save all alertnotifications from grafana
+./bin/gdg backup an upload  -- writes all local alert notifications to grafana
+./bin/gdg backup an clear -- Deletes all alert notifications
+```
+
+### Connections
+#### (was: DataSources)
+
+Note:  Starting with 0.4.6 this was renamed to connections.
+
+Connections credentials are keyed by the name of the DataSource.  See see [config example](https://github.com/esnet/gdg/blob/master/conf/importer-example.yml).  If the connection JSON doesn't have auth enabled, the credentials are ignored.  If Credentials are missing, we'll fall back on default credentials if any exist.  The password is set as a value for basicAuthPassword in the API payload.
+Datasources are imported or exported from _organization_ specified in configuration file otherwise current organization user is used.
+
+
+All commands can use `connection` or `c` to manage datasources.  (Legacy options of `datasource` and `ds` are also supported)
+
+```sh
+./bin/gdg backup c list -- Lists all current connections
+./bin/gdg backup c download -- Import all connections from grafana to local file system
+./bin/gdg backup c upload -- Exports all dashboard from local filesystem (matching folder filter) to Grafana
+./bin/gdg backup c clear -- Deletes all connections
+```
+
+
+### Dashboards
+
+Dashboards are imported or exported from _organization_ specified in configuration file otherwise current organization user is used.
+
+All commands can use `dashboards` or `dash` to manage dashboards
+
+```sh
+./bin/gdg backup dash list -- Lists all current dashboards
+./bin/gdg backup dash download -- Import all dashboards from grafana to local file system
+./bin/gdg backup dash upload -- Exports all dashboard from local filesystem (matching folder filter) to Grafana
+./bin/gdg backup dash clear -- Deletes all dashboards
+```
+
+You can also use filtering options to list or import your dashboard by folder or by tags.
+
+```sh
+./bin/gdg backup dash download -f myFolder
+./bin/gdg backup dash download -t myTag
+./bin/gdg backup dash download -t tagA -t tagB -t tagC
+```
+
+
+
+### Folders
+
+Mostly optional as Dashboards will create/delete these are needed but if there is additional metadata you wish to persist you can use this to manage them.
+
+```sh
+./bin/gdg backup folders list -- Lists all current folders
+./bin/gdg backup folders download -- Import all folders from grafana to local file system
+./bin/gdg backup folders upload -- Exports all folders from local filesystem
+./bin/gdg backup folders clear -- Deletes all folders
+```
+
+### Folder Permissions
+
+This CRUD allows you to import / export folder permissions.  Initial release will be part of v0.4.4.  There are a lot of nested relationship that go with this.
+
+Expectations:
+  - Users have to already exist.
+  - Teams (if used) need to already exist.
+  - Folders also need to already exist.
+
+The Folder Permissions will list, import and re-apply permissions.  But the expectations is that all other entities are already there.  Next few iteration will try to add more concurrency for
+this tool and more error checking when entities that don't exist are being referenced.
+
+**NOTE:** Unlike other command, permissions does not have a `clear` function.  Theoretically you could have a folder name with an emtpy array under folder-permissions to clear all known permissions to the folder, but otherwise
+clearing permissions from all folders seems too destructive to really be a useful function.
+
+```sh
+./bin/gdg backup folders list -- Lists all current folder permissions
+./bin/gdg backup folders download -- Retrieve all folders permissions from Grafana
+./bin/gdg backup folders upload -- Exports all folders from local filesystem
+```
+
+```
+┌───────────┬──────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────┬─────────────┬────────────────────────────────┬────────┬─────────────────┐
+│ FOLDER ID │ FOLDERUID                            │ FOLDER NAME                                                                       │ USERID      │ TEAM NAME                      │ ROLE   │ PERMISSION NAME │
+├───────────┼──────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┼─────────────┼────────────────────────────────┼────────┼─────────────────┤
+│ 2272      │ dfba969d-565b-481e-a930-53aa5684992c │ sub-flow                                                                          │             │                                │        │                 │
+│                                                  │     PERMISSION--->                                                                │ admin       │                                         │ Admin           │
+│ 520       │ GPmSOQNnk                            │ EngageMap (internal beta)                                                         │             │                                │        │                 │
+│                                                  │     PERMISSION--->                                                                │                                              │ Admin  │ Edit            │
+│                                                  │     PERMISSION--->                                                                │                                              │ Editor │ Edit            │
+│                                                  │     PERMISSION--->                                                                │                                              │ Viewer │ View            │
+│ 2031      │ n3xS8TwVk                            │ Team CMS - US dumb dumb                                                           │             │                                │        │                 │
+│                                                  │     PERMISSION--->                                                                │             │ authscope_team_cms             │        │ Edit            │
+│ 1746      │ pASPyoQVk                            │ Team DOE-IN-PNNL - DOE-IN Pacific Northwest National Laboratory                   │             │                                │        │                 │
+└──────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────┴─────────────┴────────────────────────────────┴────────┴─────────────────┘
+```
+
+The listing includes the folder name, followed by several lines with "PERMISSION--->" which will each list a permission.  It can a user being granted access or a team being granted a role etc.
+
+
+
+### Library Elements
+
+Library elements are components that can be shared among multiple dashboards.  Folder matching will still be applied, so any folders not monitored will be ignored unless explicitly specified.  If wildcard flag is enabled, all elements will be acted on irrelevant of folder location
+
+All commands can use `libraryelements` aliased to `library` and `lib` for laziness purposes.
+
+```sh
+./bin/gdg backup lib list -- Lists all library components
+./bin/gdg backup lib download -- Import all library components from grafana to local file system
+./bin/gdg backup lib upload -- Exports all library components from local filesystem (matching folder filter) to Grafana
+./bin/gdg backup lib clear -- Deletes all library components
+./bin/gdg backup lib  list-connections <Lib Element UID> -- Will list all of the dashboards connected to the Lib Element (Coming in v0.4.2)
+```
+
+
+
+### Organizations
+#### Auth:  Requires Grafana Admin (Tokens not supported, Org Admins don't have access)
+Command can use `organizations` or `org` to manage organizations.
+
+NOTE: this only manages top level of the orgs structure. It's mainly useful to maintain consistency.
+
+```sh
+./bin/gdg backup org list -- Lists all organizations
+./bin/gdg backup org upload -- Upload Orgs to grafana
+./bin/gdg backup org download -- Download Orgs to grafana
+```
+
+### Teams
+
+{{< alert icon="👉" text="Admin team members are unable to be exported back.  Currently all members except the server admin will be exported as regular members" />}}
+
+{{< alert icon="👉" text="Users need to be created before team export can succeed" />}}
+
+```sh
+./bin/gdg backup team list  -- Lists all known team members
+./bin/gdg backup team download -- download all known team members
+./bin/gdg backup team upload -- upload all known team members
+./bin/gdg backup team clear -- Delete all known team except admin
+```
+
+{{< details "Team Listing" >}}
+```
+
+┌────┬───────────┬───────┬───────┬─────────┬─────────────┬──────────────┬───────────────────┐
+│ ID │ NAME      │ EMAIL │ ORGID │ CREATED │ MEMBERCOUNT │ MEMBER LOGIN │ MEMBER PERMISSION │
+├────┼───────────┼───────┼───────┼─────────┼─────────────┼──────────────┼───────────────────┤
+│ 4  │ engineers │       │ 1     │ 2       │             │              │                   │
+│    │           │       │       │         │ admin       │ Admin        │                   │
+│    │           │       │       │         │ tux         │ Member       │                   │
+│ 5  │ musicians │       │ 1     │ 1       │             │              │                   │
+│    │           │       │       │         │ admin       │ Admin        │                   │
+└────┴───────────┴───────┴───────┴─────────┴─────────────┴──────────────┴───────────────────┘
+
+```
+{{< /details >}}
+
+
+### Users
+
+Only supported with basic auth.  Users is the only one where basic auth is given priority.  API Auth is not supported, so will try to use basic auth if configured otherwise will warn the user and exit.
+
+NOTE: admin user is always ignored.
+
+```sh
+./bin/gdg backup users list -- Lists all known users
+./bin/gdg backup users download -- Lists all known users
+./bin/gdg backup users upload -- Export all users (Not yet supported)
+./bin/gdg backup users clear -- Delete all known users except admin
+```
+
diff --git a/website/content/en/docs/gdg/getting_started.md b/website/content/en/docs/gdg/getting_started.md
@@ -0,0 +1,99 @@
+---
+title: "Getting Started"
+weight: 13
+---
+
+### Setup new configuration
+
+You can create new context configuration using an interactive setup.
+```
+$ ./bin/gdg ctx new mycontext
+```
+
+When creating a new context, you will be asked for authorization type, your default datasource and username/password, along with which folders you wish to manage under the context. You have three options:
+
+1. Default option ("General")
+2. List of folders you wish to manage
+3. Wildcard configuration (all folders)
+
+### Import / Download Dashboards
+
+Minimal configuration (eg. the `importer.yml` file) that you need to download your dashboards from your Grafana endpoint:
+```
+context_name: all
+
+contexts:
+  all:
+    url: https://grafana.example.org
+    token: "<<Grafana API Token>>"
+    # user_name: admin
+    # password: admin
+    output_path: exports
+    watched:
+      - Example
+      - Infrastructure
+
+global:
+  debug: true
+  ignore_ssl_errors: false
+```
+You need to adjust three parts in the configuration in order to function:
+- Grafana URL: This is just a URL where your Grafana is available.
+- API Key OR Username / Passoword for Admin user. See [authentication](configuration.md) section if you need more information.
+- Downloaded Folders: The `watched` field defines folders which will be considered for manipulation. You can see these folders in your Grafana Web UI, under Dashboards > Management. From there, you can simply define the folders you want to be downloaded in the `watched` list. The dashboards are downloaded as JSON files in the `$OUTPUT_PATH/dashboards/$GRAFANA_FOLDER_NAME` directory. Where `$OUTPUT_PATH` is the path defined in the `dashboard_output` configuration property and `$GRAFANA_FOLDER_NAME` the name of the folder from which the dashboards were downloaded
+
+After you are done, and you can execute `./bin/gdg dash list` successfully, eg.:
+```
+$ ./bin/gdg dash list
+time="2021-08-22T11:11:27+02:00" level=warning msg="Error getting organizations: HTTP error 403: returns {\"message\":\"Permission denied\"}"
+time="2021-08-22T11:11:28+02:00" level=info msg="Listing dashboards for context: 'all'"
+┌────┬───────────────────────────────────┬───────────────────────────────────┬────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────┐
+│ ID │ TITLE                             │ SLUG                              │ FOLDER         │ UID        │ URL                                                                        │
+├────┼───────────────────────────────────┼───────────────────────────────────┼────────────────┼────────────┼────────────────────────────────────────────────────────────────────────────┤
+│  8 │ AWS CloudWatch Logs               │ aws-cloudwatch-logs               │ Infrastructure │ AWSLogs00  │ https://grafana.example.org/d/AWSLogs00/aws-cloudwatch-logs                │
+│  6 │ AWS ECS                           │ aws-ecs                           │ Infrastructure │ ly9Y95XWk  │ https://grafana.example.org/d/ly9Y95XWk/aws-ecs                            │
+│  5 │ AWS ELB Application Load Balancer │ aws-elb-application-load-balancer │ Infrastructure │ bt8qGKJZz  │ https://grafana.example.org/d/bt8qGKJZz/aws-elb-application-load-balancer  │
+│  4 │ AWS RDS                           │ aws-rds                           │ Infrastructure │ kCDpC5uWk  │ https://grafana.example.org/d/kCDpC5uWk/aws-rds                            │
+│  3 │ AWS S3                            │ aws-s3                            │ Infrastructure │ AWSS31iWk  │ https://grafana.example.org/d/AWSS31iWk/aws-s3                             │
+│ 17 │ Cluster Autoscaling               │ cluster-autoscaling               │ Example        │ iHUYtABMk  │ https://grafana.example.org/d/iHUYtABMk/cluster-autoscaling                │
+└────┴───────────────────────────────────┴───────────────────────────────────┴────────────────┴────────────┴────────────────────────────────────────────────────────────────────────────┘
+```
+After executing `./bin/gdg dash import` you can find the dashboards of the `Infrastructure` folder in the local directory `dashboards/dashboards/Infrastructure` and the dashboards of the `Example` directory in the local directory `dashboards/dashboards/Example`.
+
+### Export / Upload Dashboards
+
+Minimal configuration (eg. the `importer.yml` file) that you need to upload your dashboards from your Grafana endpoint:
+```
+context_name: all
+
+contexts:
+  all:
+    url: https://grafana.example.org
+    token: "<<Grafana API Token>>"
+    # user_name: admin
+    # password: admin
+    output_path: exports
+    watched:
+      - Example
+      - Infrastructure
+
+global:
+  debug: true
+  ignore_ssl_errors: false
+```
+You need to adjust three parts in the configuration in order to function:
+- Grafana URL: This is just a URL where your Grafana is available.
+- API Key OR Username / Passoword for Admin user. See [authentication](configuration.md) section if you need more information.
+- Uploaded Folders: The `watched` field defines folders which will be considered for manipulation. The dashboards should be stored as JSON files in the `$OUTPUT_PATH/dashboards/$GRAFANA_FOLDER_NAME` directory. Where `$OUTPUT_PATH` is the path defined in the `dashboard_output` configuration property and `$GRAFANA_FOLDER_NAME` the name of the folder to which the dashboards will be uploaded. In case of the above configuration file, the dashboards should be stored locally in the `dashboards/dashboards/Example` and `dashboards/dashboards/Infrastructure/` directories.
+
+```
+├── bin
+|   └── gdg
+└── exports
+    └── dashboards
+        ├── Example
+        |   └── cluster-scaling.json
+        └── Infrastructure
+            └── aws-ecs.json
+```
+You can execute `./bin/gdg dash export` to upload the local dashboards to your Grafana. Afterwards, you can try running `./bin/gdg dash list` in order to confirm that your dashboards were uploaded successfully.