Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added basic readiness check for registry API #2997

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

soyarsauce
Copy link
Contributor

What this PR does

Fixes #2992

Checklist

  • There are unit tests to verify my changes are correct or unit tests aren't applicable
  • I've updated CHANGES.md with what I changed.
  • I've linked this PR to an issue in ZenHub (core dev team only)

@t83714 t83714 self-requested a review October 7, 2020 23:12
Copy link
Contributor

@t83714 t83714 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just some minor error handling changes are required.

Comment on lines +48 to +57
path("ready") {
DB readOnly { session =>
recordPersistence.getReadiness(Some(logger))(session)
} match {
case Some(true) => complete("OK")
case Some(false) | None =>
complete(StatusCodes.InternalServerError, "Database not ready")
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested in the cluster and it seems most exceptions are not captured.
e.g.

java.util.NoSuchElementException: key not found: to_regclass
OR org.postgresql.util.PSQLException: Connection to registry-db:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
OR org.postgresql.util.PSQLException: FATAL: role "client" does not exist

I think you probably need to capture any failure from the DB.readonly method as the execution is happening there.
you might want to try scala's util.Try instead of try/catch --- it's more common in scala as it's more flexible / offer more streamlined exception processing
https://www.scala-lang.org/api/2.9.3/scala/util/Try.html

You can do something like this:

import scala.util.{Try, Failure, Success}
Try{
        DB readOnly { session =>
          recordPersistence.getReadiness(Some(logger))(session)
        }
    } match {
        case Success(Some(true)) => complete("OK")
        case Success(Some(false)) | Success(None) =>
          complete(StatusCodes.InternalServerError, "Database not ready")
        case Failure(exception) => complete(StatusCodes.InternalServerError, "Database not ready Other")
    }

The recordPersistence.getReadiness doesn't seem to need to capture any exceptions (so you don't have to capture at multiple place) nor has a chance to produce a NONE. I guess a Boolean could simpler

@soyarsauce soyarsauce self-assigned this Oct 8, 2020
@soyarsauce
Copy link
Contributor Author

I'm sorry @t83714 I've run out of time for this one, are you able to do the error handling changes you have in mind, then get this in? 🙏 I think it would still be extremely useful for the registry API to have this check!

@t83714
Copy link
Contributor

t83714 commented Nov 18, 2020

@soyarsauce
Thanks for your work on this and just leave to me.
I will get this done after this one #3024

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

We need to re-look at liveness / readiness Probs logic of our pods
3 participants