Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Access denied for rekor-trillian-createdb pod connecting to trillian-mysql #874

Open
kevchu3 opened this issue Dec 23, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@kevchu3
Copy link

kevchu3 commented Dec 23, 2024

Description

I've deployed the Rekor Helm chart which also spins up the trillian-system chart.

The trillian-logserver, trillian-logsigner, and trillian-mysql pods come up and stay running, but the rekor-trillian-createdb job continually fails. Since it's a job, it keeps trying every few minutes. Here's a look into the goroutine panic that I'm getting:

$ kubectl logs rekor-trillian-createdb-5mhbc 
2024/12/23 19:43:57 failed to ping db: Error 1045: Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
panic: failed to ping db: Error 1045: Access denied for user 'mysql'@'10.42.0.222' (using password: YES)

goroutine 1 [running]:
log.Panicf({0xb8602c?, 0xca28f0?}, {0xc0000b7d38?, 0x32?, 0x2?})
	log/log.go:395 +0x67
main.main()
	github.com/sigstore/scaffolding/cmd/trillian/createdb/main.go:238 +0x3b2

I restarted the trillian-mysql server pod to get some fresh logs, here's a look at that as well. In the logs below, the 10.42.0.222 IP address is the rekor-trillian-createdb pod:

$ kubectl logs trillian-mysql-776fc545d9-46gqj
2024-12-23T19:42:51.125238Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2024-12-23T19:42:51.127052Z 0 [Note] mysqld (mysqld 5.7.38) starting as process 1 ...
2024-12-23T19:42:51.130118Z 0 [Note] InnoDB: PUNCH HOLE support available
2024-12-23T19:42:51.130133Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-12-23T19:42:51.130139Z 0 [Note] InnoDB: Uses event mutexes
2024-12-23T19:42:51.130144Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2024-12-23T19:42:51.130149Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-12-23T19:42:51.130153Z 0 [Note] InnoDB: Using Linux native AIO
2024-12-23T19:42:51.130725Z 0 [Note] InnoDB: Number of pools: 1
2024-12-23T19:42:51.130822Z 0 [Note] InnoDB: Using CPU crc32 instructions
2024-12-23T19:42:51.133206Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2024-12-23T19:42:51.141631Z 0 [Note] InnoDB: Completed initialization of buffer pool
2024-12-23T19:42:51.144291Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2024-12-23T19:42:51.154314Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2024-12-23T19:42:51.154334Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2024-12-23T19:42:51.154340Z 0 [Note] InnoDB: Retrying to lock the first data file
2024-12-23T19:42:52.154441Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2024-12-23T19:42:52.154463Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2024-12-23T19:42:53.165679Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2024-12-23T19:42:53.189696Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2024-12-23T19:42:53.189715Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-12-23T19:42:53.189812Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-12-23T19:42:53.237178Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-12-23T19:42:53.237945Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2024-12-23T19:42:53.237958Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2024-12-23T19:42:53.238912Z 0 [Note] InnoDB: 5.7.38 started; log sequence number 12575253
2024-12-23T19:42:53.239402Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-12-23T19:42:53.239610Z 0 [Note] Plugin 'FEDERATED' is disabled.
2024-12-23T19:42:53.258446Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2024-12-23T19:42:53.258666Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2024-12-23T19:42:53.258673Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2024-12-23T19:42:53.258678Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2024-12-23T19:42:53.259352Z 0 [Warning] CA certificate ca.pem is self signed.
2024-12-23T19:42:53.259390Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2024-12-23T19:42:53.260615Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2024-12-23T19:42:53.260669Z 0 [Note] IPv6 is available.
2024-12-23T19:42:53.260684Z 0 [Note]   - '::' resolves to '::';
2024-12-23T19:42:53.260710Z 0 [Note] Server socket created on IP: '::'.
2024-12-23T19:42:53.261395Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-12-23T19:42:53.262744Z 0 [Note] InnoDB: Buffer pool(s) load completed at 241223 19:42:53
2024-12-23T19:42:53.295771Z 0 [Note] Event Scheduler: Loaded 0 events
2024-12-23T19:42:53.296197Z 0 [Note] mysqld: ready for connections.
Version: '5.7.38'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
2024-12-23T19:43:47.970157Z 4 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:49.973826Z 5 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:51.976939Z 6 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:53.981175Z 7 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:55.984607Z 8 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)
2024-12-23T19:43:57.987249Z 9 [Note] Access denied for user 'mysql'@'10.42.0.222' (using password: YES)

Version

I'm deploying onto Red Hat Microshift, so some of the security best practices of using non-root users are more along the lines of OpenShift than vanilla Kubernetes
Rekor helm chart is version 1.3.7
I'm using a pretty vanilla values.yaml file:

server:
  ingress:
    enabled: false
@kevchu3 kevchu3 added the bug Something isn't working label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant