Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV #163

Open
spastorclovr opened this issue Apr 8, 2024 · 0 comments
Open

When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV #163

spastorclovr opened this issue Apr 8, 2024 · 0 comments

Comments

@spastorclovr
Copy link
Contributor

I have been trying to set concurrentWriters to 0 to make sure no limits are being put in the number of writers trying to write on the mounted volume.

I got it working at some point and it did improve a lot the writing thoughput for our service.
But, I noticied that the filerUI did not display the same file listing as the one I could see through the mounted volume.
So, I tried from scratch (shutting down all pods and up again) , and I have now the error below 👇
(Adjusting -cacheCapacityMB did not make any difference)

Seaweedfs version: 3.63
Seaweedfs-csi helm version: 0.2.1

I0408 08:06:59.211052 nodeserver.go:103 node publish volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.211383 nodeserver.go:137 volume pvc-80c9113b-5473-42ed-8041-c54a02cc28b7 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-80c9113b-5473-42ed-8041-c54a02cc28b7/mount
I0408 08:06:59.211409 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:06:59.212475 nodeserver.go:137 volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.212492 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:07:01.241693 meta_cache_init.go:44 ReadDirAllEntries /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624 ...
I0408 08:07:01.241748 filer_client.go:120 read directory: directory:"/buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624" limit:10001
I0408 08:07:01.243415 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/bitcoin.conf
I0408 08:07:01.363138 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json
ActiveLock 1 Read locked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368452 reader_at.go:179 zero2 [4,4096) of file size 4 bytes
I0408 08:07:01.368480 page_writer.go:54 ReadDirtyDataAt 0 [0, 4096)
ActiveLock 1 Read unlocked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368543 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json fh 0
ActiveLock 2 doFlush wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush unlocked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle unlocked 0 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368718 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp
I0408 08:07:01.368866 weedfs_attr.go:53 /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp setattr set size=0 chunks=0
ActiveLock 4 Write wait for 1 type=1 with waiters 0 active r0 w0.
ActiveLock 4 Write locked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368977 page_writer.go:34 1 AddPage [0, 4)
I0408 08:07:01.368988 dirty_pages_chunked.go:44 1 memory AddPage [0, 4)
ActiveLock 4 Write unlocked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.369174 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp fh 1
I0408 08:07:01.369184 upload_pipeline.go:171  uploaderCount 0 ++> 1
fatal error: sync: unlock of unlocked mutex
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcf2b85]

goroutine 117 [running]:
sync.fatal({0x23d8342?, 0x44?})
	/usr/local/go/src/runtime/panic.go:1061 +0x18
sync.(*Mutex).unlockSlow(0xc0005bc828, 0xffffffff)
	/usr/local/go/src/sync/mutex.go:229 +0x35
sync.(*Mutex).Unlock(0x0?)
	/usr/local/go/src/sync/mutex.go:223 +0x25
panic({0x1fbc240?, 0x3d121f0?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute(0x0, 0xc00070c520)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/limited_executor.go:32 +0x25
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed(0xc0005bc7e0, {0x2a57d90?, 0xc0006cfd80}, 0x0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:185 +0x325
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).flushChunks(0xc0005bc7e0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:159 +0xe5
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:150 +0x18
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc0006d16e0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ec840, 0xc0005fa180, 0x3e8, 0x3e8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ec840, 0x7f31eebc1f18?, 0xc0007123d8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc000712240?, 0xc000712240)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/opcode.go:402 +0x2c
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc0002d0f20, 0xc000712240)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:526 +0x267
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x0?)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:499 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest in goroutine 1
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:366 +0x53f

goroutine 1 [syscall]:
syscall.Syscall(0x4800a5?, 0xc0002d0fb8?, 0x47ffea?, 0x1e6ea60?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25
syscall.read(0xc0002d0fb8?, {0xc0019881b0?, 0xc0007e29f0?, 0xc00095d828?})
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:721 +0x38
syscall.Read(...)
	/usr/local/go/src/syscall/syscall_unix.go:181
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest.func1(...)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:334
github.com/hanwen/go-fuse/v2/fuse.handleEINTR(...)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:309
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest(0xc0002d0f20, 0xc0?)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:332 +0x1ac
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x38?)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:477 +0x7b
github.com/hanwen/go-fuse/v2/fuse.(*Server).Serve(0xc0002d0f20)
	/go/pkg/mod/github.com/hanwen/go-fuse/[email protected]/fuse/server.go:421 +0x29
github.com/seaweedfs/seaweedfs/weed/command.RunMount(0x3d4c4e0, 0x0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:271 +0x1712
github.com/seaweedfs/seaweedfs/weed/command.runMount(0x3d2fd78?, {0xc000138260?, 0x0, 0xa?})
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:55 +0x154
main.main()
	/go/src/github.com/seaweedfs/seaweedfs/weed/weed.go:80 +0x309

goroutine 19 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:884 +0x66
created by github.com/seaweedfs/seaweedfs/weed/glog.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:409 +0x1ae

goroutine 24 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 50 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/util/grace.init.0.func1()
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:38 +0x45
created by github.com/seaweedfs/seaweedfs/weed/util/grace.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:37 +0x113

goroutine 51 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001cab00)
	/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 82 [select]:
github.com/seaweedfs/seaweedfs/weed/stats.(*ServerStats).Start(0xc0000614c0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/stats/stats.go:92 +0x1c5
created by github.com/seaweedfs/seaweedfs/weed/server.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/server/common.go:41 +0x7b

goroutine 83 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9c0, {0x2a4d518, 0xc0007cde00})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 84 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9f0, {0x2a4d518, 0xc0007cde50})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 85 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cda20, {0x2a4d518, 0xc0007cdea0})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8dc0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0007c2000?, 0xc000282000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007c2000, {0xc000282000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0007c2000, {0xc000282000?, 0x1060100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005be0f0, {0xc000282000?, 0x18?, 0xc0004c3c80?})
	/usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0000b6660, {0xc0006b8120, 0x9, 0x7f31eebc1f18?})
	/usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0000b6660}, {0xc0006b8120, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0006b8120, 0x9, 0x3d4b520?}, {0x2a2dfe0?, 0xc0000b6660?})
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006b80e0)
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007e4480, 0x0?)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x1ebe

goroutine 41 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0007e4480)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:388 +0x1e4b

goroutine 43 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0006e6370, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000361c0)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x246b

goroutine 101 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc00092e1c0)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:206 +0xb6
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/util/buffer_pool.go:237 +0x196

goroutine 102 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00017c780)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59e
created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x2a6

goroutine 103 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:90 +0x12a
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:142 +0x456

goroutine 104 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:143 +0x496

goroutine 105 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:825 +0x698
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x505

goroutine 106 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:762 +0x105
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:150 +0x545

goroutine 114 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8eb8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0001cb600?, 0x20?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001cb600)
	/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc0001cb600)
	/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x448e60?)
	/usr/local/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000982d50)
	/usr/local/go/src/net/unixsock.go:260 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0004d0800, {0x2a465e0?, 0xc000982d50})
	/go/pkg/mod/google.golang.org/[email protected]/server.go:871 +0x462
created by github.com/seaweedfs/seaweedfs/weed/command.RunMount in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:264 +0x1545

goroutine 115 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0007de120)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:329 +0x7c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:344
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0006cb040, {0x21c50a0?, 0xc0005c84b0}, 0x41aae8?)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:1070 +0xc9
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x52?)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:927 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc000706fc0, 0xc000763850, 0xc000763840)
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:776 +0x13a
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000706fc0, {0x21c50a0?, 0xc0005c84b0?})
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:926 +0x125
github.com/seaweedfs/seaweedfs/weed/pb/filer_pb.(*seaweedFilerSubscribeMetadataClient).Recv(0xc0006765c0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb/filer_grpc.pb.go:294 +0x46
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x2a699b8, 0xc000984ad0})
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:78 +0x23f
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc00065c380)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:27 +0x68
github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xaf?, 0x5bfc52d2?, 0xc000763ce8, {0xc000129290, 0x26}, 0x0?, {0xc000094cd8?, 0x1?, 0x1?})
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/grpc_client_server.go:155 +0x348
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1()
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:25 +0x15e
github.com/seaweedfs/seaweedfs/weed/util.Retry({0x23a1371, 0xa}, 0xc000763df8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:16 +0xa7
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0x0?, 0x0?, 0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:18 +0x65
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x2a45f20, 0xc0006ec840}, 0xc0007c3980, 0xc0007e2660)
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:50 +0x87
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2()
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:74 +0x26
github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x23b086f, 0x11}, 0xc000763f60, 0x24e8368)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:64 +0x4e
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc0007fafc0, 0x5bfc52d2, {0x2a45f20?, 0xc0006ec840}, {0x7ffcab9b563e, 0x31}, 0x17c43f89c0ca0d72)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:72 +0x185
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:123 +0x152

goroutine 116 [sleep]:
time.Sleep(0xe33e22200)
	/usr/local/go/src/runtime/time.go:195 +0x125
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).loopCheckQuota(0xc0006ec840)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_quota.go:15 +0x31
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:124 +0x18e

goroutine 70 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847b0, {0x2a4d518, 0xc0000c13b0})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 71 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847e0, {0x2a4d518, 0xc0000c1450})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 72 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000984810, {0x2a4d518, 0xc0000c14a0})
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 91 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:391 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 115
	/go/pkg/mod/google.golang.org/[email protected]/stream.go:390 +0xe9c

goroutine 88 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0004d66c0)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:388 +0x1e4b

goroutine 89 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8cc8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005ba000?, 0xc000786000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005ba000, {0xc000786000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005ba000, {0xc000786000?, 0x60100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000782000, {0xc000786000?, 0x40b3fe?, 0x4726b2?})
	/usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0007faba0, {0xc00092e040, 0x9, 0xc0?})
	/usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0007faba0}, {0xc00092e040, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc00092e040, 0x9, 0x20b6300?}, {0x2a2dfe0?, 0xc0007faba0?})
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00092e000)
	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0004d66c0, 0x19320b2801200218?)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:398 +0x1ebe

goroutine 90 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002f1e50, 0x1)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000812000)
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:450 +0x246b

goroutine 78 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8bd0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005fa100?, 0xc0005fe000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005fa100, {0xc0005fe000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005fa100, {0xc0005fe000?, 0x751120?, 0xc0005cc000?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0007c6108, {0xc0005fe000?, 0x0?, 0xc0005036c0?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc000594480, {0xc0005fe000?, 0xc0001036e0?, 0xc000099d38?})
	/usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).fill(0xc0006d15c0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0006d15c0, 0x1)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*persistConn).readLoop(0xc000594480)
	/usr/local/go/src/net/http/transport.go:2118 +0x1b9
created by net/http.(*Transport).dialConn in goroutine 119
	/usr/local/go/src/net/http/transport.go:1776 +0x169f

goroutine 79 [select]:
net/http.(*persistConn).writeLoop(0xc000594480)
	/usr/local/go/src/net/http/transport.go:2421 +0xe5
created by net/http.(*Transport).dialConn in goroutine 119
	/usr/local/go/src/net/http/transport.go:1777 +0x16f1
E0408 08:07:01.375783 mounter.go:75 weed mount exit, pid: 23, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/3fbaadb3372904bcd46022cd9d47f5f22451cea3793cf1b8c028f2d721a0287b/globalmount, error: exit status 2
I0408 08:07:16.502410 utils.go:63 GRPC /csi.v1.Identity/Probe request
@spastorclovr spastorclovr changed the title Wen setting concurrentWriters=0 A fatal exception occurs - SIGSEGV When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@spastorclovr