[fix] [broker] Msg delivery is stuck due to items in the collection recentlyJoinedConsumers are out-of-order #22393
Triggered via pull request
December 31, 2024 10:45
Status
Failure
Total duration
1h 11m 24s
Artifacts
27
pulsar-ci.yaml
on: pull_request
Preconditions
12s
Matrix: unit-tests
Matrix: Build Pulsar java-test-image docker image
CI - Unit - Upload Coverage
0s
Run CodeQL Analysis
0s
Matrix: system-tests
Matrix: flaky-system-tests
Matrix: integration-tests
Delete system test docker image artifact
9s
Delete integration test docker image artifact
11s
Pulsar CI checks completed
5s
Annotations
5 errors and 60 warnings
CI - Unit - Brokers - Broker Group 1
Process completed with exit code 1.
|
ZkSessionExpireTest.testTopicUnloadAfterSessionRebuild:
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ZkSessionExpireTest.java#L163
Assertion condition expected [true] but found [false] within 10 seconds.
|
CI - Unit - Brokers - Client Impl
Process completed with exit code 1.
|
KeySharedSubscriptionTest.testRecentJoinedPosWillNotStuckOtherConsumer:
pulsar-broker/src/test/java/org/apache/pulsar/client/api/KeySharedSubscriptionTest.java#L2068
--- An unexpected error occurred in the server ---
Message: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null
Stacktrace:
org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null
at org.apache.pulsar.broker.service.persistent.PersistentTopic$5.deleteCursorFailed(PersistentTopic.java:1368)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationFailed(ManagedLedgerImpl.java:1076)
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncRemoveCursor$20(MetaStoreImpl.java:296)
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203)
at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null
at java.base/java.util.HashMap.putMapEntries(HashMap.java:503)
at java.base/java.util.LinkedHashMap.<init>(LinkedHashMap.java:480)
at org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumersClassic.getRecentlyJoinedConsumers(PersistentStickyKeyDispatcherMultipleConsumersClassic.java:597)
at org.apache.pulsar.broker.service.persistent.PersistentSubscription.getStatsAsync(PersistentSubscription.java:1337)
at org.apache.pulsar.broker.service.persistent.PersistentTopic.removeSubscription(PersistentTopic.java:1378)
at org.apache.pulsar.broker.service.persistent.PersistentTopic$5.deleteCursorComplete(PersistentTopic.java:1352)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationComplete(ManagedLedgerImpl.java:1070)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationComplete(ManagedLedgerImpl.java:1060)
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncRemoveCursor$19(MetaStoreImpl.java:286)
at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203)
at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
|
Pulsar CI checks completed
Process completed with exit code 1.
|
CI - Unit - Proxy
Summary: Tests in class org.apache.pulsar.proxy.server.ProxyStuckConnectionTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Proxy
Summary: Tests in class org.apache.pulsar.proxy.server.ProxyIsAHttpProxyTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Proxy
Summary: Tests in class org.apache.pulsar.proxy.server.ProxyDisableZeroCopyTest created 1040 new threads. There are now 1050 threads in total.
|
CI - Unit - Proxy
Summary: Tests in class org.apache.pulsar.proxy.server.ProxyTest created 1040 new threads. There are now 1049 threads in total.
|
CI - Unit - Brokers - Broker Group 4
Summary: Tests in class org.apache.pulsar.broker.service.ClusterMigrationTest created 2 new threads. There are now 11 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.ElasticSearchBWCTests created 6 new threads. There are now 15 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8ClientTests created 10 new threads. There are now 25 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8AuthTests created 11 new threads. There are now 35 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8SinkTests created 1 new threads. There are now 34 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8SinkRawDataTests created 3 new threads. There are now 37 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.ElasticSearchRawRecordTests created 96 new threads. There are now 133 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v7.ElasticSearch7ClientTests created 1 new threads. There are now 134 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.opensearch.OpenSearchClientTests created 3 new threads. There are now 136 threads in total.
|
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.opensearch.OpenSearchClientSslTests created 18 new threads. There are now 151 threads in total.
|
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.PulsarBrokerStarterTest created 4 new threads. There are now 13 threads in total.
|
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.stats.TransactionBatchWriterMetricsTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.service.BrokerServiceTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.service.TransactionMetadataStoreServiceTest created 72 new threads. There are now 85 threads in total.
|
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.service.ReplicatorTest created 631 new threads. There are now 640 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.zookeeper.LocalBookkeeperEnsembleTest created 41 new threads. There are now 50 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.zookeeper.ZKReconnectTest created 3 new threads. There are now 12 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.SameAuthParamsLookupAutoClusterFailoverTest created 5 new threads. There are now 52 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.delayed.DelayedDeliveryTrackerFactoryTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.EndToEndMetadataTest created 1 new threads. There are now 50 threads in total.
|
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.resourcegroup.ResourceGroupConfigListenerTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Brokers - Client Impl
Summary: Tests in class org.apache.pulsar.client.impl.ProduceWithMessageIdTest created 68 new threads. There are now 77 threads in total.
|
CI - Unit - Brokers - Client Impl
Summary: Tests in class org.apache.pulsar.client.impl.ReaderTest created 8 new threads. There are now 17 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.pulsar.metadata.MetadataStoreTest created 8 new threads. There are now 18 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorCheckAllLedgersTaskTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.BookieAutoRecoveryTest created 14 new threads. There are now 24 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorBookieTest created 27 new threads. There are now 37 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.TestReplicationWorker created 43 new threads. There are now 53 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorRollingRestartTest created 47 new threads. There are now 57 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AutoRecoveryMainTest created 21 new threads. There are now 31 threads in total.
|
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorPeriodicCheckTest created 11 new threads. There are now 21 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.http.HttpSinkTest created 74 new threads. There are now 83 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.kinesis.KinesisSinkTest created 9 new threads. There are now 19 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.text.HdfsStringSinkTests created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.seq.HdfsSequentialSinkTests created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.seq.HdfsTextSinkTests created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.flume.node.TestPollingZooKeeperConfigurationProvider created 4 new threads. There are now 13 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.flume.source.StringSourceTests created 20 new threads. There are now 29 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.flume.sink.StringSinkTests created 16 new threads. There are now 25 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.redis.sink.RedisSinkTest created 11 new threads. There are now 21 threads in total.
|
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.solr.SolrGenericRecordSinkTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.TenantEndpointsAuthorizationTest created 8 new threads. There are now 17 threads in total.
|
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.ClusterEndpointsAuthorizationTest created 12 new threads. There are now 21 threads in total.
|
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.BrokerEndpointsAuthorizationTest created 12 new threads. There are now 29 threads in total.
|
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.AdminRestTest created 1 new threads. There are now 30 threads in total.
|
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.AdminApiTest created 1 new threads. There are now 22 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMappingTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest created 16 new threads. There are now 26 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryShutdownTest created 8 new threads. There are now 30 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.impl.ManagedCursorTest created 5 new threads. There are now 15 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.offload.filesystem.impl.FileSystemOffloaderLocalFileTest created 3 new threads. There are now 14 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.offload.filesystem.impl.FileSystemManagedLedgerOffloaderTest created 6 new threads. There are now 16 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.functions.utils.FunctionCommonTest created 1 new threads. There are now 10 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.functions.worker.rest.api.v3.FunctionApiV3ResourceTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.functions.worker.rest.api.v2.FunctionApiV2ResourceTest created 1 new threads. There are now 11 threads in total.
|
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.testclient.PerformanceProducerTest created 8 new threads. There are now 18 threads in total.
|
Artifacts
Produced during runtime
Name | Size | |
---|---|---|
Jacoco-coverage-report-inttests
Expired
|
24 MB |
|
Jacoco-coverage-report-systests
Expired
|
24.5 MB |
|
Unit-BROKER_CLIENT_IMPL-dumps
|
13.7 KB |
|
Unit-BROKER_CLIENT_IMPL-surefire-reports
|
20.1 MB |
|
Unit-BROKER_GROUP_1-dumps
|
71.6 KB |
|
Unit-BROKER_GROUP_1-surefire-reports
|
21.3 MB |
|
Unit-BROKER_GROUP_2-dumps
|
32 KB |
|
Unit-BROKER_GROUP_3-dumps
|
24.7 KB |
|
Unit-BROKER_GROUP_4-dumps
|
4.17 KB |
|
Unit-METADATA-dumps
|
48.4 KB |
|
Unit-OTHER-dumps
|
66.5 KB |
|
Unit-PROXY-dumps
|
119 KB |
|
Unit-PULSAR_IO-dumps
|
60.5 KB |
|
Unit-PULSAR_IO_ELASTIC-dumps
|
72.6 KB |
|
coverage_and_deps_unittest_BROKER_CLIENT_API.tar.zst
Expired
|
2.25 MB |
|
coverage_and_deps_unittest_BROKER_GROUP_2.tar.zst
Expired
|
2.35 MB |
|
coverage_and_deps_unittest_BROKER_GROUP_3.tar.zst
Expired
|
2.27 MB |
|
coverage_and_deps_unittest_BROKER_GROUP_4.tar.zst
Expired
|
2.15 MB |
|
coverage_and_deps_unittest_CLIENT.tar.zst
Expired
|
660 KB |
|
coverage_and_deps_unittest_METADATA.tar.zst
Expired
|
264 KB |
|
coverage_and_deps_unittest_OTHER.tar.zst
Expired
|
10 MB |
|
coverage_and_deps_unittest_PROXY.tar.zst
Expired
|
212 KB |
|
coverage_and_deps_unittest_PULSAR_IO.tar.zst
Expired
|
366 KB |
|
coverage_and_deps_unittest_PULSAR_IO_ELASTIC.tar.zst
Expired
|
91.1 KB |
|
coverage_and_deps_unittest_PULSAR_IO_KAFKA_CONNECT.tar.zst
Expired
|
205 KB |
|
pulsar-maven-repository-binaries.tar.zst
Expired
|
318 MB |
|
pulsar-server-distribution.tar.zst
Expired
|
228 MB |
|