Skip to content

[Enhancement] massive memory allocation in the parsePublishMessageQueues #9926

@ShiKaiWi

Description

@ShiKaiWi

Before Creating the Enhancement Request

  • I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

During my benchmark (50K LMQ tps), the rocketmq client in my process shows massive memory allocation in the parsePublishMessageQueues.

Here is my profiling result about the memory allocation:

Image

Motivation

This multiple growths of the array list introduce a great influence on the performance under the high throughput workload.

Describe the Solution You'd Like

Here is a simple proposal:

  • do pre-allocation for the returned list of the parsePublishMessageQueues.
  • Just return the original List<MessageQueue> if the namespace is empty.

Describe Alternatives You've Considered

There are more complex solutions:

  • Upgrade the interface about the MessageQueueSelector to use size of the message queue and accessor by index to MessageQueue instead of List<MessageQueue>, so that no need to create another List<MessageQueue>.
  • Use thread-local buffer array list for the parsed List<MessageQueue> to avoid allocating memory for any to-be-sent message.

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions