Skip to content
This repository has been archived by the owner on Apr 17, 2024. It is now read-only.

[RPMP] Integration with RPMP #72

Open
wants to merge 15 commits into
base: wip_rpmp
Choose a base branch
from

Conversation

xuechendi
Copy link
Collaborator

This PR depends on HPNL branch https://github.com/Intel-bigdata/HPNL/tree/wip_rpmp

Changelog:

  1. Update RPMP codes to use shared_ptr instead of raw ptr more better memory management
  2. Add a new benchmark "remote_put" who will imitate what shuffle write and read did. do put/getMeta/read
  3. RPMP client now is using a queue to make requests from multiple threads become one.
  4. shuffle read used RPMP read api
  5. shuffle metadata update is still using original PmofTransferService to contact driver.
  6. External sort is not supported by RPMP yet, will use local filesystem to do sort.

xuechendi and others added 15 commits May 19, 2020 18:19
Signed-off-by: Chendi.Xue <[email protected]>
1. new threading model in NetworkClient
2. remove jni h file
3. add a jniUtil to load so from jar

Signed-off-by: Chendi.Xue <[email protected]>
…estructor of client pmpool

shared_ptr issue seems to be caused since there is three threads in client, one jvm client, one client worker and one callback worker. So now we will std::move callback replied replyrequestContext to main worker. And shared_ptr release issue seems gone.

Signed-off-by: Chendi.Xue <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant