Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Add custom operator support #3055

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

nwangtw
Copy link
Contributor

@nwangtw nwangtw commented Oct 24, 2018

This is a replacement of the following PR (it can't be reopen after force push...):
#3049

This is the implementation of CustomOperator based on this design doc:
https://docs.google.com/document/d/1XzF0IlfuaaW8Gx3cPx1xLtP-kgCFK0TRNS5aAzuMuMg/edit#

CustomOperator and CustomOperatorOutput are included in this PR.

Future works:

  • add/refactor windowed custom operator
  • refactor/simplify existing operators (v2 design in doc)

@nwangtw nwangtw force-pushed the nwang/add_custom_operator_refactor branch from d8d7cd5 to b9e889b Compare November 8, 2018 23:01
@nwangtw nwangtw changed the title Nwang/add custom operator refactor Add custom operator support Nov 9, 2018
@nwangtw nwangtw requested review from jerrypeng and nlu90 November 9, 2018 00:38
@jerrypeng
Copy link
Contributor

@nwangtw can you please provide an example topology that uses the custom operator? Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API? Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt)? So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

@nwangtw
Copy link
Contributor Author

nwangtw commented Nov 12, 2018

can you please provide an example topology that uses the custom operator?

Sure.

Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API? Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt). So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

Not really in this step/PR.

The goal of this step is to provide a way for users to write their own operators when needed. The low level Bolt is supported by previous PRs.

@jerrypeng
Copy link
Contributor

@nwangtw If that is the case. I don't think that is appropriate then. Streamlet aims to break away from the spout and bolt API. With this change we are kind of bring the streamlet API back to the bolt and spout API.

@nwangtw
Copy link
Contributor Author

nwangtw commented Nov 13, 2018

It does blur the line a bit to allow users to write their own operators with all functions available in low level API. However these operators still has type safety and can be used the same way as other pre-defined operators. Therefore, from API point of view, I don't think it "bring the streamlet API back to the bolt and spout API." Instead, it brings more powers from the bolt and spout API to streamlet.

Basically Streamlet is a functional API, and function API is overall nice and elegant. However I believe, a functional API is HARDLY a general purpose API and it is only good for "nice" problems. It might enforce uses to change the way of thinking and solving problems, which can be good sometimes (use general/standard operations) and bad in other times (more like hacks). Therefore I think supporting custom operator is important for Heron.

That being said, the original feature I needed was to reuse existing Bolts and Spouts which has been completed. This design doc and PR are just an extension from the original feature so it is not as urgent and we still have time to discuss.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants