Design Discussions Join Manager v2

Design Discussions - Join Manager v2

Currently join logic is the responsibility of two components, the join manager that receives events from source tables, it stores a link between a source table (such as instruments) and a join table (such as instrumentprices) that requires it. In v1 of the Join Manager Esper provides the ability to map source events to their dependent join tables and to generate 1 or n events to propogate the tick:

Use Cases for Join Manager

1 - Left Outer Joins

Tick one to one or many:

Prices

Ric	Bid	Ask
VOD.L	100	101
BT.L	310	311

Orders

OrderId	Ric	Quantity
1	VOD.L	1000
2	VOD.L	5000
3	BT.L	7000

In this example assuming order id's 1,2 & 3 have already ticked through the join manager as soon as a new tick comes in for VOD.L, we would expect the input in the join manager to be:

instrumentsEvent: { "prices.ric" -> VOD.L "prices._isDeleted" -> False }

The output from the join manager would be:

{ "prices.ric" -> "VOD.L", "prices._isDeleted" -> False, "orders.orderId" -> 1, "orders._isDeleted" -> False, }

{ "prices.ric" -> "VOD.L", "prices._isDeleted" -> False, "orders.orderId" -> 2, "orders._isDeleted" -> False, }

{ "prices.ric" -> "VOD.L", "prices._isDeleted" -> False, "orders.orderId" -> 3, "orders._isDeleted" -> False, }

2 - Left Inner Joins

In the example of the use case above, inner joins would suppress events going to the join table until bother sides are satisfied. So if there were no orders in the table, we would not generate an event anyway to the join table for the price tick.

3 - Unions (do we need this..? probably not)

4 Compound Key Joins (Do we need this, also..? CompoundKeys are really just n single values concatenated together in a hash)

5 - Dynamic Session Tables

On of the big downsides of using Esper is that it wants us to register on startup all the event types we're interested in. So for example, if a user wanted to have a temporary table (orderInput for example) that was created when she started entering data, that lived in her viewport while she was entering the data, and then was removed when she'd completed the transaction, at the moment this would be provided by a session table. Session tables are by nature transient things, but what if while I was entering orders into my orderInput session table, I also wanted to see live prices ticking. In this case we would create an orderInputPrices join typically, however because orderInput is local to each users viewport this wouldn't work.

We can resolve this in the new join manager by:

1. Allowing new joins to be added to the join manager at runtime, as we do with tables in the table manager.

   joinManager.addSessionJoinDefinition(table: JoinSessionTable)

2. When a new table is added, we tick through any relevant values from the join (prices as an example) into the join table sink.

3. When viewport is expired we remove the session table from the join manager.

Data Structures

For every join we have in the system we will need to store the relevant keys on each side. For example in the prices/orders example above we would need a data structure that would be something like:

val pricesJoinMap = Map["Orders" -> Map[ "VOD.L", [ 1, ,2 ] ]

and the converse:

val ordersJoinMap = Map["Prices", -> Map[ 1 -> "VOD.L", 2 -> "VOD.L" ] ]

Event Data Flow

When we flow data into the join manager as events the sequence will be:

1. Add event object into inbound queue of join manager
2. Check if joins exist for event (i.e if a prices event, do we have orders for that..?) if we have more than one join we have to iterate them all.
3. When we've found the joins, we need to check whether the cache of data contains any keys, if it does, we need to tick the resulting map for each join row through to the outbound queue.