Scaling Vuu - How to abstract Vuu interfaces to allow different implementations of key classes by adding "features" on construction.

[chrisjstevo]

Introduction

Vuu is currently a largely in memory, near caching solution for UIs. In order to allow Vuu to scale to large data sets (10bn rows) we cannot cache all the data in memory, we need to be able to use big data stores, and move a remote cursor around in that data set as users scroll.

In order to facilitate this requirement we have started to think about Vuu rather than as a deeply entwined 'core' library, but more as a series of internal interfaces. Some examples of these interfaces are:

DataTable
ViewPort
ViewPortKeys() //work in progress
Provider
JoinManager
LoginValidator
Authenticator

If we take the first 5 interfaces, these relate to how we store and join data in memory. When we move to a paradigm where all the data is not in memory, but simply a window of the data is in memory, we will need to provide different implementations of these interfaces to facilitate this.

Features & Plugins

Features, as described here, are a way of grouping alternative implementations to some or all of the above interfaces with non standard implementations. For example, we may want to have the situation where we want to back Tables and Viewports by a Big Data store. In this case we would want to have some construct, conceptually like below, where a user could "plugin" a different implementation to use.

Example:

  //in a normal config object
  val config = VuuServerConfig(
    VuuHttp2ServerOptions()
      //only specify webroot if we want to load the source locally, we'll load it from the jar
     //</snip>
    VuuSecurityOptions()
      .withAuthenticator(authenticator)
      .withLoginValidator(new AlwaysHappyLoginValidator),
    VuuThreadingOptions()
      .withViewPortThreads(4)
      .withTreeThreads(4)
  ).withModule(PriceModule())
    .withModule(BasketModule())
    //add big data plugin 
    withPlugin(BigDataTableAndViewportPlugin())

We would then have a plugin type, that would allow the provision of some or all of these interfaces:

class BigDataTableAndViewportPlugin{
     
         registerFeature[DataTable](BigDataTableFeature())
         registerFeature[JoinManager](BigDataJoinManagerFeature())
}

[rumakt]

Agreed,

I would also add:

1. Metrics interface
2. Documentation to the forementioned interfaces + common code
3. Some sort of acceptance test suite so an implementer of a plugin is able to confirm that a plugin is working as intented.

[chrisjstevo]

I like this, especially number 3. I'm interested to see how it would work with the Big Data Impl. Maybe its simple in that we just load (a small amount, say) 2000 rows into a test big data store running locally and we move around in the container and check it works.

[chrisjstevo]

In addition to the previous list of interfaces:

DataTable
ViewPort
ViewPortKeys() //work in progress
Provider
JoinManager
LoginValidator
Authenticator

We would also need:

Filter
Sort

This would be required as the filter and sort functionality would be delegated down to the big data solution, rather than performed in memory.

[naleeha]

1. The concept & naming of Plugin sounds good. The objectives will be to give ability for users to opt in to

• Vuu implementation related to the plugin purpose
• Any third party client library require by the plugin
• Other related useful tools - like the metrics, testing mentioned above

2. Scope of plugins are a bit less clear to me - we could have concept of VUU Big data solution with VUU implementation for using Virtual ViewportKeys and VirtualDataTable. This might be reused by different big data solutions - e.g. Apache Ignite
   For example we could have

• IgnitePlugin

  • VuuBigData - VirtualDataTable, VirtualViewportKeys
  • IgniteAdapters - DataSourceAdapters, QueryAdapters
  • Ignite related tools

• HadoopPlugin

  • VuuBigData - VirtualDataTable, VirtualViewportKeys
  • HadoopAdapters -DataSourceAdapters, QueryAdapters
  • Hadoop related tools

3. I'm not sure we need the concept of Feature yet - especially at the level of filter, sort, groupby, aggregate etc
   given the Virtual vs In-memory use case, we are unlikely to mix and match different Virtual solutions with In-memory solutions.
   Also the word feature currently is used to describe grouping of functionality on the UI side so it is a bit confusing. - e.g. VUU table, VUU Basket Trading

[naleeha]

In context of project structure for plugins and big data

Currently

+- Vuu (server)
    +- core tables
    ...
+- plugin
    +- virtualized-table-plugin
    +- ignite-plugin

This means when we want to test some of the core functionalities for the virtual table use case, we can't have the test in the vuu core area - e.g. test for type ahead RPC function using in memory is under Vuu but type ahead RPC function for virtual table has to be in the virtualized-table-plugin
Also core logic of vuu cannot know about the virtual table - do different things based on table /provider types

Proposal

+- Vuu (server)
    +- core tables
    +- virtualized-table
    ...
+- plugin
    +- ignite-plugin

This ensures that

• all core functionality we test are tested for all use cases in one place(core and virtual table)
• apache ignite dependency remains separated from the main vuu package