Gives the previous cursor in the scroll block (#38)

* Gives the previous cursor in the scroll block * apply changes for mongo * encrypt with the previous option * take values before the first page * keep the ordering when fetching previous records * keep the ordering when fetching previous records * add entry to the change log * Fix with Ruby 2.6 * Update the README and CHANGELOG * minor changes * change to use type * change to use an iterator object * refactor * Fix typos Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> * throw an error when type is unsupported --------- Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>
mongoid · Aug 29, 2024 · 56f04f5 · 56f04f5
1 parent ddedd8c
commit 56f04f5
Show file tree

Hide file tree

Showing 17 changed files with 270 additions and 89 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,6 @@
-### 1.0.2 (Next)
+### 2.0.0 (Next)
 
+* [#38](https://github.com/mongoid/mongoid-scroll/pull/38): Allow to reverse the scroll - [@GCorbel](https://github.com/GCorbel).
 * Your contribution here.
 
 ### 1.0.1 (2023/03/15)

diff --git a/README.md b/README.md
@@ -69,27 +69,38 @@ end
 Scroll by `:position` and save a cursor to the last item.
 
 ```ruby
-saved_cursor = nil
-Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
+saved_iterator = nil
+
+Feed::Item.desc(:position).limit(5).scroll do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
-Resume iterating using the previously saved cursor.
+Resume iterating using saved cursor and save the cursor to go backward.
+
+```ruby
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
+  # each record, one-by-one
+  saved_iterator = iterator
+end
+```
+
+Loop over the first records again.
 
 ```ruby
-Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.previous_cursor) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
 The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.
 
 ```ruby
-Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll(saved_iterator.next_cursor) do |record, iterator|
   # each record, one-by-one
+  saved_iterator = iterator
 end
 ```
 
@@ -98,19 +109,19 @@ end
 Scroll a `Mongo::Collection::View` and save a cursor to the last item. You must also supply a `field_type` of the sort criteria.
 
 ```ruby
-saved_cursor = nil
-client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
+saved_iterator = nil
+client[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
 Resume iterating using the previously saved cursor.
 
 ```ruby
-session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
+session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_iterator.next_cursor, { field_type: DateTime }) do |record, iterator|
   # each record, one-by-one
-  saved_cursor = next_cursor
+  saved_iterator = iterator
 end
 ```
 
@@ -179,15 +190,15 @@ Feed::Item.desc(:created_at).scroll(cursor) # Raises a Mongoid::Scroll::Errors::
 
 ### Standard Cursor
 
-The `Mongoid::Scroll::Cursor` encodes a value and a tiebreak ID separated by `:`, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options. 
+The `Mongoid::Scroll::Cursor` encodes a value and a tiebreak ID separated by `:`, and does not include other options, such as scroll direction. Take extra care not to pass a cursor into a scroll with different options.
 
 ### Base64 Encoded Cursor
 
 The `Mongoid::Scroll::Base64EncodedCursor` can be used instead of `Mongoid::Scroll::Cursor` to generate a base64-encoded string (using RFC 4648) containing all the information needed to rebuild a cursor.
 
 ```ruby
-Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, next_cursor|
-   # next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
+Feed::Item.desc(:position).limit(5).scroll(Mongoid::Scroll::Base64EncodedCursor) do |record, iterator|
+   # iterator.next_cursor is of type Mongoid::Scroll::Base64EncodedCursor
 end
 ```
 

diff --git a/UPGRADING.md b/UPGRADING.md
@@ -1,5 +1,25 @@
 # Upgrading
 
+## Upgrading to >= 2.0.0
+
+The second argument yielded in the block in `Mongoid::Criteria::Scrollable#scroll` and `Mongo::Scrollable#scroll` has changed from a cursor to an instance of `Mongoid::Criteria::Scrollable` which provides `next_cursor` and `previous_cursor`. The `next_cursor` method returns the same cursor as in versions prior to 2.0.0.
+
+For example, this code:
+
+```ruby
+Feed::Item.asc(field_name).limit(2).scroll(cursor) do |_, next_cursor|
+  cursor = next_cursor
+end
+```
+
+Should be updated to:
+
+```
+Feed::Item.asc(field_name).limit(2).scroll(cursor) do |_, iterator|
+  cursor = iterator.next_cursor
+end
+```
+
 ## Upgrading to >= 1.0.0
 
 ### Mismatched Sort Fields
@@ -9,6 +29,6 @@ Both `Mongoid::Criteria::Scrollable#scroll` and `Mongo::Scrollable` now raise a
 For example, the following code will now raise a `MismatchedSortFieldsError` because we set a different field name (`position`) from the `created_at` field used to sort in `scroll`.
 
 ```ruby
-cursor.field_name = "position" 
+cursor.field_name = "position"
 Feed::Item.desc(:created_at).scroll(cursor)
 ```
diff --git a/lib/config/locales/en.yml b/lib/config/locales/en.yml
@@ -27,4 +27,7 @@ en:
             message: "Unsupported field type."
             summary: "The type of the field '%{field}' is not supported: %{type}."
             resolution: "Please open a feature request in https://github.com/mongoid/mongoid-scroll."
-
+          unsupported_type:
+            message: "Unsupported type."
+            summary: "The type supplied in the cursor is not supported: %{type}."
+            resolution: "The cursor type can be either ':previous' or ':next'."
diff --git a/lib/mongo/scrollable.rb b/lib/mongo/scrollable.rb
@@ -16,21 +16,41 @@ def scroll(cursor_or_type = nil, options = nil, &_block)
       cursor_options = { field_name: scroll_field, direction: scroll_direction }.merge(options)
       cursor = cursor && cursor.is_a?(cursor_type) ? cursor : cursor_type.new(cursor, cursor_options)
       raise_mismatched_sort_fields_error!(cursor, cursor_options) if different_sort_fields?(cursor, cursor_options)
-      # make a view
-      view = Mongo::Collection::View.new(
-        view.collection,
-        view.selector.merge(cursor.criteria),
-        sort: (view.sort || {}).merge(_id: scroll_direction),
-        skip: skip,
-        limit: limit
-      )
+
+      records = nil
+      if cursor.type == :previous
+        # scroll backwards by reversing the sort order, limit and then reverse again
+        pipeline = [
+          { '$match' => view.selector.merge(cursor.criteria) },
+          { '$sort' => { scroll_field => -scroll_direction } },
+          { '$limit' => limit },
+          { '$sort' => { scroll_field => scroll_direction } }
+        ]
+        aggregation_options = view.options.except(:sort)
+        records = view.aggregate(pipeline, aggregation_options)
+      else
+        # make a view
+        records = Mongo::Collection::View.new(
+          view.collection,
+          view.selector.merge(cursor.criteria),
+          sort: (view.sort || {}).merge(_id: scroll_direction),
+          skip: skip,
+          limit: limit
+        )
+      end
       # scroll
       if block_given?
-        view.each do |record|
-          yield record, cursor_type.from_record(record, cursor_options)
+        previous_cursor = nil
+        records.each do |record|
+          previous_cursor ||= cursor_type.from_record(record, cursor_options.merge(type: :previous))
+          iterator = Mongoid::Criteria::Scrollable::Iterator.new(
+            previous_cursor: previous_cursor,
+            next_cursor: cursor_type.from_record(record, cursor_options)
+          )
+          yield record, iterator
         end
       else
-        view
+        records
       end
     end
   end

diff --git a/lib/mongoid-scroll.rb b/lib/mongoid-scroll.rb
@@ -11,5 +11,6 @@
 require 'mongoid/scroll/base64_encoded_cursor'
 require 'mongoid/criteria/scrollable/fields'
 require 'mongoid/criteria/scrollable/cursors'
+require 'mongoid/criteria/scrollable/iterator'
 require 'mongo/scrollable' if Object.const_defined?(:Mongo)
 require 'mongoid/criteria/scrollable'
diff --git a/lib/mongoid/criteria/scrollable.rb b/lib/mongoid/criteria/scrollable.rb
@@ -12,13 +12,19 @@ def scroll(cursor_or_type = nil, &_block)
         cursor_options = build_cursor_options(criteria)
         cursor = cursor.is_a?(cursor_type) ? cursor : new_cursor(cursor_type, cursor, cursor_options)
         raise_mismatched_sort_fields_error!(cursor, cursor_options) if different_sort_fields?(cursor, cursor_options)
-        cursor_criteria = build_cursor_criteria(criteria, cursor)
+        records = find_records(criteria, cursor)
         if block_given?
-          cursor_criteria.order_by(_id: scroll_direction(criteria)).each do |record|
-            yield record, cursor_from_record(cursor_type, record, cursor_options)
+          previous_cursor = nil
+          records.each do |record|
+            previous_cursor ||= cursor_from_record(cursor_type, record, cursor_options.merge(type: :previous))
+            iterator = Mongoid::Criteria::Scrollable::Iterator.new(
+              previous_cursor: previous_cursor,
+              next_cursor: cursor_from_record(cursor_type, record, cursor_options)
+            )
+            yield record, iterator
           end
         else
-          cursor_criteria
+          records
         end
       end
 
@@ -60,10 +66,21 @@ def new_cursor(cursor_type, cursor, cursor_options)
         cursor_type.new(cursor, cursor_options)
       end
 
-      def build_cursor_criteria(criteria, cursor)
+      def find_records(criteria, cursor)
         cursor_criteria = criteria.dup
         cursor_criteria.selector = { '$and' => [criteria.selector, cursor.criteria] }
-        cursor_criteria
+        if cursor.type == :previous
+          pipeline = [
+            { '$match' => cursor_criteria.selector },
+            { '$sort' => { cursor.field_name => -cursor.direction } },
+            { '$limit' => criteria.options[:limit] },
+            { '$sort' => { cursor.field_name => cursor.direction } }
+          ]
+          aggregation = cursor_criteria.view.aggregate(pipeline)
+          aggregation.map { |record| Mongoid::Factory.from_db(cursor_criteria.klass, record) }
+        else
+          cursor_criteria.order_by(_id: scroll_direction(criteria))
+        end
       end
 
       def cursor_from_record(cursor_type, record, cursor_options)

diff --git a/lib/mongoid/criteria/scrollable/iterator.rb b/lib/mongoid/criteria/scrollable/iterator.rb
@@ -0,0 +1,14 @@
+module Mongoid
+  class Criteria
+    module Scrollable
+      class Iterator
+        attr_accessor :previous_cursor, :next_cursor
+
+        def initialize(previous_cursor:, next_cursor:)
+          @previous_cursor = previous_cursor
+          @next_cursor = next_cursor
+        end
+      end
+    end
+  end
+end
diff --git a/lib/mongoid/scroll/base64_encoded_cursor.rb b/lib/mongoid/scroll/base64_encoded_cursor.rb
@@ -18,7 +18,8 @@ def initialize(value, options = {})
             field_name: parsed['field_name'],
             direction: parsed['direction'],
             include_current: parsed['include_current'],
-            tiebreak_id: parsed['tiebreak_id'] && !parsed['tiebreak_id'].empty? ? BSON::ObjectId.from_string(parsed['tiebreak_id']) : nil
+            tiebreak_id: parsed['tiebreak_id'] && !parsed['tiebreak_id'].empty? ? BSON::ObjectId.from_string(parsed['tiebreak_id']) : nil,
+            type: parsed['type'].try(:to_sym)
           }
         else
           super nil, options
@@ -32,7 +33,8 @@ def to_s
           field_name: field_name,
           direction: direction,
           include_current: include_current,
-          tiebreak_id: tiebreak_id && tiebreak_id.to_s
+          tiebreak_id: tiebreak_id && tiebreak_id.to_s,
+          type: type
         }.to_json)
       end
     end

diff --git a/lib/mongoid/scroll/base_cursor.rb b/lib/mongoid/scroll/base_cursor.rb
@@ -1,7 +1,7 @@
 module Mongoid
   module Scroll
     class BaseCursor
-      attr_accessor :value, :tiebreak_id, :field_type, :field_name, :direction, :include_current
+      attr_accessor :value, :tiebreak_id, :field_type, :field_name, :direction, :include_current, :type
 
       def initialize(value, options = {})
         @value = value
@@ -10,6 +10,9 @@ def initialize(value, options = {})
         @field_name = options[:field_name]
         @direction = options[:direction] || 1
         @include_current = options[:include_current] || false
+        @type = options[:type] || :next
+
+        raise Mongoid::Scroll::Errors::UnsupportedTypeError.new(type: @type) if ![:previous, :next].include?(@type)
       end
 
       def criteria
@@ -86,20 +89,23 @@ def extract_field_options(options)
             field_type: field_type.to_s,
             field_name: field_name.to_s,
             direction: options[:direction] || 1,
-            include_current: options[:include_current] || false
+            include_current: options[:include_current] || false,
+            type: options[:type].try(:to_sym) || :next
           }
         elsif options && (field = options[:field])
           {
             field_type: field.type.to_s,
             field_name: field.name.to_s,
             direction: options[:direction] || 1,
-            include_current: options[:include_current] || false
+            include_current: options[:include_current] || false,
+            type: options[:type].try(:to_sym) || :next
           }
         end
       end
 
       def compare_direction
-        direction == 1 ? '$gt' : '$lt'
+        dir = type == :previous ? -direction : direction
+        dir == 1 ? '$gt' : '$lt'
       end
 
       def tiebreak_compare_direction

diff --git a/lib/mongoid/scroll/errors.rb b/lib/mongoid/scroll/errors.rb
@@ -6,3 +6,4 @@
 require 'mongoid/scroll/errors/invalid_base64_cursor_error'
 require 'mongoid/scroll/errors/no_such_field_error'
 require 'mongoid/scroll/errors/unsupported_field_type_error'
+require 'mongoid/scroll/errors/unsupported_type_error'
diff --git a/lib/mongoid/scroll/errors/unsupported_type_error.rb b/lib/mongoid/scroll/errors/unsupported_type_error.rb
@@ -0,0 +1,11 @@
+module Mongoid
+  module Scroll
+    module Errors
+      class UnsupportedTypeError < Mongoid::Scroll::Errors::Base
+        def initialize(opts = {})
+          super(compose_message('unsupported_type', opts))
+        end
+      end
+    end
+  end
+end
diff --git a/lib/mongoid/scroll/version.rb b/lib/mongoid/scroll/version.rb
@@ -1,5 +1,5 @@
 module Mongoid
   module Scroll
-    VERSION = '1.0.2'.freeze
+    VERSION = '2.0.0'.freeze
   end
 end