Skip to content

Commit

Permalink
New Cypher runtimes chapter (#688)
Browse files Browse the repository at this point in the history
To be released alongside Parallel runtime.


Includes changes from the following PRs:
- #667
- #676
- #712
- #732
  • Loading branch information
JPryce-Aklundh authored Oct 18, 2023
1 parent ef8e1ac commit 8c5585b
Show file tree
Hide file tree
Showing 32 changed files with 1,832 additions and 1,075 deletions.
25 changes: 12 additions & 13 deletions modules/ROOT/content-nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -84,19 +84,17 @@
** xref:constraints/syntax.adoc[]
** xref:constraints/examples.adoc[]
* xref:query-tuning/index.adoc[]
** xref:query-tuning/query-options.adoc[]
** xref:query-tuning/query-profile.adoc[]
** xref:query-tuning/indexes.adoc[]
** xref:query-tuning/basic-example.adoc[]
** xref:query-tuning/advanced-example.adoc[]
** xref:query-tuning/using.adoc[]
* xref:execution-plans/index.adoc[]
** xref:execution-plans/db-hits.adoc[]
** xref:execution-plans/operator-summary.adoc[]
** xref:execution-plans/operators.adoc[]
** xref:execution-plans/shortestpath-planning.adoc[]
* xref:planning-and-tuning/index.adoc[]
** xref:planning-and-tuning/execution-plans.adoc[]
** xref:planning-and-tuning/operators/index.adoc[]
*** xref:planning-and-tuning/operators/operators-detail.adoc[]
** xref:planning-and-tuning/runtimes/index.adoc[]
*** xref:planning-and-tuning/runtimes/concepts.adoc[Concepts]
*** xref:planning-and-tuning/runtimes/reference.adoc[]
** xref:planning-and-tuning/query-tuning/index.adoc[]
*** xref:planning-and-tuning/query-tuning/indexes.adoc[]
*** xref:planning-and-tuning/query-tuning/using.adoc[]
*** xref:planning-and-tuning/query-tuning/query-options.adoc[]
* xref:query-caches/index.adoc[]
** xref:query-caches/unified-query-caches.adoc[]
Expand All @@ -116,3 +114,4 @@
* Appendix
** xref:styleguide.adoc[]
** xref:appendix/tutorials/index.adoc[]
41 changes: 41 additions & 0 deletions modules/ROOT/images/runtimes_cypher_lifecycle.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 9 additions & 0 deletions modules/ROOT/images/runtimes_execution_graph1.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 9 additions & 0 deletions modules/ROOT/images/runtimes_execution_graph2.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
:description: Example of some more subtle optimizations based on native index capabilities.


[[advanced-query-tuning-example]]
= Advanced query tuning example

This page describes advanced query optimizations based on native index capabilities.

One of the most important and useful ways of optimizing Cypher queries involves creating appropriate indexes.
This is described in more detail in xref::indexes-for-search-performance.adoc[], and demonstrated in xref::query-tuning/basic-example.adoc[].
This is described in more detail in xref::indexes-for-search-performance.adoc[], and demonstrated in xref::appendix/tutorials/basic-query-tuning.adoc[].
In summary, an index will be based on the combination of a `Label` and a `property`.
Any Cypher query that searches for nodes with a specific label and some predicate on the property (equality, range or existence) will be planned to use
the index if the cost planner deems that to be the most efficient solution.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -607,7 +607,7 @@ RETURN p
This query will find the *'Tom Hanks'* node but as the number of nodes in the database increase it will become slower and slower.
We can profile the query to find out why that is.

You can learn more about the options for profiling queries in xref::query-tuning/query-options.adoc[] but in this case you are going to prefix our query with `PROFILE`:
You can learn more about the options for profiling queries in xref::planning-and-tuning/query-tuning/query-options.adoc[] but in this case you are going to prefix our query with `PROFILE`:

[source, cypher]
----
Expand Down Expand Up @@ -646,9 +646,9 @@ ready to start consuming query after 17 ms, results consumed after another 10 ms
The first thing to keep in mind when reading execution plans is that you need to read from the bottom up.

In that vein, starting from the last row, the first thing you notice is that the value in the `Rows` column seems high given there is only one node with the name property *'Tom Hanks'* in the database.
If you look across to the `Operator` column, you will see that xref::execution-plans/operators.adoc#query-plan-all-nodes-scan[AllNodesScan] has been used which means that the query planner scanned through all the nodes in the database.
If you look across to the `Operator` column, you will see that xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-all-nodes-scan[AllNodesScan] has been used which means that the query planner scanned through all the nodes in the database.

The xref::execution-plans/operators.adoc#query-plan-filter[Filter] operator which will check the `name` property on each of the nodes passed through by `AllNodesScan`.
The xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-filter[Filter] operator which will check the `name` property on each of the nodes passed through by `AllNodesScan`.

This seems like an inefficient way of finding *'Tom Hanks'* given that you are looking at many nodes that are not even people and therefore are not what you are looking for.

Expand Down Expand Up @@ -700,7 +700,7 @@ Total database accesses: 379, total allocated memory: 184
----

This time the `Rows` value on the last row has reduced so you are not scanning some nodes that you were before which is a good start.
The xref::execution-plans/operators.adoc#query-plan-node-by-label-scan[NodeByLabelScan] operator indicates that you achieved this by first doing a linear scan of all the `Person` nodes in the database.
The xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-node-by-label-scan[NodeByLabelScan] operator indicates that you achieved this by first doing a linear scan of all the `Person` nodes in the database.

Once you have done that, you can again scan through all those nodes using the `Filter` operator, comparing the name property of each one.

Expand Down Expand Up @@ -764,5 +764,5 @@ Total database accesses: 5, total allocated memory: 184
1 row
----

Our execution plan is down to a single row and uses the xref::execution-plans/operators.adoc#query-plan-node-index-seek[Node Index Seek] operator which does an index seek (see xref::indexes-for-search-performance.adoc[]) to find the appropriate node.
Our execution plan is down to a single row and uses the xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-node-index-seek[Node Index Seek] operator which does an index seek (see xref::indexes-for-search-performance.adoc[]) to find the appropriate node.

6 changes: 6 additions & 0 deletions modules/ROOT/pages/appendix/tutorials/index.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
:description: List of available tutorials in the Cypher Manual.
= Tutorials and extended examples

* xref:appendix/tutorials/basic-query-tuning.adoc[]
* xref:appendix/tutorials/advanced-query-tuning.adoc[]
* xref:appendix/tutorials/shortestpath-planning.adoc[] - information about how to plan queries using the xref:patterns/concepts.adoc#shortest-path[`shortestPath()` function].
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Therefore, in these cases, it is recommended to set `cypher.forbid_exhaustive_sh
== Shortest path -- fast algorithm


.Query evaluated with the fast algorithm
.Query evaluated with the fast algorith
======
////
Expand Down
4 changes: 2 additions & 2 deletions modules/ROOT/pages/clauses/clause_composition.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -951,8 +951,8 @@ materialize the table of intermediate results before executing the next clause.
This approach would consume a lot of memory for materializing the tables of intermediate results and would generally not perform well.

Instead, Cypher will in general try to interleave the execution of clauses.
This is called xref::execution-plans/index.adoc#eagerness-laziness[lazy evaluation].
This is called xref::planning-and-tuning/execution-plans.adoc#laze-eager-evaluation[lazy evaluation].
It only materializes intermediate results when needed.
In many read-write queries it is unproblematic to execute clauses interleaved, but when it is not,
Cypher must ensure that the table of intermediate results gets materialized at the right time(s).
This is done by inserting an xref::execution-plans/operators.adoc#query-plan-eager[`Eager`] operator into the execution plan.
This is done by inserting an xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-eager[`Eager`] operator into the execution plan.
10 changes: 5 additions & 5 deletions modules/ROOT/pages/clauses/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -234,22 +234,22 @@ m| xref:clauses/transaction-clauses.adoc#query-terminate-transactions[TERMINATE
== Reading hints

These comprise clauses used to specify planner hints when tuning a query.
More details regarding the usage of these -- and query tuning in general -- can be found in xref::query-tuning/using.adoc[Planner hints and the USING keyword].
More details regarding the usage of these -- and query tuning in general -- can be found in xref::planning-and-tuning/query-tuning/using.adoc[Planner hints and the USING keyword].

[options="header"]
|===
| Hint | Description

m| xref::query-tuning/using.adoc#query-using-index-hint[USING INDEX]
m| xref::planning-and-tuning/query-tuning/using.adoc#query-using-index-hint[USING INDEX]
| Index hints are used to specify which index, if any, the planner should use as a starting point.

m| xref::query-tuning/using.adoc#query-using-index-hint[USING INDEX SEEK]
m| xref::planning-and-tuning/query-tuning/using.adoc#query-using-index-hint[USING INDEX SEEK]
| Index seek hint instructs the planner to use an index seek for this clause.

m| xref::query-tuning/using.adoc#query-using-scan-hint[USING SCAN]
m| xref::planning-and-tuning/query-tuning/using.adoc#query-using-scan-hint[USING SCAN]
| Scan hints are used to force the planner to do a label scan (followed by a filtering operation) instead of using an index.

m| xref::query-tuning/using.adoc#query-using-join-hint[USING JOIN]
m| xref::planning-and-tuning/query-tuning/using.adoc#query-using-join-hint[USING JOIN]
| Join hints are used to enforce a join operation at specified points.

|===
Expand Down
2 changes: 1 addition & 1 deletion modules/ROOT/pages/clauses/match.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Cypher is declarative, and so usually the query itself does not specify the algo
Neo4j will automatically work out the best approach to finding start nodes and matching patterns.
Predicates in `WHERE` parts can be evaluated before pattern matching, during pattern matching, or after finding matches.
However, there are cases where you can influence the decisions taken by the query compiler.
Read more about indexes in xref::indexes-for-search-performance.adoc[], and more about specifying hints to force Neo4j to solve a query in a specific way in xref::query-tuning/using.adoc[Planner hints and the USING keyword].
Read more about indexes in xref::indexes-for-search-performance.adoc[], and more about specifying hints to force Neo4j to solve a query in a specific way in xref::planning-and-tuning/query-tuning/using.adoc[Planner hints and the USING keyword].


[[match-example-graph]]
Expand Down
2 changes: 1 addition & 1 deletion modules/ROOT/pages/clauses/order-by.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This last rule is to make sure that `ORDER BY` does not change the results, only

The performance of Cypher queries using `ORDER BY` on node properties can be influenced by the existence and use of an index for finding the nodes.
If the index can provide the nodes in the order requested in the query, Cypher can avoid the use of an expensive `Sort` operation.
Read more about this capability in xref::query-tuning/advanced-example.adoc#advanced-query-tuning-example-index-backed-order-by[Index-backed ORDER BY].
Read more about this capability in xref::appendix/tutorials/advanced-query-tuning.adoc#advanced-query-tuning-example-index-backed-order-by[Index-backed ORDER BY].

The following graph is used for the examples below:

Expand Down
4 changes: 2 additions & 2 deletions modules/ROOT/pages/clauses/transaction-clauses.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -82,11 +82,11 @@ m| MAP
m| planner
a|
The name of the Cypher planner used to plan the query currently executing in this transaction, or an empty string if no query is currently executing.
For details, see xref::query-tuning/index.adoc#cypher-planner[Cypher planner].
For details, see xref::planning-and-tuning/query-tuning/index.adoc#cypher-planner[Cypher planner].
m| STRING

m| runtime
a| The name of the Cypher runtime used by the query currently executing in this transaction, or an empty string if no query is currently executing. For details, see xref::query-tuning/index.adoc#cypher-runtime[Cypher runtime].
a| The name of the Cypher runtime used by the query currently executing in this transaction, or an empty string if no query is currently executing. For details, see xref::planning-and-tuning/runtimes/index.adoc[Cypher runtime].
m| STRING

m| indexes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,18 @@ USE graph.byElementId(elementId :: STRING)
----

| New graph function, xref:functions/graph.adoc#functions-graph-by-elementid[graph.byElementId()], that resolves the constituent graph to which a given element id belongs.

a|
label:functionality[]
label:new[]

----
CYPHER runtime = parallel
----

|
Introduction of the xref:planning-and-tuning/runtimes/concepts.adoc#runtimes-parallel-runtime[parallel runtime].
This runtime is designed for analytical, graph-global read queries run on machines with several available CPUs.
|===

[[cypher-deprecations-additions-removals-5.12]]
Expand Down Expand Up @@ -411,7 +423,7 @@ New operator: `Repeat(Trail)`

a|
The `Repeat(Trail)` operator is used to solve xref::patterns/concepts.adoc#quantified-path-patterns[quantified path patterns].
More information can be found xref::execution-plans/operators.adoc#query-plan-repeat[here].
More information can be found xref:planning-and-tuning/operators/operators-detail.adoc#query-plan-repeat[here].

a|
label:functionality[]
Expand Down Expand Up @@ -534,7 +546,7 @@ New operator: `AssertSameRelationship`

a|
The `AssertSameRelationship` operator is used to ensure that no relationship property uniqueness constraints are violated in the slotted and interpreted runtime.
More information can be found xref::execution-plans/operators.adoc#query-plan-assert-same-relationship[here].
More information can be found xref:planning-and-tuning/operators/operators-detail.adoc#query-plan-assert-same-relationship[here].

|===

Expand Down Expand Up @@ -783,7 +795,7 @@ New operator: `IntersectionNodeByLabelsScan`

a|
The `IntersectionNodeByLabelsScan` operator fetches all nodes that have all of the provided labels from the node label index.
More information can be found xref::execution-plans/operators.adoc#query-plan-intersection-node-by-labels-scan[here].
More information can be found xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-intersection-node-by-labels-scan[here].

|===

Expand Down Expand Up @@ -868,7 +880,7 @@ New operator: `NodeByElementIdSeek`

a|
The `NodeByElementIdSeek` operator reads one or more nodes by ID from the node store, specified via the function xref::functions/scalar.adoc#functions-elementid[elementId()].
More information can be found xref::execution-plans/operators.adoc#query-plan-node-by-elementid-seek[here].
More information can be found xref::planning-and-tuning/operators/operators-detail.adoc#query-plan-node-by-elementid-seek[here].

|===

Expand Down
64 changes: 0 additions & 64 deletions modules/ROOT/pages/execution-plans/db-hits.adoc

This file was deleted.

Loading

0 comments on commit 8c5585b

Please sign in to comment.