Skip to content

Commit

Permalink
Update options.md
Browse files Browse the repository at this point in the history
Add comment about using threads on datasets where the amount of objects is less the amount of threads.
  • Loading branch information
kindly authored Jun 12, 2023
1 parent d2186c4 commit 238cbca
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion docs/options.md
Original file line number Diff line number Diff line change
Expand Up @@ -754,7 +754,9 @@ flatterer.flatten('inputfile.json', 'ouput_dir', preview=10)

The number of threads used to process the data. Default to 1. If set to 0 will use amount of CPUs.

Works best with new line delimited JSON `--ndjson` as JSON parsing can then be done by each thread. This can about a x3 times improvement with 6 threads if you have that many CPU cores. Without `--ndjson` makes only about x1.24 improvement on 2 threads and not worth going over 2 as it will not lead to performance improvement.
Works best with new line delimited JSON `--ndjson` as JSON parsing can then be done by each thread. This can about a x3 times improvement with 6 threads if you have that many CPU cores. Without `--ndjson` makes only about x1.24 improvement on 2 threads and not worth going over 2 as it will not lead to performance improvement. For very small datasets (less than 100 object) using threads will most likely be slower.

**Warning:** Will fail for small inputs where the amount of objects is less that the amount of threads specified.

**Warning:** May have issues with inline-one-to-one as each thread will determine what should be inlined.

Expand Down

0 comments on commit 238cbca

Please sign in to comment.