Feature request: jldoctest examples without testing against expected results

[jannefiluren]

Hi!

I want to add examples to my functions, but in many cases it is hard or not necessary to test the expected output. However, it may still be useful to test whether the functions/examples runs through or crashes. Is it possible to have a jldoctest with such a behavior? Perhaps by omitting the line(s) with expected outcomes as shown below:

"""
    func_with_untestable_output(invar)

```jldoctest
julia> func_with_untestable_output(invar)
# Nothing to test against
```
"""
function func_with_untestable_output(invar) ...

The behavior I am looking for is the equivalent to adding this piece of "code" to the markdown documents:

```@example
a = 1
b = 2
a + b
nothing # hide
```

However, I think it is more convenient to place the examples and testing whether they work or not directly in the files containing the Julia code, rather than in the markdown documents.

[mortenpi]

Don't @example blocks work in docstrings?

[jannefiluren]

They work, but the code is not tested. Here is an example:

module TestDocs

export test_it


"""
    function test_it()

Here is an example that documenter does not run:
	
```@example
a = 1
b = 2
a + c
nothing # hide
```
		
"""

function test_it()

    println("Hallo")

end

end

When I generate the documentation with the make.jl-file, the code in the @example block does not get executed. You find the whole package at this link:

https://github.com/jmgnve/TestDocs.jl

Have I done something wrong?

[mortenpi]

Ah, no, you're right, apparently they don't work. I assumed that they might. I don't see any other objection to enabling them other than that @example etc. blocks are not supported by core Julia in any way and so they don't play nice with other systems accessing docstrings, e.g. the ? command in the REPL.

[mortenpi]

But back to doctests then -- in principle reducing the test down to "was there an exception or not" should be quite possible.

However, with doctests we shouldn't inject the output of the run into the document (because of the objections above). Are doctests/example blocks without output actually useful? Is there are use case you could show?

Also, we need syntax -- either we'd give it a new label (e.g. jldoctest-error-only), or we could introduce options for blocks (e.g. jldoctest compare=false).

[jannefiluren]

Are doctests/example blocks without output actually useful? Is there are use case you could show?

For me, I think they would be quite useful. For example, I have models that produce a rather large set of outputs that are difficult to show and test in a compact way. But it would be nice to show examples on how to run the code, and be sure that they are always up-to-date.

However, I am quite new to Julia and if it is difficult/tedious to implement such a feature, it would of course be helpful if more people give their opinion. For me it is more a good-to-have feature than something that is really needed.

[fredrikekre]

You can skip comparison pretty trivial now by adding filter = r".*". However, since Documenter catches exception this can not be used for the purpose of:

However, it may still be useful to test whether the functions/examples runs through or crashes.

[davidanthoff]

I'd love to see an option where code is run and one checks whether it crashes, but no output is compared. We have a lot of use cases where maintaining the output check is not practical (or important), but we do want to make sure things run.

[timholy]

Sometimes I have some setup that I'd like to associate with a particular named doctest sequence, interleaving explanation and code. But some steps in the sequence have no need for a "correctness" test, they just need "did it not throw an error?" validation. Here is an example, where there is no reason to test the result of loading the image, but we'd like to use it for later demo tests:

Let's load an image for testing:

```jldoctest demo
using RegisterDeformation, TestImages
img = testimage("lighthouse")
```

Now we create a deformation over the span of the image:

```jldoctest demo
# Create a deformation
gridsize = (5, 5)                   # a coarse grid
u = 20*randn(2, gridsize...)        # each displacement is 2-dimensional
# The nodes specify the location of each value in the `u` array
# relative to the image that we want to warp. This choice spans
# the entire image.
nodes = map(axes(img), gridsize) do ax, g
    range(first(ax), stop=last(ax), length=g)
end
ϕ = GridDeformation(u, nodes)

# output

5×5 GridDeformation{Float64} over a domain 1.0..512.0×1.0..768.0

[...more docs follow...]

Unfortunately this gives an error:

┌ Warning: invalid doctest block in src/index.md:32-35
│ Requires `julia> ` or `# output`
│ 
│ ```jldoctest demo
│ using RegisterDeformation, TestImages
│ img = testimage("lighthouse")
│ ```
└ @ Documenter.DocTests ~/.julia/packages/Documenter/O67pl/src/DocTests.jl:176
┌ Error: doctest failure in src/index.md:39-54
│ 
│ ```jldoctest demo
│ # Create a deformation
│ gridsize = (5, 5)                   # a coarse grid
│ u = 20*randn(2, gridsize...)        # each displacement is 2-dimensional
│ # The nodes specify the location of each value in the `u` array
│ # relative to the image that we want to warp. This choice spans
│ # the entire image.
│ nodes = map(axes(img), gridsize) do ax, g
│     range(first(ax), stop=last(ax), length=g)
│ end
│ ϕ = GridDeformation(u, nodes)
│ 
│ # output
│ 
│ 5×5 GridDeformation{Float64} over a domain 1.0..512.0×1.0..768.0
│ ```
│ 
│ Subexpression:
│ 
│ # Create a deformation
│ gridsize = (5, 5)                   # a coarse grid
│ u = 20*randn(2, gridsize...)        # each displacement is 2-dimensional
│ # The nodes specify the location of each value in the `u` array
│ # relative to the image that we want to warp. This choice spans
│ # the entire image.
│ nodes = map(axes(img), gridsize) do ax, g
│     range(first(ax), stop=last(ax), length=g)
│ end
│ ϕ = GridDeformation(u, nodes)
│ 
│ Evaluated output:
│ 
│ ERROR: UndefVarError: img not defined
│ Stacktrace:
│  [1] top-level scope at /home/tim/.julia/dev/RegisterDeformation/docs/make.jl:4
...

I could mimic that with adequate setup code, but it just repeats the stuff that's in the demo.
Or, I can change that first block to

```jldoctest demo; output=false
using RegisterDeformation, TestImages
img = testimage("lighthouse")
summary(img)

# output

"512×768 Array{RGB{N0f8},2} with eltype RGB{Normed{UInt8,8}}"
```

but that feels pretty silly and might confuse users ("why do I have to call summary?")

[rafaqz]

I have the same request. In GeoData.jl the examples are often downloading some data source, applying an operation, and plotting. The output is not worth testing, but I would like to make sure that the example code actually runs. Mostly now there are just empty # output lines at the end because the output is nothing, from plotting. It would be good to just remove those lines.

[TorkelE]

Just also mentioning that this would be useful in Catalyst.jl as well. Every once in a while we get an issue where someone has tried to run something in the docs, but it no longer works and produces an error. It would not be feasible to compare the actual output, because either:

• The output is rather stochastic and writing something which would accept it would be very cumbersome (and would be unlikely to catch anything which a crash wouldn't).
• The output is some extensive data structure. We could just auto-update the expected output every once in a while. However, it would not be feasible to actually check it in detail, so in practice, this would not provide any additional benefit over a simple check whenever the code block caused an error or not.

[mortenpi]

I think the actionable decision here is to implement an option (e.g. jldoctest; mode=:errorsonly) that disables output checking, but does fail if any errors were thrown (unlike the current doctests). That seems like a reasonable way to extend the doctesting functionality. Note: doctests=:fix should still update the output of such a doctest, in case it has become outdated.

[TorkelE]

such a test would be very useful

[matthewgcooper]

Just wondering if this is still a planned feature. It would be useful for my package.

[fingolfin]

Would also like this.

As a fun complication, in my case there is a function called in doctest which internally executes a @testset. To elaborate, think of a function like this:

function test_conformance_with_specification(obj)
  @testset "Check conformance" begin
    @test ...
  end
end

And then I have a sequence of jldoctests that explain how to implement a type matching the specification... And the very last one looks like this:

Now that we have completed the implementation, we can use `test_conformance_with_specification` to test its conformance. It is
recommend that you add an invocation to your package tests.

```jldoctest running_example; filter = r".*"
test_conformance_with_specification(constant_polynomial_ring(ZZ))

# output

```

Note that I already use filter = r".*" but for some reason it doesn't work.

Anyway, note that since Documenter wraps the doctests also in a @testset -- at least in some scenarios -- this block may or may not produce output in Documenter. So I really need to suppress the output, and that's fine. As in this case really all that matters is that no test failed and no unhandled exception was thrown.

[mortenpi]

Just wondering if this is still a planned feature. It would be useful for my package.

Just needs someone to take on the implementation. Would be happy to accept a PR implementing some version of this: #452 (comment)

Anyway, note that since Documenter wraps the doctests also in a @testset -- at least in some scenarios

Only if you run the doctest function. Arguably, even though it's an edge case, it's a bit of a bug that the Documenter's testset interferes with any testsets inside. If that's possible with the Test stdlib APIs, I think we should reset the global test-related state to mimick a vanilla REPL/top-level state.

It's orthogonal to this issue though. It would probably be good to have a separate issue to document and track this issue.

[fingolfin]

You are right, will do.

UPDATE done via #2605