Curry underscore arguments to create anonymous functions [Go/No Go]

[LilithHafner]

Triage previously decided that, in the context of #24990, when it comes to special parsing for underscores to create anonymous functions, "the only viable options are (1) do nothing, or (2) consume one call followed by zero or more operator calls."

I would like to either

• (1) get a triage decision that special handling of _ to create autonomous functions is not for Julia 1.x so that I can give up on this; or
• (2) approve, implement, and merge option 2.

To that end, I'm opening a discussion here explicitly scoped to the question: of options (1) and (2), which is better? Triage's decisions can certainly be revised, but I'd like to keep that discourse out of this discussion thread. For additional options, please feel free to continue the discussion at #24990 or related issues and PRs. This discussion thread is intended to solicit asynchronous public comment (traige is also open to the public) on the specific question above, hopefully in a concise enough format that folks can actually read it.

Edited to include @tpapp's correction

[tpapp]

Strictly speaking, those are not the only two options to get headless anonymous function syntax --- that comment was within the context of that particular proposal.

Note that there are other proposals in this family with the triage label, eg #38713, which has turned into a bunch of various proposals too.

[antoine-levitt]

It's not clear to me what this rule means: is it one and exactly one call, or at most one call? Could someone maybe write down a few examples of how option (2) would be parsed? Eg how would it work with things like map(_, x), map(_+1, x), map(sqrt(_+1),x), f = sqrt(_+1)?

[akdienes]

if I understand correctly:

1. j -> map(j, x) # probably not what was intended

2. map(j -> j+1, x)

3. map(sqrt(j -> j+1), x) # probably not what was intended

[LilithHafner]

An operator call is a type of call so "exactly 1 call + 0 or more operator calls" is the same as "at most 1 call + 0 or more operator calls".

[LilithHafner]

There is actually a difference in the case where no calls at all are available (e.g. [_ for a in 1:10] or if _; 1 else 2 end) in which case I think we should throw an error rather than treat _ as identity. So "exactly 1 call + 0 or more operator calls".

[oschulz]

Personally, I'd vote for option (2) - having f(a, _, b, _) would just be really convenient.

It would be very valuable to have this implemented by a dispatchable type that Base.Fix1 and Base.Fix2 would then become special cases of.

[oschulz]

@LilithHafner would triage option two allow for something like f(a, _...) ?

[LilithHafner]

f(x...) lowers to Core._apply_iterate(Base.iterate, f, x), so I think this would be covered by the "one function call" rule and be equivalent to x -> f(a, x...).

[akdienes]

just to check my understanding, so with broadcasting

f.(a, _...)     =>    x -> f.(a, x...)

would be equivalent?

[LilithHafner]

I believe those would be equivalent.

A broadcast expression counts as a single function call for this because nobody would want f.(_) to lower to materialize(x -> broadcasted(f, x)) and plenty of people would want x -> f.(x)

[oschulz]

That would come in so handy. :-)

[akdienes]

one more (maybe less useful, but more confusing)
x -> _ + x would return like a "curry closure" ? so equivalent to (x -> (j -> j + x)). I guess there shouldn't be anything special about a -> block, so it still only consumes the singular +

[stevengj]

I think (2) would be better than nothing, though it seems like a pain to implement.

[LilithHafner]

It's plausible that being difficult to implement corresponds with being difficult to understand.

[akdienes]

Personally, I (think) I find the rule straightforward to apply in most cases, but I would bet it for sure increases the number of Julia 'WATs'

[aplavin]

An issue with special-casing "operator calls" is that the distinction is quite blurry.
See these two sequences of examples, each starts with something that is clearly an "operator" call, and ends with something that's clearly not.

g(_) ∈ x  # definitely an "operator"
g(_) in x
in(g(_), x)
occursin(g(_), x)
h(g(_), x)  # definitely not an "operator"

or

-f(_)  # definitely an "operator"
√f(_)
√(f(_))
sqrt(f(_))
exp(f(_))
g(f(_))  # definitely not an "operator"

No matter at what step Julia decides that the input isn't an "operator" anymore, two extremely similar/equivalent forms will parse in completely different ways.

And the error messages in case of "unexpected" parsing wouldn't really be helpful, according to the todays situation.

[LilithHafner]

For reference, here is a concrete list (Subject to revision) of what is considered an operator

• Anything Base.isoperator returns true for except |> and =. (! != !== % & ' * + - / // : < <: << <= == => > >: >= >> >>> \ ^ | |> ~ ÷ ∈ ∉ ∋ ∌ ∘ √ ∛ ∩ ∪ ≈ ≉ ≠ ≡ ≢ ≤ ≥ ⊆ ⊇ ⊈ ⊉ ⊊ ⊋ ⊻ ⊼ ⊽)
• X[ ], [ ], (,)
  (source)

This progression is also dubious, but necessary because x[] and Int[] are indistinguishable at lowering time.

x[f(_)]              # operator
Int[f(_), 2, 3]      # operator
[f(_), 2, 3]         # operator
[f(_) for _ in 1:10] # not operator

[uniment]

Triage previously decided

Why isn't scala-like tight-binding an option (i.e. consume 1 call)? Weighing pros/cons:

1 call + operators

Pro:
a.) Can write _^2>3 and 1-_-3.

Con:
a.) Requires parser changes, may break macros, difficult to implement, requires new understanding of parsing.
b.) Can write sin(_)/2+1 but not sin(_/2)+1 nor sin(_/2+1).

Qn: Would _^2+_^3 be x->x^2+x^3 or (x,y)->x^2+y^2, or would it be disallowed? Either way, atan(_^2,_^3) wouldn't work.

1 call (tight-binding)

Pro:
a.) Lowers to typed partial function applicator functor (e.g. Fix1), useful for dispatch and avoiding recompilation.
b.) No parser rewrite, no macro breakage.
c.) Rules and special cases are simpler and easier to understand.
d.) Multiple-arg functions are easy to understand (e.g. map(_==_, [1,2,3], [3,2,1]) returns [false,true,false]).

Con:
a.) Would write (_>3)∘(_^2) and (_-3)∘(1-_), which is uglier than _^2>3 and 1-_-3.
b.) Something like _^2+_^3 is impossible.

The surface area of unintuitive cases for 1 call+ops (e.g. _+1==2 working but isequal(_+1, 2) not), seems much greater than the surface area of unintuitive cases for tight-binding. Also, the concern over 1-_-3 seems overdone, when it could be addressed by documentation.

Here's a macro-based demo for one vision for how tight-binding might work, here.

[LilithHafner]

Triage's previous decision can certainly be revisited and reversed, but I'd like to keep this discussion focused on the question "of options (1) and (2), which is better?" from the OP.

[schlichtanders]

How does (2) parse lower _.a + _.b?

How _ + _.a + _.b + _?

[schlichtanders]

thank you,

I've been using Query.jl and Pipe.jl as current options of _ syntax. They unfortunate clash with these examples, as they rather prefer the single argument interpretation.

Hence, let me ask, was there already a case why every underscore is interpreted as a new argument?

In my practice, functions are rather using one or two arguments, not 3, 4, or more. I think it could help adaptation if this is respected in the lowering with some extra support.

Like

• _.a + __.b lowers to (_1, _2) -> _1.a + _2.b
• _.a + _.b lowers to (_1) -> _1.a + _2.b)

or the use of _1, _2, ... instead of _, __, ...

• _1.a + _2.b lowers to (_1, _2) -> _1.a + _2.b

I would prefer _ and __ because it is easier to parse with the eye and three or more anonymous arguments are unlikely in Julia.

[aplavin]

Agree that the same symbol in the same scope should always mean the same thing!
This underscore behavior is absolutely dominant in the current packages. In addition to Query and Pipe:

• (my) DataPipes.jl, eg @p A |> map(_.x + _.y)
• not just piping, see Accessors.jl with f = @o _[length(_) ÷ 2], f([1,2,3,4]) == 2

For more arguments, DataPipes uses _2 etc: @p map(_1 + _2, [1,2,3], [4,5,6]). Accessors only make sense for a single argument, so they only have _.

[LilithHafner]

They unfortunate clash with these examples, as they rather prefer the single argument interpretation.

Pipes and accessors produce objects that often necessarily accept only one argument, so it makes sense for those packages to interpret repeated underscores as the same argument, while in a general case it is relatively more common to have multiple argument functions.

Existing packages will continue to work as expected because macro expansion happens before lowering. No code will break.

was there already a case why every underscore is interpreted as a new argument?

Empirically, it is more less common to define something like (a, b) -> a + b than a -> a + a (see #24990 (comment)) but looking back I think I was just assuming multiple underscores = multiple arguments was clearly the way to go, I don't think we ever actually thoroughly discussed that.

[schlichtanders]

I shortly took a look at your annotated examples in the comment you linked. I disagree with your conclusion - empirically it seems to suggest that most cases actually reuse variables multiple times.

The only example which uses two variables exactly once is along the line of args...; kwargs..., which nicely could be summarized with _...; __....

[LilithHafner]

Yay for reproducible data! You are correct. I only skimmed the first couple of examples which included (t,x) -> show(t, "text/plain", x), but looking through the remainder, I only see that example and the args..., kwargs... example with two args each used once and I see 5 that reuse a single arg. (And the ... example doesn't really count because it would not be possible to pass on keyword arguments with the current proposal)

[oschulz]

Agree that the same symbol in the same scope should always mean the same thing!
This underscore behavior is absolutely dominant in the current packages.

To play devil's advocate - one could argue that we currently have a different LHS behavior in the language:

_, b, _, d = (1, "b", :c, 'd')

The natural interpretation here is, I would argue, that we assign 1 and :c to two different (and differently typed) _-variables (which are throwaway, of course). One could of course say that there's only one _ and it gets overwritten, changing type, but that seems less natural to me.

[schlichtanders]

That is plainly reassigning on to the same variable.

julia> a, a, a = (1,2,3)
(1, 2, 3)

julia> a
3

This is not in any way confusing people to believe that in a + a, the name a could refer to different variables. It is crystal clear that it is the same variable a referenced twice.

[LilithHafner]

It is true that _ is not an ordinary symbol. [ does not mean the same thing everywhere it occurs and I put _ in a similar category.

[uniment]

That is plainly reassigning on to the same variable.

You interpret it as reassigning the same variable, just as @oschulz interprets it as two different throwaway variables. I interpret it the same way @oschulz does, but in reality nothing is being assigned.

The original issue is worth a read here. The motivating example that got underscore special-cased is this comment.

[oschulz]

@oschulz : _, b, _, d = (1, "b", :c, 'd')
@schlichtanders : That is plainly reassigning on to the same variable.

Hm, what about this case, though (taken from the comment linked by @uniment ):

foo(f) = f(1, 2)

foo() do _, _
    println("ignores arguments")
end

In this case I would say that the two underscores should semantically be interpreted as different dummy variables, because

foo() do a, a
    println("ignores arguments")
end

is legal, but

foo() do a, b
    println("ignores arguments")
end

is not (ERROR: syntax: function argument name not unique: "a").

So I would say that at least in LHS-use the language currently implies that repeated underscores are separate variables, like in Scala. Sure, _ is special, but maybe it could be special with similar semantics in LHS and RHS use?