Preprocessor directives don't flow from the IDE to the compiler

[davidwengier]

When the new Roslyn tokenizer is enabled in the compiler, it also requires a CSharpParseOptions to be provided. Currently the IDE does not serialize this information across the wire, so the language server cannot supply the right one for IDE scenarios, and users get default behaviour.

It would be good if we can focus the compiler dependency down to just what is needed (proprocessor directives is a best guess) and then ensure we are including that information in the RazorConfiguration that the IDE provides.

And if we're changing how UseRoslynTokenizer gets to the compiler, it would be worth reviewing this comment too:
#11092 (comment)

[ryzngard]

❓ is there a reason we don't just add CSharpParseOptions to RazorConfiguration? Is it just serialization problems or something else?

[davidwengier]

Yes, just serialization. We need Json and MessagePack serialization for it, which it doesn't natively support, so we'd have to write out own formatters, and that would mean keeping up with new properties added to it, which sounds like a future bug farm when someone pulls on a random property from CSharpParseOptions, not realising we didn't serialize that one, so they get the default value in the IDE, and expected value in the command line.

[chsienki]

@333fred Please take a look.

[333fred]

@davidwengier @chsienki I'm sorry, but I have absolutely no idea what the actual ask here is. What is this issue actually tracking, and what are we expecting to do about it?

[333fred]

Also likely relevant pieces of existing Roslyn code: https://github.com/dotnet/roslyn/blob/main/src/Workspaces/CSharp/Portable/Serialization/CSharpOptionsSerializationService.cs, https://github.com/dotnet/roslyn/blob/main/src/Workspaces/Core/Portable/Serialization/AbstractOptionsSerializationService.cs#L77-L104

[davidwengier]

What is this issue actually tracking, and what are we expecting to do about it?

Right now the Razor compiler takes in CSharpParseOptions which is something the LSP server cannot provide, so in the IDE everything will just use the default. From your comments in #11091 (comment) it seemed like only PreprocessorSymbolNames were actually needed, however. So if the compiler can be changed to only take that value, then we can plumb it around everywhere, and IDE will reflect the users real project intentions.

[333fred]

So if the compiler can be changed to only take that value, then we can plumb it around everywhere, and IDE will reflect the users real project intentions.

This isn't something we can do; we should use the real CSharpParseOptions, either from cohosting where no serializer is required, or from exposing the serializer/deserializer that the Roslyn IDE already uses via EAP. We are passing the entire parse options object to the Roslyn tokenizer, and I do not think the Razor compiler can or should make guarantees about which pieces of that object it needs; it is likely to get it wrong.