Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix a token incompatibility for
Prism::Translation::Parser::Lexer
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis. ## Parser gem (Expected) Returns `tLPAREN2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 \ -ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tLPAREN` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` After the update, the parser now returns `tLPAREN2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`. With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`: ```diff - when :tLPAREN - actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2 ```
- Loading branch information