summaryrefslogtreecommitdiff
path: root/lib/prism/translation/parser
AgeCommit message (Collapse)Author
2024-09-09[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument. ## Parser gem (Expected) Returns `tDSTAR` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tPOW` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` After the update, the parser now returns `tDSTAR` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]], [:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]], [:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]] ``` With this change, the following code could be removed from test/prism/ruby/parser_test.rb: ```diff - when :tPOW - actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR ``` `tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["a", #<Parser::Source::Range example.rb 0...1>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 2...4>]], [:tIDENTIFIER, ["b", #<Parser::Source::Range example.rb 5...6>]]] ``` https://github.com/ruby/prism/commit/66bde35a44
2024-09-07[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis. ## Parser gem (Expected) Returns `tLPAREN2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 \ -ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tLPAREN` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` After the update, the parser now returns `tLPAREN2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23] [[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]], [:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]] ``` The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`. With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`: ```diff - when :tLPAREN - actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2 ``` https://github.com/ruby/prism/commit/04d6f3478d
2024-08-29[ruby/prism] Remove deprecated lib/prism/translation/parser/rubocop.rb fileKoichi ITO
Follow up https://github.com/ruby/prism/pull/2558. This PR removes deprecated lib/prism/translation/parser/rubocop.rb file. The file was a workaround to allow setting `TargetRubyVersion: 80_82_73_83_77.xx` until Prism (`Prism::Translation::Parser`) is integrated into RuboCop. RuboCop already supports Prism (`Prism::Translation::Parser`) as of https://github.com/rubocop/rubocop/pull/12724. It has been several months since the file was deprecated in ruby/prism#2558. And, yesterday, Prism 1.0.0 was released, but perhaps the file should have been removed before then. Although this might be seen as incompatible with semver 1.0.0, I think there is no longer a reason to keep the file. https://github.com/ruby/prism/commit/646a10270e
2024-08-28[PRISM] Field renamingKevin Newton
Rename some fields that do not quite make sense. * CaseMatchNode#consequent -> CaseMatchNode#else_clause * CaseNode#consequent -> CaseNode#else_clause * IfNode#consequent -> IfNode#subsequent * RescueNode#consequent -> RescueNode#subsequent * UnlessNode#consequent -> UnlessNode#else_clause Notes: Merged: https://github.com/ruby/ruby/pull/11480
2024-06-18[ruby/prism] (parser) Fix up tokens for empty symbolKevin Newton
https://github.com/ruby/prism/commit/5985ab7687
2024-06-07[ruby/prism] Handle chomped bytesize with lines without newlinesKevin Newton
https://github.com/ruby/prism/commit/1528d3c019
2024-06-07[ruby/prism] Ensure inner heredoc nodes have the correct locationKevin Newton
https://github.com/ruby/prism/commit/100340bc6b
2024-06-07[ruby/prism] Use correct newlines for heredoc inner linesKevin Newton
https://github.com/ruby/prism/commit/4a9a7a62af Co-authored-by: Jason Kim <[email protected]> Co-authored-by: Adam Hess <[email protected]>
2024-06-04[ruby/prism] (parser) split up regexp content by linesKevin Newton
https://github.com/ruby/prism/commit/85b4a5f804
2024-06-04[ruby/prism] (parser) handle quoted symbols in hash patternsKevin Newton
https://github.com/ruby/prism/commit/461aa5e658
2024-06-04[ruby/prism] (parser) fix up srange_find to anchor at the start of the sliceKevin Newton
https://github.com/ruby/prism/commit/aecce571d8
2024-06-04[ruby/prism] (parser) fix up nested multi writeKevin Newton
https://github.com/ruby/prism/commit/12e079c97e
2024-06-04[ruby/prism] Revert "Revert "Properly destructure procarg0 in parser ↵Kevin Newton
translation"" This reverts commit https://github.com/ruby/prism/commit/d8ae19d0334a. https://github.com/ruby/prism/commit/df1eda2811
2024-06-03[ruby/prism] Revert "Properly destructure procarg0 in parser translation"Kevin Newton
This reverts commit https://github.com/ruby/prism/commit/823e931ff230. https://github.com/ruby/prism/commit/d8ae19d033
2024-06-03[ruby/prism] Properly destructure procarg0 in parser translationKevin Newton
https://github.com/ruby/prism/commit/823e931ff2
2024-05-31[ruby/prism] Fix up heredoc location translation for parserKevin Newton
https://github.com/ruby/prism/commit/a4e164e22b
2024-05-31[ruby/prism] Match % strings in parserKevin Newton
https://github.com/ruby/prism/commit/840185110f
2024-05-31[ruby/prism] Use correct opening and closing parenthesis for array pattern ↵Kevin Newton
in parser https://github.com/ruby/prism/commit/beed43922c
2024-05-31[ruby/prism] Match match_hash_var when quotes are usedKevin Newton
https://github.com/ruby/prism/commit/f2a327449a
2024-05-31[ruby/prism] Match parser for match_rest in patternKevin Newton
https://github.com/ruby/prism/commit/785de2c39d
2024-05-22[ruby/prism] Fix support for 'it' implicit local variableKevin Newton
https://github.com/ruby/prism/commit/53bbcfe513
2024-05-21[ruby/prism] Reconfigure rationalsKevin Newton
This eliminates the subnode on RationalNode and replaces it with two integer fields, which represent the ratio for the rational. It also reduces those two integers if they both fit into 32 bits. Importantly, this PR does not implement bignum reduction. That's something I'd like to consider for the future, but it's simple enough for now to leave them unreduced, which makes it more useful than it used to be. https://github.com/ruby/prism/commit/86e06c7068
2024-05-10[ruby/prism] operator renameKevin Newton
https://github.com/ruby/prism/commit/b5e47f5c42
2024-05-03[ruby/prism] Change ConstantPathNode#child to ConstantPathNode#{name,name_loc}Kevin Newton
This has been requested for a long time, and I'm finally doing it now. Unfortunately this is a breaking change for all of the APIs. I've added in a Ruby method for `#child` that is deprecated so that existing usage doesn't break, but for everyone else this is going to be a bit of a pain. https://github.com/ruby/prism/commit/9cbe74464e
2024-04-23[ruby/prism] srange_find should only look on current lineKevin Newton
https://github.com/ruby/prism/commit/3604aa15e7
2024-04-19[ruby/prism] Fix it parameters for parser translationKevin Newton
https://github.com/ruby/prism/commit/2f3feb8d51
2024-04-18[ruby/prism] "Fix" transpose issue in parser compilerKevin Newton
https://github.com/ruby/prism/commit/593d637178
2024-04-12[ruby/prism] Fix parser translation's heredoc whitespace calculationStan Lo
Given this example: ```rb <<~HEREDOC #{x} HEREDOC ``` Both the parser gem and Prism's translation layer would generate the following AST: ``` s(:dstr, s(:begin, s(:int, 1)), s(:str, " a\n")) ``` However, the parser gem inserts a empty string node into this node's location, like: ``` <Parser::Source::Map::Heredoc:0x0000000104ce73b8 @expression=#<Parser::Source::Range (string) 0...10>, @heredoc_body=#<Parser::Source::Range (string) 11...20>, @heredoc_end=#<Parser::Source::Range (string) 20...27>, @node=s(:dstr, s(:str, ""), s(:begin, s(:int, 1)), s(:str, " a\n"))> ``` This is required to calculate the correct whitespace for the heredoc body. We need to adjust the translation layer to account for this. With this fix, we also won't need to ignore the tilde heredoc fixture anymore. https://github.com/ruby/prism/commit/e7372e3ba5
2024-04-04[ruby/prism] Fix up some comments in the parser compilerKevin Newton
https://github.com/ruby/prism/commit/e2147cddd8
2024-03-18[ruby/prism] Use `require_relative` in the Prism codebaseKoichi ITO
If there are many searches in the `$LOAD_PATH` in the user environment, require will perform unnecessary searches that are not needed. In contrast, `require_relative` is efficient because it uses a relative path. https://github.com/ruby/prism/commit/438ccc67bd
2024-03-16[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using backquoted heredoc indetiner: ```ruby <<-` FOO` a b FOO ``` ## Parser gem (Expected) Returns `tXSTRING_BEG` as the first token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, [" FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING_BEG` token and value of `tSTRING_END` token. ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tSTRING_BEG, ["<<\"", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, ["` FOO`", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bunlde exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, [" FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` https://github.com/ruby/prism/commit/308f8d85a1
2024-03-15[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using escaped backslash in string literal: ```ruby "\\ foo \\ bar" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` The reproduction test is based on the following strings.txt and exists: https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79 But, the restoration has not yet been performed due to remaining other issues in strings.txt. https://github.com/ruby/prism/commit/2c44e7e307
2024-03-15[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for the heredocs_leading_whitespace.txt test. https://github.com/ruby/prism/commit/7d45fb1eed
2024-03-15[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for dstring literal: ```ruby "foo #{bar}" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:dstr, s(:str, "foo\n"), s(:str, " ")), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` https://github.com/ruby/prism/commit/c1652a9ee7
2024-03-15[ruby/prism] Shareable constant nodesKevin Newton
https://github.com/ruby/prism/commit/473cfed6d0
2024-03-13[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for empty xstring literal. ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different: ```console $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` https://github.com/ruby/prism/commit/4ac89dcbb5
2024-03-13[ruby/prism] Fix an AST incompatibility for `Prism::Translation::Parser`Koichi ITO
Fixes ruby/prism#2480. This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser` for xstring literal with line breaks. The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576: ```ruby "foo bar" ``` https://github.com/ruby/prism/commit/cf85e72c55
2024-03-12[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`. The former is used in xstrings like `` `foo` ``, while the latter is utilized as a back reference in contexts like `` A::` ``. This PR will make corrections to differentiate the use of `BACKTICK`. This mistake was discovered through the investigation of xstring.txt file. The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail, hence it will be separated into xstring_with_backslash.txt file. This separation will facilitate addressing the correction at a different time. https://github.com/ruby/prism/commit/49ad8df40a
2024-03-12[ruby/prism] Fix some whitequark/parser lexer compatibilitiesKevin Newton
https://github.com/ruby/prism/commit/34e521d071
2024-03-12[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`: ## Parser gem (Expected) Returns `tBACK_REF2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tXSTRING_BEG` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tXSTRING_BEG, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` After the update, the parser now returns `tBACK_REF2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` This correction enables the restoration of `constants.txt` as a test case. https://github.com/ruby/prism/commit/7f63b28f98
2024-03-12[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for beginless range: ## Parser gem (Expected) Returns `tBDOT2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tDOT2` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]] ``` After the update, the parser now returns `tBDOT2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]] ``` This correction enables the restoration of `endless_range_in_conditional.txt` as a test case. https://github.com/ruby/prism/commit/f624b99ab0
2024-03-12[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` Fixes https://github.com/ruby/prism/pull/2515. This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for string literal with line breaks. https://github.com/ruby/prism/commit/c58466e5bf
2024-03-08[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser`Koichi ITO
Fixes https://github.com/ruby/prism/pull/2512. This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `HEREDOC_END` with a newline. https://github.com/ruby/prism/commit/b67d1e0c6f
2024-03-08Fix an error for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes the following error for `Prism::Translation::Parser::Lexer` on the main branch: ```console $ cat example.rb 'a' # aあ " #{x} " $ bundle exec rubocop Parser::Source::Range: end_pos must not be less than begin_pos /Users/koic/.rbenv/versions/3.0.4/lib/ruby/gems/3.0.0/gems/parser-3.3.0.5/lib/parser/source/range.rb:39:in `initialize' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `new' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `block in to_a' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `map' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `to_a' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:263:in `build_tokens' /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:92:in `tokenize' ``` This change was made in https://github.com/ruby/prism/pull/2557, and it seems there was an inconsistency in Range due to forgetting to apply `offset_cache` to `start_offset`.
2024-03-07[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` Fixes https://github.com/ruby/prism/pull/2506. This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for symbols quoted with line breaks. https://github.com/ruby/prism/commit/06ab4df8cd
2024-03-06[ruby/prism] Use the diagnostic types in the parser translation layerKevin Newton
https://github.com/ruby/prism/commit/1a8a0063dc
2024-03-06[ruby/prism] Small changes to make type-checking passUfuk Kayserilioglu
https://github.com/ruby/prism/commit/5b2970e75b
2024-03-06[ruby/prism] Fix some type-checking errors by using different method callsUfuk Kayserilioglu
For example, use `.fetch` or `.dig` instead of `[]`, and use `===` instead of `is_a?` for checking types of objects. https://github.com/ruby/prism/commit/548b54915f
2024-03-06[ruby/prism] Fix implicit local variables in hashesKevin Newton
https://github.com/ruby/prism/commit/05e0c6792c
2024-03-06[ruby/prism] Add stubs for remaining ripper visit methodsKevin Newton
https://github.com/ruby/prism/commit/4ba9abf664
generated by cgit v1.2.3 (git 2.25.1) at 2025-06-17 18:24:50 +0000