summaryrefslogtreecommitdiff
path: root/lib/prism
AgeCommit message (Collapse)Author
2024-05-10[ruby/prism] Prism::CallNode#full_message_locKevin Newton
https://github.com/ruby/prism/commit/fa6fe9be84
2024-05-08[PRISM] Sync prism version bumpKevin Newton
2024-05-04[ruby/prism] Use `version: 3.3.1` against `Translation::Parser`Koichi ITO
Follow up https://github.com/ruby/prism/pull/2760. This PR updates the `Translation::Parser` to use version 3.3.1 when the version 3.3 is specified. The Parser gem is structured to support the latest patch versions, hence this aligns with Parser-compatible versioning. As noted in https://github.com/ruby/prism/pull/2760, the behavior remains unchanged with this switch from 3.3.0 to 3.3.1. https://github.com/ruby/prism/commit/efde09d318
2024-05-03[ruby/prism] Prism::Location#adjoinKevin Newton
https://github.com/ruby/prism/commit/a298db68e3
2024-05-03[ruby/prism] Prism::Node#tunnelKevin Newton
https://github.com/ruby/prism/commit/0bf5d651da
2024-05-03[ruby/prism] Create specialized `ASCIISource` with asciionly optimizationsVinicius Stock
https://github.com/ruby/prism/commit/40993166a8
2024-05-03[ruby/prism] Remove deprecated #child usageKevin Newton
https://github.com/ruby/prism/commit/14d9b431e2
2024-05-03[ruby/prism] Change ConstantPathNode#child to ConstantPathNode#{name,name_loc}Kevin Newton
This has been requested for a long time, and I'm finally doing it now. Unfortunately this is a breaking change for all of the APIs. I've added in a Ruby method for `#child` that is deprecated so that existing usage doesn't break, but for everyone else this is going to be a bit of a pain. https://github.com/ruby/prism/commit/9cbe74464e
2024-05-03[ruby/prism] Assume eval context for ruby_parser and ripperKevin Newton
https://github.com/ruby/prism/commit/e4d6984892
2024-05-03[ruby/prism] Assume an eval context for `Prism::Translation::Parser`Earlopain
This is similar to https://github.com/davidwessman/syntax_tree-erb/issues/81 but for RuboCop The parser gem doesn't support these types of checks, see https://github.com/whitequark/parser?tab=readme-ov-file#syntax-check-of-block-exits While this is technically a bug in the parser gem, it does increase compatibility and allows prism to be used when linting erb or haml with a RuboCop extension. https://github.com/ruby/prism/commit/6c59ae6a00
2024-05-03[ruby/prism] Remove generics from sorbet typesKevin Newton
https://github.com/ruby/prism/commit/080d84fd03
2024-05-02[ruby/prism] Node#script_lines and supporting infraKevin Newton
https://github.com/ruby/prism/commit/cb4a8ab772
2024-05-01[ruby/prism] Support passing version 3.3.1Kevin Newton
https://github.com/ruby/prism/commit/445a0f0d22
2024-04-26[ruby/prism] Bump to v0.27.0Kevin Newton
https://github.com/ruby/prism/commit/c9edeef91a
2024-04-26[ruby/prism] Location#slice_lines, Node#slice_linesKevin Newton
https://github.com/ruby/prism/commit/9b61f6fdb3
2024-04-25[ruby/prism] Remove need for Natalie patchesKevin Newton
https://github.com/ruby/prism/commit/7fc7e13476
2024-04-24[ruby/prism] Change inspect from recursive to a queueKevin Newton
We would previously cause a stack overflow if we parsed a file that was too deeply nested when we were calling inspect. Instead, we now use a queue of commands to do it linearly so we don't. https://github.com/ruby/prism/commit/0f21f5bfe1
2024-04-23[ruby/prism] srange_find should only look on current lineKevin Newton
https://github.com/ruby/prism/commit/3604aa15e7
2024-04-23[ruby/prism] Fix up rdocKevin Newton
Fixes https://github.com/ruby/prism/issues/2572 https://github.com/ruby/prism/commit/a446580e75
2024-04-19[ruby/prism] Split parse result based on typeKevin Newton
https://github.com/ruby/prism/commit/17194e096d
2024-04-19[ruby/prism] Fix up ruby_parser translation for dstrKevin Newton
https://github.com/ruby/prism/commit/b0fa4b7cd8
2024-04-19[ruby/prism] Fix it parameters for parser translationKevin Newton
https://github.com/ruby/prism/commit/2f3feb8d51
2024-04-18[ruby/prism] Bump to v0.26.0Kevin Newton
https://github.com/ruby/prism/commit/eadb09ef36
2024-04-18[ruby/prism] "Fix" transpose issue in parser compilerKevin Newton
https://github.com/ruby/prism/commit/593d637178
2024-04-17add #2709’s new RBIs to `.gemspec`ParadoxV5
2024-04-17[ruby/prism] Add a reflection API for determining the fields of a nodeKevin Newton
https://github.com/ruby/prism/commit/f3f9950a74
2024-04-17[ruby/prism] Inline pm_state_stackKevin Newton
2024-04-12[ruby/prism] Fix parser translation's heredoc whitespace calculationStan Lo
Given this example: ```rb <<~HEREDOC #{x} HEREDOC ``` Both the parser gem and Prism's translation layer would generate the following AST: ``` s(:dstr, s(:begin, s(:int, 1)), s(:str, " a\n")) ``` However, the parser gem inserts a empty string node into this node's location, like: ``` <Parser::Source::Map::Heredoc:0x0000000104ce73b8 @expression=#<Parser::Source::Range (string) 0...10>, @heredoc_body=#<Parser::Source::Range (string) 11...20>, @heredoc_end=#<Parser::Source::Range (string) 20...27>, @node=s(:dstr, s(:str, ""), s(:begin, s(:int, 1)), s(:str, " a\n"))> ``` This is required to calculate the correct whitespace for the heredoc body. We need to adjust the translation layer to account for this. With this fix, we also won't need to ignore the tilde heredoc fixture anymore. https://github.com/ruby/prism/commit/e7372e3ba5
2024-04-11[ruby/prism] Add Location#chopKevin Newton
https://github.com/ruby/prism/commit/5dd57f4b84
2024-04-05[ruby/prism] Bump to v0.25.0Kevin Newton
https://github.com/ruby/prism/commit/4da514456f
2024-04-04[ruby/prism] Fix up some comments in the parser compilerKevin Newton
https://github.com/ruby/prism/commit/e2147cddd8
2024-03-28[ruby/prism] CLI -x flagKevin Newton
https://github.com/ruby/prism/commit/2068e3c30a
2024-03-26[ruby/prism] Freeze internal parts, againKevin Newton
https://github.com/ruby/prism/commit/50372fee5c
2024-03-25[ruby/prism] Fix incorrect paring when using invalid regexp optionsKoichi ITO
Fixes https://github.com/ruby/prism/pull/2617. There was an issue with the lexer as follows. The following are valid regexp options: ```console $ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/io").value.map {|token| token[0].type }' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :EOF] ``` The following are invalid regexp options. Unnecessary the `IDENTIFIER` token is appearing: ```console $ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/az").value.map {|token| token[0].type }' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :IDENTIFIER, :EOF] ``` As a behavior of Ruby, when given `A` to `Z` and `a` to `z`, they act as invalid regexp options. e.g., ```console $ ruby -e '/regexp/az' -e:1: unknown regexp options - az /regexp/az -e: compile error (SyntaxError) ``` Thus, it should probably not be construed as `IDENTIFIER` token. Therefore, `pm_byte_table` has been adapted to accept those invalid regexp option values. Whether it is a valid regexp option or not is checked by `pm_regular_expression_flags_create`. For invalid regexp options, `PM_ERR_REGEXP_UNKNOWN_OPTIONS` is added to diagnostics. https://github.com/ruby/prism/commit/d2a6096fcf
2024-03-25[ruby/prism] Use Sexp#line_max not Sexp#max_lineJustin Collins
for RubyParser translation https://github.com/ruby/prism/commit/a37169621a
2024-03-19[ruby/prism] Fix a diagnostic incompatibilityKoichi ITO
This PR fixes a diagnostic incompatibility when using no anonymous keyword rest parameter: ```ruby foo(**) ``` Note, although the actual update applies only to the `foo(**)` case, for reference, `foo(*)` and `foo(&) are also mentioned below. ## Ruby (Expected) ```console $ ruby -cve 'foo(*)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] -e:1: no anonymous rest parameter -e: compile error (SyntaxError) $ ruby -cve 'foo(**)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] -e:1: no anonymous keyword rest parameter -e: compile error (SyntaxError) $ ruby -cve 'foo(&)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] -e:1: no anonymous block parameter -e: compile error (SyntaxError) ``` ## Prism (Actual) Before: ```console $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(*)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:argument_no_forwarding_star @message="unexpected `*` when the parent method is not forwarding" @location=#<Prism::Location @start_offset=4 @length=1 start_line=1> @level=:fatal>] $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(**)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:expect_expression_after_splat_hash @message="expected an expression after `**` in a hash" @location=#<Prism::Location @start_offset=4 @length=2 start_line=1> @level=:fatal>] $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(&)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:argument_no_forwarding_amp @message="unexpected `&` when the parent method is not forwarding" @location=#<Prism::Location @start_offset=4 @length=1 start_line=1> @level=:fatal>] ``` After: ```console $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(*)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:argument_no_forwarding_star @message="unexpected `*` when the parent method is not forwarding" @location=#<Prism::Location @start_offset=4 @length=1 start_line=1> @level=:fatal>] $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(**)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:argument_no_forwarding_star_star @message="unexpected `**` when the parent method is not forwarding" @location=#<Prism::Location @start_offset=4 @length=2 start_line=1> @level=:fatal>] $ bundle exec ruby -Ilib -rprism -wve 'p Prism.parse("foo(&)").errors' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [#<Prism::ParseError @type=:argument_no_forwarding_amp @message="unexpected `&` when the parent method is not forwarding" @location=#<Prism::Location @start_offset=4 @length=1 start_line=1> @level=:fatal>] ``` https://github.com/ruby/prism/commit/633c9d9fd4
2024-03-19[ruby/prism] Fix a diagnostic incompatibility for `Prism::Translation::Parser`Koichi ITO
This PR fixes a diagnostic incompatibility for `Prism::Translation::Parser` when using constant argument: ```ruby def foo(A) end ``` ## Parser gem (Expected) Displays `formal argument cannot be a constant (Parser::SyntaxError)`: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve 'p Parser::Ruby33.parse("def foo(A) end")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] (string):1:9: error: formal argument cannot be a constant (string):1: def foo(A) end (string):1: ^ /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72: in `process': formal argument cannot be a constant (Parser::SyntaxError) from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:274:in `diagnostic' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/ruby33.rb:12177:in `_reduce_663' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/racc-1.7.3/lib/racc/parser.rb:267:in `_racc_do_parse_c' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/racc-1.7.3/lib/racc/parser.rb:267:in `do_parse' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:190:in `parse' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse' from -e:1:in `<main>' ``` ## `Prism::Translation::Parser` (Actual) Previously, the error messages displayed by the Parser gem were different. Before: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'Prism::Translation::Parser33.parse("def foo(A) end")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] (string):1:9: error: (string):1: def foo(A) end (string):1: ^ /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72: in `process': Parser::SyntaxError (Parser::SyntaxError) from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `block in unwrap' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `each' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `unwrap' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse' from -e:1:in `<main>' ``` After: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'Prism::Translation::Parser33.parse("def foo(A) end")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] (string):1:9: error: formal argument cannot be a constant (string):1: def foo(A) end (string):1: ^ /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72: in `process': formal argument cannot be a constant (Parser::SyntaxError) from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `block in unwrap' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `each' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `unwrap' from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse' from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse' from -e:1:in `<main>' ``` https://github.com/ruby/prism/commit/4f2af88520
2024-03-18[ruby/prism] Use `require_relative` in the Prism codebaseKoichi ITO
If there are many searches in the `$LOAD_PATH` in the user environment, require will perform unnecessary searches that are not needed. In contrast, `require_relative` is efficient because it uses a relative path. https://github.com/ruby/prism/commit/438ccc67bd
2024-03-16[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using backquoted heredoc indetiner: ```ruby <<-` FOO` a b FOO ``` ## Parser gem (Expected) Returns `tXSTRING_BEG` as the first token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, [" FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING_BEG` token and value of `tSTRING_END` token. ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tSTRING_BEG, ["<<\"", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, ["` FOO`", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bunlde exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "a\n"), s(:str, "b\n")), [], [[:tXSTRING_BEG, ["<<`", #<Parser::Source::Range example.rb 0...10>]], [:tSTRING_CONTENT, ["a\n", #<Parser::Source::Range example.rb 11...13>]], [:tSTRING_CONTENT, ["b\n", #<Parser::Source::Range example.rb 13...15>]], [:tSTRING_END, [" FOO", #<Parser::Source::Range example.rb 15...23>]], [:tNL, [nil, #<Parser::Source::Range example.rb 10...11>]]]] ``` https://github.com/ruby/prism/commit/308f8d85a1
2024-03-15[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using escaped backslash in string literal: ```ruby "\\ foo \\ bar" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]], [:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]] ``` The reproduction test is based on the following strings.txt and exists: https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79 But, the restoration has not yet been performed due to remaining other issues in strings.txt. https://github.com/ruby/prism/commit/2c44e7e307
2024-03-15[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for the heredocs_leading_whitespace.txt test. https://github.com/ruby/prism/commit/7d45fb1eed
2024-03-15[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for dstring literal: ```ruby "foo #{bar}" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:dstr, s(:str, "foo\n"), s(:str, " ")), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` https://github.com/ruby/prism/commit/c1652a9ee7
2024-03-15[ruby/prism] Shareable constant nodesKevin Newton
https://github.com/ruby/prism/commit/473cfed6d0
2024-03-13[ruby/prism] Fix diagnostic incompatibility for `Prism::Translation::Parser`Koichi ITO
In the case of the `**` and `&` ambiguous prefixes, incompatibilities remained among https://github.com/ruby/prism/issues/2513. https://github.com/ruby/prism/commit/3b8b231aae
2024-03-13[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for empty xstring literal. ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different: ```console $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` https://github.com/ruby/prism/commit/4ac89dcbb5
2024-03-13[ruby/prism] Track parentheses in patternsKevin Newton
https://github.com/ruby/prism/commit/62db99f156
2024-03-13[ruby/prism] Fix an AST incompatibility for `Prism::Translation::Parser`Koichi ITO
Fixes ruby/prism#2480. This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser` for xstring literal with line breaks. The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576: ```ruby "foo bar" ``` https://github.com/ruby/prism/commit/cf85e72c55
2024-03-12[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`. The former is used in xstrings like `` `foo` ``, while the latter is utilized as a back reference in contexts like `` A::` ``. This PR will make corrections to differentiate the use of `BACKTICK`. This mistake was discovered through the investigation of xstring.txt file. The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail, hence it will be separated into xstring_with_backslash.txt file. This separation will facilitate addressing the correction at a different time. https://github.com/ruby/prism/commit/49ad8df40a
2024-03-12[ruby/prism] Fix some whitequark/parser lexer compatibilitiesKevin Newton
https://github.com/ruby/prism/commit/34e521d071
2024-03-12[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`Koichi ITO
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`: ## Parser gem (Expected) Returns `tBACK_REF2` token: ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the parser returned `tXSTRING_BEG` token when parsing the following: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tXSTRING_BEG, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` After the update, the parser now returns `tBACK_REF2` token for the same input: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]], [:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]] ``` This correction enables the restoration of `constants.txt` as a test case. https://github.com/ruby/prism/commit/7f63b28f98
generated by cgit v1.2.3 (git 2.25.1) at 2025-06-20 05:39:22 +0000