| Age | Commit message (Collapse) | Author |
|---|
|
The macro SafeStringValue() became just StringValue() in c5c05460ac2,
and it is deprecated nowadays.
This patch replaces remaining macro usage. Some occurrences are left in
ext/stringio and ext/win32ole, they should be fixed upstream.
The macro itself is not deleted, because it may be used in extensions.
|
|
|
|
This reverts commit 4ba8f0dc993953d3ddda6328e3ef17a2fc2cbde5.
|
|
This reverts commit 6185cfdf38e26026c6d38220eeca48689e54cdcf.
|
|
This reverts commit bdafad879093ef16a9a649154c4b2e4ebf492656.
|
|
This reverts commit d10bc3a2b8300cffc383e10c3730871e851be24c.
|
|
ASAN leaves a pointer to the fake frame on the stack; we can use the
__asan_addr_is_in_fake_stack API to work out the extent of the fake
stack and thus mark any VALUEs contained therein.
[Bug #20001]
|
|
__has_feature is a clang-ism, and GCC has a different way to tell if
sanitizers are enabled. For this reason, I don't want to spray
__has_feature all over the codebase for other places where conditional
compilation based on sanitizers is required.
[Bug #20001]
|
|
Where a local variable is used as part of the stack bounds detection, it
has to actually be on the stack. ASAN can put local variable on "fake
stacks", however, with addresses in different memory mappings. This
completely destroys the stack bounds calculation, and can lead to e.g.
things not getting GC marked on the machine stack or stackoverflow
checks that always fail.
The __asan_addr_is_in_fake_stack helper can be used to get the _real_
stack address of such variables, and thus perform the stack size
calculation properly
[Bug #20001]
|
|
The implementation of `native_thread_init_stack` for the various
threading models can use the address of a local variable as part of the
calculation of the machine stack extents:
* pthreads uses it as a lower-bound on the start of the stack, because
glibc (and maybe other libcs) can store its own data on the stack
before calling into user code on thread creation.
* win32 uses it as an argument to VirtualQuery, which gets the extent of
the memory mapping which contains the variable
However, the local being used for this is actually allocated _inside_
the `native_thread_init_stack` frame; that means the caller might
allocate a VALUE on the stack that actually lies outside the bounds
stored in machine.stack_{start,end}.
A local variable from one level above the topmost frame that stores
VALUEs on the stack must be drilled down into the call to
`native_thread_init_stack` to be used in the calculation. This probably
doesn't _really_ matter for the win32 case (they'll be in the same
memory mapping so VirtualQuery should return the same thing), but
definitely could matter for the pthreads case.
[Bug #20001]
|
|
The environ is malloc'd, so it gets reported as a memory leak. This
commit adds ruby_free_proctitle which frees it during shutdown when
RUBY_FREE_AT_EXIT is set.
STACK OF 1 INSTANCE OF 'ROOT LEAK: <calloc in ruby_init_setproctitle>':
5 dyld 0x18b7090e0 start + 2360
4 ruby 0x10000e3a8 main + 100 main.c:58
3 ruby 0x1000b4dfc ruby_options + 180 eval.c:121
2 ruby 0x1001c5f70 ruby_process_options + 200 ruby.c:3014
1 ruby 0x10035c9fc ruby_init_setproctitle + 76 setproctitle.c:105
0 libsystem_malloc.dylib 0x18b8c7b78 _malloc_zone_calloc_instrumented_or_legacy + 100
|
|
I was trying to debug an (unrelated) issue in the GC, and wanted to turn
on the trace-level GC output by compiling it with -DRGENGC_DEBUG=5.
Unfortunately, this actually causes a crash in newobj_init() because the
code there tries to log the obj_info() of the newly created object.
However, the object is not actually sufficiently set up for some of the
things that obj_info() tries to do:
* The instance variable table for a class is not yet initialized, and
when using variable-length RVALUES, said ivar table is embedded in
as-yet unitialized memory after the struct RValue. Attempting to read
this, as obj_info() does, causes a crash.
* T_DATA variables need to dereference their ->type field to print out
the underlying C type name, which is not set up until newobj_fill() is
called.
To fix this, create a new method `obj_info_basic`, which dumps out only
the parts of the object that are valid before the object is fully
initialized.
[Fixes #18795]
|
|
`:sym` was managed by `NODE_LIT` with `Symbol` object.
This commit introduces `NODE_SYM` so that
1. Symbol literal is detectable from AST Node
2. Reduce dependency on ruby object
|
|
Change the argument to align with other node value functions
like `rb_node_line_lineno_val`.
|
|
|
|
Before this patch, the MN scheduler waits for the IO with the
following steps:
1. `poll(fd, timeout=0)` to check fd is ready or not.
2. if fd is not ready, waits with MN thread scheduler
3. call `func` to issue the blocking I/O call
The advantage of advanced `poll()` is we can wait for the
IO ready for any fds. However `poll()` becomes overhead
for already ready fds.
This patch changes the steps like:
1. call `func` to issue the blocking I/O call
2. if the `func` returns `EWOULDBLOCK` the fd is `O_NONBLOCK`
and we need to wait for fd is ready so that waits with MN
thread scheduler.
In this case, we can wait only for `O_NONBLOCK` fds. Otherwise
it waits with blocking operations such as `read()` system call.
However we don't need to call `poll()` to check fd is ready
in advance.
With this patch we can observe performance improvement
on microbenchmark which repeats blocking I/O (not
`O_NONBLOCK` fd) with and without MN thread scheduler.
```ruby
require 'benchmark'
f = open('/dev/null', 'w')
f.sync = true
TN = 1
N = 1_000_000 / TN
Benchmark.bm{|x|
x.report{
TN.times.map{
Thread.new{
N.times{f.print '.'}
}
}.each(&:join)
}
}
__END__
TN = 1
user system total real
ruby32 0.393966 0.101122 0.495088 ( 0.495235)
ruby33 0.493963 0.089521 0.583484 ( 0.584091)
ruby33+MN 0.639333 0.200843 0.840176 ( 0.840291) <- Slow
this+MN 0.512231 0.099091 0.611322 ( 0.611074) <- Good
```
|
|
`__FILE__` was managed by `NODE_STR` with `String` object.
This commit introduces `NODE_FILE` and `struct rb_parser_string` so that
1. `__FILE__` is detectable from AST Node
2. Reduce dependency ruby object
|
|
`__LINE__` was managed by `NODE_LIT` with `Integer` object.
This commit introduces `NODE_LINE` so that
1. `__LINE__` is detectable from AST Node
2. Reduce dependency ruby object
|
|
st_replace and st_init_existing_table_with_size are functions used
internally in Ruby and should not be publicly visible.
|
|
|
|
|
|
We should set the m_tbl right after allocation before anything that can
trigger GC to avoid clone_p from becoming old and needing to fire write
barriers.
Co-authored-by: Aaron Patterson <[email protected]>
|
|
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
|
|
Objects with the same shape must always have the same "embeddedness"
(either embedded or heap allocated) because YJIT assumes so. However,
using remove_instance_variable, it's possible that some objects are
embedded and some are heap allocated because it does not re-embed heap
allocated objects.
This commit changes remove_instance_variable to re-embed Object
instance variables when it becomes small enough.
|
|
Embedded shared strings cannot be moved because strings point into the
slot of the shared string. There may be code using the RSTRING_PTR on
the stack, which would pin the string but not pin the shared string,
causing it to move.
|
|
When generic instance variable has a shape, it is marked movable. If it
it transitions to too complex, it needs to update references otherwise
it may have incorrect references.
|
|
Too complex classes use a hash table to store ivs, and should always pin
their IVs. We shouldn't touch those classes in compaction.
|
|
That function is a bit too low level to called from multiple
places. It's always used in tandem with `rb_shape_set_too_complex`
and both have to know how the object is laid out to update the
`iv_ptr`.
So instead we can provide two higher level function:
- `rb_obj_copy_ivs_to_hash_table` to prepare a `st_table` from an
arbitrary oject.
- `rb_obj_convert_to_too_complex` to assign the new `st_table`
to the old object, and safely free the old `iv_ptr`.
Unfortunately both can't be combined into one, because `rb_obj_copy_ivar`
need `rb_obj_copy_ivs_to_hash_table` to copy from one object
to another.
|
|
It's only used to allocate the table with the right size,
but in some case we were passing `rb_shape_get_shape_by_id(SHAPE_OBJ_TOO_COMPLEX)`
which `next_iv_index` is a bit undefined.
So overall we're better to just allocate a table the size of the existing
object, it should be close enough in the vast majority of cases,
and that's already a de-optimizaton path anyway.
|
|
Right now the `rb_shape_get_next` shape caller need to
first check if there is capacity left, and if not call
`rb_shape_transition_shape_capa` before it can call `rb_shape_get_next`.
And on each of these it needs to checks if we got a TOO_COMPLEX
back.
All this logic is duplicated in the interpreter, YJIT and RJIT.
Instead we can have `rb_shape_get_next` do the capacity transition
when needed. The caller can compare the old and new shapes capacity
to know if resizing is needed. It also can check for TOO_COMPLEX
only once.
|
|
|
|
|
|
This commit changes generic ivars to respect the capacity transition in
shapes rather than growing the capacity independently.
|
|
|
|
Previously the growth was 3(embed), 6, 12, 24, ...
With this change it's now 3(embed), 8, 16, 32, 64, ... by default.
However, since power of two isn't the best size for all allocators,
if `malloc_usable_size` is vailable, we use it to discover the best
offset.
On Linux/glibc 2.35 for instance, the growth will be 3(embed), 7, 15, 31
to avoid wasting 8B per object.
Test program:
```c
size_t test(size_t slots) {
size_t allocated = slots * VALUE_SIZE;
void *test_ptr = malloc(allocated);
size_t wasted = malloc_usable_size(test_ptr) - allocated;
free(test_ptr);
fprintf(stderr, "slots = %lu, wasted_bytes = %lu\n", slots, wasted);
return wasted;
}
int main(int argc, char *argv[]) {
size_t best_padding = 0;
size_t padding = 0;
for (padding = 0; padding <= 2; padding++) {
size_t wasted = test(8 - padding);
if (wasted == 0) {
best_padding = padding;
break;
}
}
size_t index = 0;
fprintf(stderr, "=============== naive ================\n");
size_t list_size = 4;
for (index = 0; index < 10; index++) {
test(list_size);
list_size *= 2;
}
fprintf(stderr, "=============== auto-padded (-%lu) ================\n", best_padding);
list_size = 4;
for (index = 0; index < 10; index ++) {
test(list_size - best_padding);
list_size *= 2;
}
fprintf(stderr, "\n\n");
return 0;
}
```
```
===== glibc ======
slots = 8, wasted_bytes = 8
slots = 7, wasted_bytes = 0
=============== naive ================
slots = 4, wasted_bytes = 8
slots = 8, wasted_bytes = 8
slots = 16, wasted_bytes = 8
slots = 32, wasted_bytes = 8
slots = 64, wasted_bytes = 8
slots = 128, wasted_bytes = 8
slots = 256, wasted_bytes = 8
slots = 512, wasted_bytes = 8
slots = 1024, wasted_bytes = 8
slots = 2048, wasted_bytes = 8
=============== auto-padded (-1) ================
slots = 3, wasted_bytes = 0
slots = 7, wasted_bytes = 0
slots = 15, wasted_bytes = 0
slots = 31, wasted_bytes = 0
slots = 63, wasted_bytes = 0
slots = 127, wasted_bytes = 0
slots = 255, wasted_bytes = 0
slots = 511, wasted_bytes = 0
slots = 1023, wasted_bytes = 0
slots = 2047, wasted_bytes = 0
```
```
========== jemalloc =======
slots = 8, wasted_bytes = 0
=============== naive ================
slots = 4, wasted_bytes = 0
slots = 8, wasted_bytes = 0
slots = 16, wasted_bytes = 0
slots = 32, wasted_bytes = 0
slots = 64, wasted_bytes = 0
slots = 128, wasted_bytes = 0
slots = 256, wasted_bytes = 0
slots = 512, wasted_bytes = 0
slots = 1024, wasted_bytes = 0
slots = 2048, wasted_bytes = 0
=============== auto-padded (-0) ================
slots = 4, wasted_bytes = 0
slots = 8, wasted_bytes = 0
slots = 16, wasted_bytes = 0
slots = 32, wasted_bytes = 0
slots = 64, wasted_bytes = 0
slots = 128, wasted_bytes = 0
slots = 256, wasted_bytes = 0
slots = 512, wasted_bytes = 0
slots = 1024, wasted_bytes = 0
slots = 2048, wasted_bytes = 0
```
|
|
... because GCC 13 warns it.
```
In file included from class.c:24:
In function ‘RCLASS_SET_ALLOCATOR’,
inlined from ‘class_alloc’ at class.c:251:5,
inlined from ‘rb_module_s_alloc’ at class.c:1045:17:
internal/class.h:159:43: warning: array subscript 0 is outside array bounds of ‘rb_classext_t[0]’ {aka ‘struct rb_classext_struct[]’} [-Warray-bounds=]
159 | RCLASS_EXT(klass)->as.class.allocator = allocator;
| ^
```
https://rubyci.s3.amazonaws.com/arch/ruby-master/log/20231015T030003Z.log.html.gz
|
|
|
|
|
|
|
|
This patch introduce M:N thread scheduler for Ractor system.
In general, M:N thread scheduler employs N native threads (OS threads)
to manage M user-level threads (Ruby threads in this case).
On the Ruby interpreter, 1 native thread is provided for 1 Ractor
and all Ruby threads are managed by the native thread.
From Ruby 1.9, the interpreter uses 1:1 thread scheduler which means
1 Ruby thread has 1 native thread. M:N scheduler change this strategy.
Because of compatibility issue (and stableness issue of the implementation)
main Ractor doesn't use M:N scheduler on default. On the other words,
threads on the main Ractor will be managed with 1:1 thread scheduler.
There are additional settings by environment variables:
`RUBY_MN_THREADS=1` enables M:N thread scheduler on the main ractor.
Note that non-main ractors use the M:N scheduler without this
configuration. With this configuration, single ractor applications
run threads on M:1 thread scheduler (green threads, user-level threads).
`RUBY_MAX_CPU=n` specifies maximum number of native threads for
M:N scheduler (default: 8).
This patch will be reverted soon if non-easy issues are found.
[Bug #19842]
|
|
Each bit run is upto the right shift count, so the each mask does not
need more upper bits.
|
|
This commit moves IO#readline to Ruby. In order to call C functions,
keyword arguments must be converted to hashes. Prior to this commit,
code like `io.readline(chomp: true)` would allocate a hash. This
commits moves the keyword "denaturing" to Ruby, allowing us to send
positional arguments to the C API and avoiding the hash allocation.
Here is an allocation benchmark for the method:
```
x = GC.stat(:total_allocated_objects)
File.open("/usr/share/dict/words") do |f|
f.readline(chomp: true) until f.eof?
end
p ALLOCATIONS: GC.stat(:total_allocated_objects) - x
```
Before this commit, the output was this:
```
$ make run
./miniruby -I./lib -I. -I.ext/common -r./arm64-darwin22-fake ./test.rb
{:ALLOCATIONS=>707939}
```
Now it is this:
```
$ make run
./miniruby -I./lib -I. -I.ext/common -r./arm64-darwin22-fake ./test.rb
{:ALLOCATIONS=>471962}
```
[Bug #19890] [ruby-core:114803]
|
|
|
|
It has precedence over the environment variable `RUBY_BUGREPORT_PATH`.
|
|
|
|
|
|
Revert commit "Directly allocate FrozenCore as an ICLASS",
813a5f4fc46a24ca1695d23c159250b9e1080ac7.
|
|
WeakMap can crash during compaction because the st_insert could allocate
memory.
|
|
If we're during incremental marking, then Ruby code can execute that
deallocates certain memory buffers that have been called with
rb_gc_mark_weak, which can cause use-after-free bugs.
Notes:
Merged: https://github.com/ruby/ruby/pull/8375
|
|
Previously we used the next character following the found prefix to
determine if the match ended on a broken character.
This had caused surprising behaviour when a valid character was followed
by a UTF-8 continuation byte.
This commit changes the behaviour to instead look for the end of the
last character in the prefix.
[Bug #19784]
Co-authored-by: ywenc <[email protected]>
Co-authored-by: Nobuyoshi Nakada <[email protected]>
Notes:
Merged: https://github.com/ruby/ruby/pull/8348
|
