| Age | Commit message (Collapse) | Author |
|---|
|
* Add `open_timeout` as an overall timeout option for `Socket.tcp`
[Background]
Currently, `TCPSocket.new` and `Socket.tcp` accept two kind of timeout options:
- `resolv_timeout`, which controls the timeout for DNS resolution
- `connect_timeout`, which controls the timeout for the connection attempt
With the introduction of Happy Eyeballs Version 2 (as per [RFC 8305](https://datatracker.ietf.org/doc/html/rfc8305)) in[ Feature #20108](https://bugs.ruby-lang.org/issues/20108) and [Feature #20782](https://bugs.ruby-lang.org/issues/20782), both address resolution and connection attempts are now parallelized.
As a result, the sum of `resolv_timeout` and `connect_timeout` no longer represents the total timeout duration. This is because, in HEv2, name resolution and connection attempts are performed concurrently, causing the two timeouts to overlap.
Example:
When `resolv_timeout: 200ms` and `connect_timeout: 100ms` are set:
1. An IPv6 address is resolved after the method starts immediately (IPv4 is still being resolved).
2. A connection attempt is initiated to the IPv6 address
3. After 100ms, `connect_timeout` is exceeded. However, since `resolv_timeout` still has 100ms left, the IPv4 resolution continues.
4. After 200ms from the start, the method raises a `resolv_timeout` error.
In this case, the total elapsed time before a timeout is 200ms, not the expected 300ms (100ms + 200ms).
Furthermore, in HEv2, connection attempts are also parallelized.
It starts a new connection attempts every 250ms for resolved addresses. This makes the definition of `connect_timeout` even more ambiguous—specifically, it becomes unclear from which point the timeout is counted.
Additionally, these methods initiate new connection attempts every 250ms (Connection Attempt Delay) for each candidate address, thereby parallelizing connection attempts. However, this behavior makes it unclear from which point in time the connect_timeout is actually measured.
Currently, a `connect_timeout` is raised only after the last connection attempt exceeds the timeout.
Example:
When `connect_timeout: 100ms` is set and 3 address candidates:
1. Start a connection attempt to the address `a`
2. 250ms after step 1, start a new connection attempt to the address `b`
3. 500ms after step 1, start a new connection attempt to the address `c`
4. 1000ms after step 3 (1000ms after starting the connection to `c`, 1250ms after starting the connection to `b,` and 1500ms after starting the connection to `a`) `connect_timeout` is raised
This behavior aims to favor successful connections by allowing more time for each attempt, but it results in a timeout model that is difficult to reason about.
These methods have supported `resolv_timeout` and `connect_timeout` options even before the introduction of HEv2. However, in many use cases, it would be more convenient if a timeout occurred after a specified duration from the start of the method. Similar functions in other languages (such as PHP, Python, and Go) typically allow specifying only an overall timeout.
[Proposal]
I propose adding an `open_timeout` option to `Socket.tcp` in this PR, which triggers a timeout after a specified duration has elapsed from the start of the method.
The name `open_timeout` aligns with the existing accessor used in `Net::HTTP`.
If `open_timeout` is specified together with `resolv_timeout` and `connect_timeout`, I propose that only `open_timeout` be used and the others be ignored. While it is possible to support combinations of `open_timeout`, `resolv_timeout`, and `connect_timeout`, doing so would require defining which timeout takes precedence in which situations. In this case, I believe it is more valuable to keep the behavior simple and easy to understand, rather than supporting more complex use cases.
If this proposal is accepted, I also plan to extend `open_timeout` support to `TCPSocket.new`.
While the long-term future of `resolv_timeout` and `connect_timeout` may warrant further discussion, I believe the immediate priority is to offer a straightforward way to specify an overall timeout.
[Outcome]
If `open_timeout` is also supported by `TCPSocket.new`, users would be able to manage total connection timeouts directly in `Net::HTTP#connect` without relying on `Timeout.timeout`.
https://github.com/ruby/ruby/blob/aa0f689bf45352c4a592e7f1a044912c40435266/lib/net/http.rb#L1657
---
* Raise an exception if it is specified together with other timeout options
> If open_timeout is specified together with resolv_timeout and connect_timeout, I propose that only open_timeout be used and the others be ignored.
Since this approach may be unclear to users, I’ve decided to explicitly raise an `ArgumentError` if these options are specified together.
* Add doc
* Fix: open_timeout error should be raised even if there are still addresses that have not been tried
Notes:
Merged-By: shioimm <[email protected]>
|
|
Allow Addrinfo objects to be shared among Ractors. Addrinfo objects are
already immutable, so I think it's safe for us to tag them as
RUBY_TYPED_FROZEN_SHAREABLE shareable too.
Notes:
Merged: https://github.com/ruby/ruby/pull/13388
|
|
We no longer execute those files with Solaris platforms.
Notes:
Merged: https://github.com/ruby/ruby/pull/13037
|
|
* Use `assert_raise_kind_of` instead of `rescue` and `flunk`.
* Use `assert_include` for the pattern that may contain regexp meta
characters.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12934
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12740
|
|
(#12678)
* Do not save ResolutionError if resolution succeeds for any address family
Socket with Happy Eyeballs Version 2 performs connection attempts and name resolution in parallel.
In the existing implementation, if a connection attempt failed for one address family while name resolution was still in progress for the other, and that name resolution later failed, the method would terminate with a name resolution error.
This behavior was intended to ensure that the final error reflected the most recent failure, potentially overriding an earlier error.
However, [Bug #21088](https://bugs.ruby-lang.org/issues/21088) made me realize that terminating with a name resolution error is unnatural when name resolution succeeded for at least one address family.
This PR modifies the behavior so that if name resolution succeeds for one address family, any name resolution error from the other is not saved.
This PR includes the following changes:
* Do not display select(2) as the system call that caused the raised error, as it is for internal processing
* Fix bug: Get errno with Socket::SO_ERROR in Windows environment with a workaround for tests not passing
Notes:
Merged-By: shioimm <[email protected]>
|
|
* TCPSocket.new: Close resources in ensure
* TCPSocket.new: Remove unnecessary comments
* Socket.tcp: Make assert_separately in TestSocket more readable
* Socket.tcp: Returning instead of exiting
* Socket.tcp: Close resources in ensure
* Socket.tcp: Avoid test failures on hosts that only support IPv4
Notes:
Merged-By: shioimm <[email protected]>
|
|
This change includes the following updates:
- Added an environment variable `RUBY_TCP_NO_FAST_FALLBACK` to control enabling/disabling fast_fallback
- Updated documentation and man pages
- Revised the implementation of Socket.tcp_fast_fallback= and Socket.tcp_fast_fallback, which previously performed dynamic name resolution of constants and variables. As a result, the following performance improvements were achieved:
(Case of 1000 executions of `TCPSocket.new` to the local host)
Rehearsal -----------------------------------------
before 0.031462 0.147946 0.179408 ( 0.249279)
after 0.031164 0.146839 0.178003 ( 0.346935)
-------------------------------- total: 0.178003sec
user system total real
before 0.027584 0.138712 0.166296 ( 0.233356)
after 0.025953 0.127608 0.153561 ( 0.237971)
Notes:
Merged-By: shioimm <[email protected]>
|
|
To verify the behavior of HEv2, some tests were prepared. But unexpected failures occur in certain environments.
This happens in environments where "localhost" resolves only to an IPv4 address during tests that verify connections to IPv6.
For example, the following situation can occur:
- The server process is bound to ::1.
- The client socket always resolves "localhost" to 127.0.0.1 and attempts to connect to 127.0.0.1.
- Since no server is bound to 127.0.0.1, an ECONNREFUSED error is raised.
In such situations, the behavior of `TCPSocket.new` remains unchanged from before the introduction of HEv2.
(The failures occur because tests explicitly binding to ::1 were added to verify HEv2 behavior.)
This change ensures that the affected tests are skipped in environments of this kind.
Notes:
Merged-By: shioimm <[email protected]>
|
|
* Do not save the last_error if there are no sockets waiting to be connected
In this implementation, the results of both name resolution and connection attempts are awaited using select(2).
When it returned, the implementation attempted to check for connections even if there were no sockets currently attempting to connect, treating the absence of connected sockets as a connection failure.
With this fix, it will no longer check for connections when there are no sockets waiting to be connected.
Additionally, the following minor fixes have been made:
* Handle failure of getsockopt(2) and removed unnecessary continue in the loop
* Tweak: Use common API to check in_progress_fds
* Safely call TCPServer.new in test
* Set empty writefds when there is no socket waiting to be connected
* Enable fast_fallback option
Notes:
Merged-By: shioimm <[email protected]>
|
|
* Disable HEv2 tests temporarily
To suppress error log output in CI.
They should have been DISABLE in PR #12070.
---
Additionally, the following fixes have been made:
- Remove unnecessary `assert_separately` from the related tests
Notes:
Merged-By: shioimm <[email protected]>
|
|
to suppress failing output in CI.
Notes:
Merged-By: shioimm <[email protected]>
|
|
TCPSocket.new (#11653)
* Introduction of Happy Eyeballs Version 2 (RFC8305) in TCPSocket.new
This is an implementation of Happy Eyeballs version 2 (RFC 8305) in `TCPSocket.new`.
See https://github.com/ruby/ruby/pull/11653
1. Background
Prior to this implementation, I implemented Happy Eyeballs Version 2 (HEv2) for `Socket.tcp` in https://github.com/ruby/ruby/pull/9374.
HEv2 is an algorithm defined in [RFC 8305](https://datatracker.ietf.org/doc/html/rfc8305), aimed at improving network connectivity.
For more details on the specific cases that HEv2 helps, please refer to https://bugs.ruby-lang.org/issues/20108.
2. Proposal & Outcome
This proposal implements the same HEv2 algorithm in `TCPSocket.new`.
Since `TCPSocket.new` is used more widely than `Socket.tcp`, this change is expected to broaden the impact of HEv2's benefits.
Like `Socket.tcp`, I have also added `fast_fallback` keyword argument to `TCPSocket.new`.
This option is set to true by default, enabling the HEv2 functionality.
However, users can explicitly set it to false to disable HEv2 and use the previous behavior of `TCPSocket.new`.
It should be noted that HEv2 is enabled only in environments where pthreads are available.
This specification follows the approach taken in https://bugs.ruby-lang.org/issues/19965 , where name resolution can be interrupted.
(In environments where pthreads are not available, the `fast_fallback` option is ignored.)
3. Performance
Below is the benchmark of 100 requests to `www.ruby-lang.org` with the fast_fallback option set to true and false, respectively.
While there is a slight performance degradation when HEv2 is enabled, the degradation is smaller compared to that seen in `Socket.tcp`.
```
~/s/build ❯❯❯ ../install/bin/ruby ../ruby/test.rb
Rehearsal --------------------------------------------------------
fast_fallback: true 0.017588 0.097045 0.114633 ( 1.460664)
fast_fallback: false 0.014033 0.078984 0.093017 ( 1.413951)
----------------------------------------------- total: 0.207650sec
user system total real
fast_fallback: true 0.020891 0.124054 0.144945 ( 1.473816)
fast_fallback: false 0.018392 0.110852 0.129244 ( 1.466014)
```
* Update debug prints
Co-authored-by: Nobuyoshi Nakada <[email protected]>
* Remove debug prints
* misc
* Disable HEv2 in Win
* Raise resolution error with hostname resolution
* Fix to handle errors
* Remove warnings
* Errors that do not need to be handled
* misc
* Improve doc
* Fix bug on cancellation
* Avoid EAI_ADDRFAMILY for resolving IPv6
* Follow upstream
* misc
* Refactor connection_attempt_fds management
- Introduced allocate_connection_attempt_fds and reallocate_connection_attempt_fds for improved memory allocation of connection_attempt_fds
- Added remove_connection_attempt_fd to resize connection_attempt_fds dynamically.
- Simplified the in_progress_fds function to only check the size of connection_attempt_fds.
* Rename do_pthread_create to raddrinfo_pthread_create to avoid conflicting
---------
Co-authored-by: Nobuyoshi Nakada <[email protected]>
Notes:
Merged-By: shioimm <[email protected]>
|
|
Some packaging systems that include support for running tests,
such as OpenBSD's, do not allow outbound network connections
during testing for security reasons. EACCES is the error raised by
OpenBSD in this case.
Notes:
Merged: https://github.com/ruby/ruby/pull/11535
Merged-By: jeremyevans <[email protected]>
|
|
|
|
|
|
* Introduction of Happy Eyeballs Version 2 (RFC8305) in Socket.tcp
This is an implementation of Happy Eyeballs version 2 (RFC 8305) in Socket.tcp.
[Background]
Currently, `Socket.tcp` synchronously resolves names and makes connection attempts with `Addrinfo::foreach.`
This implementation has the following two problems.
1. In name resolution, the program stops until the DNS server responds to all DNS queries.
2. In a connection attempt, while an IP address is trying to connect to the destination host and is taking time, the program stops, and other resolved IP addresses cannot try to connect.
[Proposal]
"Happy Eyeballs" ([RFC 8305](https://datatracker.ietf.org/doc/html/rfc8305)) is an algorithm to solve this kind of problem. It avoids delays to the user whenever possible and also uses IPv6 preferentially.
I implemented it into `Socket.tcp` by using `Addrinfo.getaddrinfo` in each thread spawned per address family to resolve the hostname asynchronously, and using `Socket::connect_nonblock` to try to connect with multiple addrinfo in parallel.
[Outcome]
This change eliminates a fatal defect in the following cases.
Case 1. One of the A or AAAA DNS queries does not return
---
require 'socket'
class Addrinfo
class << self
# Current Socket.tcp depends on foreach
def foreach(nodename, service, family=nil, socktype=nil, protocol=nil, flags=nil, timeout: nil, &block)
getaddrinfo(nodename, service, Socket::AF_INET6, socktype, protocol, flags, timeout: timeout)
.concat(getaddrinfo(nodename, service, Socket::AF_INET, socktype, protocol, flags, timeout: timeout))
.each(&block)
end
def getaddrinfo(_, _, family, *_)
case family
when Socket::AF_INET6 then sleep
when Socket::AF_INET then [Addrinfo.tcp("127.0.0.1", 4567)]
end
end
end
end
Socket.tcp("localhost", 4567)
---
Because the current `Socket.tcp` cannot resolve IPv6 names, the program stops in this case. It cannot start to connect with IPv4 address.
Though `Socket.tcp` with HEv2 can promptly start a connection attempt with IPv4 address in this case.
Case 2. Server does not promptly return ack for syn of either IPv4 / IPv6 address family
---
require 'socket'
fork do
socket = Socket.new(Socket::AF_INET6, :STREAM)
socket.setsockopt(:SOCKET, :REUSEADDR, true)
socket.bind(Socket.pack_sockaddr_in(4567, '::1'))
sleep
socket.listen(1)
connection, _ = socket.accept
connection.close
socket.close
end
fork do
socket = Socket.new(Socket::AF_INET, :STREAM)
socket.setsockopt(:SOCKET, :REUSEADDR, true)
socket.bind(Socket.pack_sockaddr_in(4567, '127.0.0.1'))
socket.listen(1)
connection, _ = socket.accept
connection.close
socket.close
end
Socket.tcp("localhost", 4567)
---
The current `Socket.tcp` tries to connect serially, so when its first name resolves an IPv6 address and initiates a connection to an IPv6 server, this server does not return an ACK, and the program stops.
Though `Socket.tcp` with HEv2 starts to connect sequentially and in parallel so a connection can be established promptly at the socket that attempted to connect to the IPv4 server.
In exchange, the performance of `Socket.tcp` with HEv2 will be degraded.
---
100.times { Socket.tcp("www.ruby-lang.org", 80) }
---
This is due to the addition of the creation of IO objects, Thread objects, etc., and calls to `IO::select` in the implementation.
* Avoid NameError of Socket::EAI_ADDRFAMILY in MinGW
* Support Windows with SO_CONNECT_TIME
* Improve performance
I have additionally implemented the following patterns:
- If the host is single-stack, name resolution is performed in the main thread. This reduces the cost of creating threads.
- If an IP address is specified, name resolution is performed in the main thread. This also reduces the cost of creating threads.
- If only one IP address is resolved, connect is executed in blocking mode. This reduces the cost of calling IO::select.
Also, I have added a fast_fallback option for users who wish not to use HE.
Here are the results of each performance test.
```ruby
require 'socket'
require 'benchmark'
HOSTNAME = "www.ruby-lang.org"
PORT = 80
ai = Addrinfo.tcp(HOSTNAME, PORT)
Benchmark.bmbm do |x|
x.report("Domain name") do
30.times { Socket.tcp(HOSTNAME, PORT).close }
end
x.report("IP Address") do
30.times { Socket.tcp(ai.ip_address, PORT).close }
end
x.report("fast_fallback: false") do
30.times { Socket.tcp(HOSTNAME, PORT, fast_fallback: false).close }
end
end
```
```
user system total real
Domain name 0.015567 0.032511 0.048078 ( 0.325284)
IP Address 0.004458 0.014219 0.018677 ( 0.284361)
fast_fallback: false 0.005869 0.021511 0.027380 ( 0.321891)
````
And this is the measurement result when executed in a single stack environment.
```
user system total real
Domain name 0.007062 0.019276 0.026338 ( 1.905775)
IP Address 0.004527 0.012176 0.016703 ( 3.051192)
fast_fallback: false 0.005546 0.019426 0.024972 ( 1.775798)
```
The following is the result of the run on Ruby 3.3.0.
(on Dual stack environment)
```
user system total real
Ruby 3.3.0 0.007271 0.027410 0.034681 ( 0.472510)
```
(on Single stack environment)
```
user system total real
Ruby 3.3.0 0.005353 0.018898 0.024251 ( 1.774535)
```
* Do not cache `Socket.ip_address_list`
As mentioned in the comment at https://github.com/ruby/ruby/pull/9374#discussion_r1482269186, caching Socket.ip_address_list does not follow changes in network configuration.
But if we stop caching, it becomes necessary to check every time `Socket.tcp` is called whether it's a single stack or not, which could further degrade performance in the case of a dual stack.
From this, I've changed the approach so that when a domain name is passed, it doesn't check whether it's a single stack or not and resolves names in parallel each time.
The performance measurement results are as follows.
require 'socket'
require 'benchmark'
HOSTNAME = "www.ruby-lang.org"
PORT = 80
ai = Addrinfo.tcp(HOSTNAME, PORT)
Benchmark.bmbm do |x|
x.report("Domain name") do
30.times { Socket.tcp(HOSTNAME, PORT).close }
end
x.report("IP Address") do
30.times { Socket.tcp(ai.ip_address, PORT).close }
end
x.report("fast_fallback: false") do
30.times { Socket.tcp(HOSTNAME, PORT, fast_fallback: false).close }
end
end
user system total real
Domain name 0.004085 0.011873 0.015958 ( 0.330097)
IP Address 0.000993 0.004400 0.005393 ( 0.257286)
fast_fallback: false 0.001348 0.008266 0.009614 ( 0.298626)
* Wait forever if fallback addresses are unresolved, unless resolv_timeout
Changed from waiting only 3 seconds for name resolution when there is no fallback address available, to waiting as long as there is no resolv_timeout.
This is in accordance with the current `Socket.tcp` specification.
* Use exact pattern to match IPv6 address format for specify address family
|
|
This reverts commit 673ed41c81cf5a6951bcb2c3dec82d7bd6ea7440.
|
|
This reverts commit abf192eb16ea845ac11743a32bd2f3e2d234488b.
|
|
|
|
|
|
When making an outgoing TCP or UDP connection, set AI_ADDRCONFIG in the
hints we send to getaddrinfo(3) (if supported). This will prompt the
resolver to _NOT_ issue A or AAAA queries if the system does not
actually have an IPv4 or IPv6 address (respectively).
This makes outgoing connections marginally more efficient on
non-dual-stack systems, since we don't have to try connecting to an
address which can't possibly work.
More importantly, however, this works around a race condition present
in some older versions of glibc on aarch64 where it could accidently
send the two outgoing DNS queries with the same DNS txnid, and get
confused when receiving the responses. This manifests as outgoing
connections sometimes taking 5 seconds (the DNS timeout before retry) to
be made.
Fixes #19144
|
|
https://github.com/ruby/ruby/pull/9088#discussion_r1411490445
|
|
The test for Socket::ResolutionError#error_code fails in the FreeBSD environment with this test condition. Because Socket::ResolutionError#error_code returns Socket::EAI_FAIL instead of Socket::EAI_FAMILY.
https://rubyci.s3.amazonaws.com/freebsd12/ruby-master/log/20231130T103002Z.fail.html.gz
This PR avoids the test failure by relaxing the condition.
Also changed the domain for testing to `example.com`.
|
|
https://rubyci.s3.amazonaws.com/freebsd12/ruby-master/log/20231130T103002Z.fail.html.gz
|
|
|
|
rsock_raise_socket_error is called only when getaddrinfo and getaddrname fail
|
|
Since Linux 4.5, sendmsg(2) fails with ETOOMANYREFS if the number of
"in-flight" IOs, which has been sent by sendmsg(2) but has not yet
accepted by recvmsg(2), exceeds the RLIMIT_NOFILE resource limit.
https://rubyci.s3.amazonaws.com/arm64-neoverse-n1/ruby-master/log/20231025T090004Z.fail.html.gz
```
1) Error:
TestSocket_UNIXSocket#test_fd_passing_race_condition:
Errno::ETOOMANYREFS: Too many references: cannot splice - sendmsg(2)
```
This change reduces the number of times of IO passing under 1024,
which is a default limit in many environments.
|
|
|
|
|
|
|
|
|
|
|
|
OpenBSD and Solaris behave differently here.
Linux does deliver the empty packet, which is questionable
as it's undistinguishable from a closed connection.
It seems that OpenBSD and Solaris simply drop it.
We could test the platform before doing the assertion, but
it would likely be fragile, and the entire web recommend
to not ever send an empty packet, so the value of this
assertion is low.
Notes:
Merged: https://github.com/ruby/ruby/pull/8338
|
|
[Bug #19012]
man recvmsg(2) states:
> Return Value
> These calls return the number of bytes received, or -1 if an error occurred.
> The return value will be 0 when the peer has performed an orderly shutdown.
Not too sure how one is supposed to make the difference between a packet of
size 0 and a closed connection.
Notes:
Merged: https://github.com/ruby/ruby/pull/6407
|
|
It seems like it never succeeds on this CI.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/7462
|
|
|
|
|
|
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <[email protected]>
Notes:
Merged-By: ioquatix <[email protected]>
|
|
Notes:
Merged-By: ioquatix <[email protected]>
|
|
we should handle ensure block when omit this test
|
|
http://rubyci.s3.amazonaws.com/freebsd13/ruby-master/log/20220906T043002Z.fail.html.gz
http://rubyci.s3.amazonaws.com/freebsd13/ruby-master/log/20220905T103002Z.fail.html.gz
|
|
|
|
Notes:
Merged-By: k0kubun <[email protected]>
|
|
This reverts commit 27fb9d272daaae89089dfb61849ebe8e7aa6c833.
The test failure on Solaris 10 is due to incomplete IPv6 configuration
on the CI server, that have already been fixed.
Reference for the fix: https://centrify.force.com/support/Article/KB-1179-X11-Forwarding-fails-with-Centrify-OpenSSH-5-0-Solaris/
|
|
test-unit migration
|
|
The test fails on Solaris 10. Maybe due to the IPv6 configuration on the
server, but I have no idea at all. I've asked @ngoto to investigate the
issue, so will tentatively skip the tests on Solaris
http://rubyci.s3.amazonaws.com/solaris10-gcc/ruby-master/log/20210729T040002Z.fail.html.gz
|
|
|
