Hi Yuancheng,
Awesome! I noticed the impressive number of opened issues, and given your background I guessed you
were working on some new fuzzer.
I’ve been thinking that the fuzzing corpus of this new fuzzer (and of the existing oss-fuzz fuzzer
SAPI, currently being ran by Google) may be improved by also adding all of the code of the community
libraries (i.e. those defined by the nightly GitHub workflow at https://github.com/php/php-src/blob/master/.github/workflows/nightly.yml#L485).
I understand this might cause issues due to the volume of additional code being permutated by the
fuzzer, but even running the community tests themselves without fuzzing already uncovers multiple
segfaults (some of which are still being failing on master as of today).
Some time ago, I submitted https://github.com/php/php-src/pull/12406,
which does multiple things to improve the coverage of the JIT compiler:
- Add a few more popular CPU-intensive community libs like phpseclib, psalm, phpstan, etc: all prime
candidates for addition to the nightly tests, as they're all extremely CPU-intensive libraries
which benefit a lot from JIT, but suffer from its bugs (i.e. the psalm/phpseclib phpunit testsuites
currently fail with segfaults with tracing JIT, phpseclib even explicitly disables JIT on windows to
avoid a yet unsolved bug, etc...).
- Parallelise community tests using a custom new runner, addressing concerns from Ilya which was
worried about CI run times caused by the addition of new community tests
- Add --repeat 2 to all tests (including community tests), which manages to catch some nasty JIT
bugs by re-invoking the same script twice without actually recompiling the PHP code and zend byte
code twice, managing to find issues caused by side effects of the first compilation.
- Improve JIT flags of community and phpt tests to detect more JIT bugs, mainly copying them from https://github.com/danog/jit_bugs/blob/master/php.ini
I don’t currently have the free time to clean up the pull request and fix the numerous JIT bugs it
currently detects (at least without a support contract), but all of these approaches may be reused
if you or anyone else decides to upstream FlowFusion (though I would love at least a @danog mention
in the pull request :)
Regards,
Daniil Gentili
—
Daniil Gentili - Senior software engineer
Portfolio: https://daniil.it
<https://daniil.it/>Telegram: https://t.me/danogentili
> On 15 Nov 2024, at 14:20, Yuancheng Jiang <[email protected]> wrote:
>
> Hi all,
>
> I have been submitting hundreds of bugs (see https://github.com/php/php-src/issues/created_by/YuanchengJiang)
> during the past months and I first thank all the developers who take time to fix these issues to
> make PHP better.
>
> I am thrilled to introduce one fully automated fuzz testing tool, FlowFusion, for discovering
> various bugs of the PHP interpreter.
>
> The core idea behind FlowFusion is to leverage dataflow as an effective representation of test
> cases (.phpt files) maintained by PHP developers, merging two (or more) test cases to produce fused
> test cases with more complex code semantics. We connect two (or more) test cases via interleaving
> their dataflows, i.e., bringing the code context from one test case to another. This enables
> interactions among existing test cases, which are mostly the unit tests verifying one single
> functionality, making fused test cases interesting with merging code semantics.
>
> FlowFusion additionally fuzzes all defined functions and class methods using the code contexts
> of fused test cases. Available functions, classes, and methods are pre-collected and stored in
> sqlite3 with necessary information like the number of parameters. FlowFusion will be automatically
> upgrading if phpt files keep updating. Any new single test can bring thousands of new fused tests.
>
> The search space of FlowFusion is huge, which means it can cover various corner cases. Reasons
> for the huge search space are three-fold: (i) two random combinations of around 20,000 test cases
> can generate 400,000,000 test cases, and we can combine even more; (ii) the interleaving has
> randomness, given two test cases, there could be multiple ways to connect them; and (iii) FlowFusion
> also mutates the test case, fuzzes the runtime environment/configuration like JIT.
>
> I can open-source the tool under my personal repository. I wonder by any chance if I can
> contribute it as the official PHP tool under https://github.com/php, and I would be happy to maintain it for a
> long time.
>
> Best,
> Yuancheng