[ruby-core:120435] [Ruby master Bug#20990] Ripper.tokenize splits `"\C-\あ"` into tokens with invalid byte sequence
From:
"tompng (tomoya ishida) via ruby-core" <ruby-core@...>
Date:
2024-12-28 04:06:51 UTC
List:
ruby-core #120435
SXNzdWUgIzIwOTkwIGhhcyBiZWVuIHJlcG9ydGVkIGJ5IHRvbXBuZyAodG9tb3lhIGlzaGlkYSku DQoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCkJ1ZyAjMjA5OTA6 IFJpcHBlci50b2tlbml6ZSBzcGxpdHMgYCJcQy1c44GCImAgaW50byB0b2tlbnMgd2l0aCBpbnZh bGlkIGJ5dGUgc2VxdWVuY2UNCmh0dHBzOi8vYnVncy5ydWJ5LWxhbmcub3JnL2lzc3Vlcy8yMDk5 MA0KDQoqIEF1dGhvcjogdG9tcG5nICh0b21veWEgaXNoaWRhKQ0KKiBTdGF0dXM6IE9wZW4NCiog cnVieSAtdjogcnVieSAzLjQuMSAoMjAyNC0xMi0yNSByZXZpc2lvbiA0OGQ0ZWZjYjg1KSArWUpJ VCArTU4gW2FybTY0LWRhcndpbjIyXQ0KKiBCYWNrcG9ydDogMy4xOiBVTktOT1dOLCAzLjI6IFVO S05PV04sIDMuMzogVU5LTk9XTiwgMy40OiBVTktOT1dODQotLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tDQpJUkIgY3Jhc2hlcyB3aGVuIGEgY29kZSBpcyB0b2tlbml6ZWQg dG8gYW4gaW52YWxpZCBieXRlIHNlcXVlbmNlLg0KDQp+fn5ydWJ5DQpSaXBwZXIudG9rZW5pemUg JyJcQy1c44GCIicNCiM9PiBbIlwiIiwgIlxcQy1cXFx4RTNceDgxIiwgIlx4ODIiLCAiXCIiXQ0K fn5+DQoNCg0KSSB0aGluayB0aGUgZXJyb3IgZXZhbHVhdGluZyBgIlxDLVzjgYIiYCBzaG91bGQg YmUgYEludmFsaWQgZXNjYXBlIGNoYXJhY3RlciBzeW50YXhgIGp1c3QgbGlrZSBgIlxDLeOBgiJg DQp+fn4NCiQgLi9ydWJ5IC0tcGFyc2VyPXBhcnNlLnkgLWUgJyJcQy3jgYIiJw0KLWU6MTogSW52 YWxpZCBlc2NhcGUgY2hhcmFjdGVyIHN5bnRheA0KIlxDLeOBgiINCg0KJCAuL3J1YnkgLS1wYXJz ZXI9cGFyc2UueSAtZSAnIlxDLVzjgYIiJw0KLWU6MTogaW52YWxpZCBtdWx0aWJ5dGUgY2hhciAo VVRGLTgpDQotZToxOiBpbnZhbGlkIG11bHRpYnl0ZSBjaGFyIChVVEYtOCkNCi4vcnVieTogY29t cGlsZSBlcnJvciAoU3ludGF4RXJyb3IpDQp+fn4NCg0KDQoNCg0KLS0gDQpodHRwczovL2J1Z3Mu cnVieS1sYW5nLm9yZy8NCiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCiBydWJ5LWNvcmUgbWFpbGluZyBsaXN0IC0tIHJ1YnktY29yZUBtbC5ydWJ5LWxhbmcu b3JnCiBUbyB1bnN1YnNjcmliZSBzZW5kIGFuIGVtYWlsIHRvIHJ1YnktY29yZS1sZWF2ZUBtbC5y dWJ5LWxhbmcub3JnCiBydWJ5LWNvcmUgaW5mbyAtLSBodHRwczovL21sLnJ1YnktbGFuZy5vcmcv bWFpbG1hbjMvbGlzdHMvcnVieS1jb3JlLm1sLnJ1YnktbGFuZy5vcmcv