From: janosch-x via ruby-core Date: 2023-02-13T10:55:42+00:00 Subject: [ruby-core:112396] [Ruby master Bug#19417] Regexp \p{Word} and [[:word:]] do not match Unicode Other_Number character Issue #19417 has been updated by janosch-x (Janosch M��ller). regarding the documentation, `letter` in the upstream doc is also incorrect, so the downstream doc actually has two errors. as implemented [here](https://github.com/ruby/ruby/blob/9821f6d0e5957a680bb4ce39708ebc86e23d85d0/tool/enc-unicode.rb#L130), `word` actually matches anything with the `alphabetic` property (effectively a superset of the `letter` category comprising about ~1600 chars more). demonstration: ```ruby %w[ word letter mark decimal_number connector_punctuation alpha ].select { |p| eval("/\\p{#{p}}/ =~ ?���") } # roman eight # => ["word", "alpha"] ``` a better wording might be: ``` A character with the _Alphabetic_ unicode property or one of the following Unicode general categories: Mark, Decimal\_Number, Connector\_Punctuation ``` regarding the behavior, i think it could be changed to match `number` instead of `decimal_number`. some scripts (e.g. Malayalam) have characters for numbers higher than 9, and these would disrupt matching at the moment (e.g. the Malayalam 9 is matched but the 10 is not). this change would also make `word` match fractions and superscripts as the one mentioned by OP ("��"). to me, this would seem like the less unexpected behavior. ---------------------------------------- Bug #19417: Regexp \p{Word} and [[:word:]] do not match Unicode Other_Number character https://bugs.ruby-lang.org/issues/19417#change-101838 * Author: ObjectBoxPC (Philip Chung) * Status: Open * Priority: Normal * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- According to the [documentation for Regexp](https://ruby-doc.org/3.2.0/Regexp.html), `\p{Word}` and `[[:word:]]` both match a character in one of the following Unicode general categories: Letter, Mark, Number, Connector_Punctuation. However, neither matches U+00B2, which is in the Other_Number category (which is a subcategory of Number). ``` ruby puts "Ruby version: %s" % RUBY_VERSION puts "\p{Word} matches? %s" % /\p{Word}/u.match?("\u00B2") puts "[[:word:]] matches? %s" % /[[:word:]]/u.match?("\u00B2") puts "Is a Number charater? %s" % /\p{Number}/u.match?("\u00B2") puts "Is an Other_Number character? %s" % /\p{Other_Number}/u.match?("\u00B2") ``` Expected output: ``` Ruby version: 3.2.0 p{Word} matches? true [[:word:]] matches? true Is a Number charater? true Is an Other_Number character? true ``` Actual output: ``` Ruby version: 3.2.0 p{Word} matches? false [[:word:]] matches? false Is a Number charater? true Is an Other_Number character? true ``` I notice that the [upstream Onigmo library doc](https://github.com/k-takata/Onigmo/blob/master/doc/RE) defines the `[[:word:]]` class as "Letter | Mark | Decimal_Number | Connector_Punctuation", meaning that it only matches certain number characters (which would exclude U+00B2). I am not sure how `\p{Word}` is defined though. But perhaps the documentation needs to be changed? -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/