Fix regression when selecting emojis using VS-16

This amends 58907dfa812de8ce6c63295589f644bdf4257d15,
which tried to make sure we select a color variant of a
character when combining it with the VS-16 symbol, the
emoji selector (and vice versa with VS-15).

This is something which is not well-handled in Qt, and
the original work-around assumed that any font supporting the
conversion of Unicode characters to color glyphs would include
the VS-16 character in its map. This was true for the symptomatic
font on macOS, so it solved the problem there.

However, if you manually select a color font and then request a color
glyph, the font system should just pass it on and accept this. No
need for any magic adaptation. Same if you select a non-color font
and use VS-15.

To avoid regression, we detect this case and disable the behavior
introduced in 58907dfa812de8ce6c63295589f644bdf4257d15 when the existing
font selection is already meeting the criteria.

This is admittedly very hacky, and the plan is to do a larger refactor
where sequences of emojis are separated out ahead of time so that the
font selection can pick from a curated emoji font list for these.

But to fix the immediate regression, this will basically just return
to previous behavior for some cases which are currently prone to break,
so despite the added layer of conditions, it does improve the situation
temporarily.

Task-number: QTBUG-111801
Fixes: QTBUG-118642
Change-Id: I1a3ca711426a042b22003b887883646132335a9f
Reviewed-by: Tor Arne Vestbø <[email protected]>
(cherry picked from commit 35266dc3080dc13641616ed20e257d7b4c57778c)
Reviewed-by: Qt Cherry-pick Bot <[email protected]>
(cherry picked from commit e56db0394a8e1a658a48e4f4f2fff6b35d95aa7d)
(cherry picked from commit 7a5aee806f0698d8082c25bc6e6a9fcd9b5cbe20)

2 files changed, 27 insertions, 10 deletions

diff --git a/src/gui/text/qfontengine.cpp b/src/gui/text/qfontengine.cpp
index ae6894e55fc..85317d890cc 100644
--- a/src/gui/text/qfontengine.cpp
+++ b/src/gui/text/qfontengine.cpp
@@ -1905,17 +1905,32 @@ bool QFontEngineMulti::stringToCMap(const QChar *str, int len,
                 int precedingCharacterFontEngine = glyphs->glyphs[glyph_pos - 1] >> 24;
 
                 if (selectorFontEngine != precedingCharacterFontEngine) {
-                    QFontEngine *engine = m_engines.at(selectorFontEngine);
-                    glyph_t glyph = engine->glyphIndex(previousUcs4);
-                    if (glyph != 0) {
-                        glyphs->glyphs[glyph_pos - 1] = glyph;
-                        if (!(flags & GlyphIndicesOnly)) {
-                            QGlyphLayout g = glyphs->mid(glyph_pos - 1, 1);
-                            engine->recalcAdvances(&g, flags);
+                    // Emoji variant selectors are specially handled and should affect font
+                    // selection. If VS-16 is used, then this means we want to select a color
+                    // font. If the selected font is already a color font, we do not need search
+                    // again. If the VS-15 is used, then this means we want to select a non-color
+                    // font. If the selected font is not a color font, we don't do anything.
+                    const QFontEngine *selectedEngine = m_engines.at(precedingCharacterFontEngine);
+                    const bool colorFont = selectedEngine->isColorFont();
+                    const char32_t vs15 = 0xFE0E;
+                    const char32_t vs16 = 0xFE0F;
+                    bool adaptVariantSelector = ucs4 < vs15
+                                                || (ucs4 == vs15 && colorFont)
+                                                || (ucs4 == vs16 && !colorFont);
+
+                    if (adaptVariantSelector) {
+                        QFontEngine *engine = m_engines.at(selectorFontEngine);
+                        glyph_t glyph = engine->glyphIndex(previousUcs4);
+                        if (glyph != 0) {
+                            glyphs->glyphs[glyph_pos - 1] = glyph;
+                            if (!(flags & GlyphIndicesOnly)) {
+                                QGlyphLayout g = glyphs->mid(glyph_pos - 1, 1);
+                                engine->recalcAdvances(&g, flags);
+                            }
+
+                            // set the high byte to indicate which engine the glyph came from
+                            glyphs->glyphs[glyph_pos - 1] |= (selectorFontEngine << 24);
                         }
-
-                        // set the high byte to indicate which engine the glyph came from
-                        glyphs->glyphs[glyph_pos - 1] |= (selectorFontEngine << 24);
                     }
                 }
             }
diff --git a/src/gui/text/qfontengine_p.h b/src/gui/text/qfontengine_p.h
index b6e786512d0..9d597c0af3f 100644
--- a/src/gui/text/qfontengine_p.h
+++ b/src/gui/text/qfontengine_p.h
@@ -152,6 +152,8 @@ public:
         return subPixelPositionFor(QFixedPoint(x, 0)).x;
     }
 
+    bool isColorFont() const { return glyphFormat == Format_ARGB; }
+
     virtual QFixed emSquareSize() const { return ascent(); }
 
     /* returns 0 as glyph index for non existent glyphs */