Support test-case selection based on global data tag

In the same ways as for blacklisting. In the process, move reporting
of an unrecognised tag outside the global-data loop, as the tag might
be, or start with, a global data tag; and also report the global data
alongside the test-specific data. Indent the lines reporting tags, as
this makes the structure of the lists more evident. Converted the tag
argument to QLatin1String() to facilitate searching and matching.

Updated documentation, including expanding on the option of specifying
the data tags on the command-line. Drive-by: fixed a link in the
documentation, removed a snippet that was nowhere used.

[ChangeLog][QtTest] When specifying what tests to run on a QtTest
command-line, it is now possible to include a global data tag in the
same way as a function-specific data tag or in combination with it.
Thus if the test reports itself as tst_Class::function(global:local),
command-line option function:global:local will select that specific
test and function:global will run all data-rows for the function on
the given global data tag.

Fixes: QTBUG-102269
Change-Id: I7c86d6026933b05f7994bfc3663a2ea6a47f5f38
Reviewed-by: Qt CI Bot <[email protected]>
Reviewed-by: Volker Hilsheimer <[email protected]>

4 files changed, 83 insertions, 29 deletions

diff --git a/src/testlib/doc/snippets/code/doc_src_qtestlib.qdoc b/src/testlib/doc/snippets/code/doc_src_qtestlib.qdoc
index 1f372fadedf..68378ad403b 100644
--- a/src/testlib/doc/snippets/code/doc_src_qtestlib.qdoc
+++ b/src/testlib/doc/snippets/code/doc_src_qtestlib.qdoc
@@ -69,9 +69,16 @@ testname [options] [testfunctions[:testdata]]...
 
 
 //! [6]
-cetest [options] ...
+./testqlocale roundTripInt:zero
 //! [6]
 
+//! [7]
+./testqlocale roundTripInt:C
+//! [7]
+
+//! [8]
+./testqlocale roundTripInt:C:zero
+//! [8]
 
 //! [9]
 /myTestDirectory$ qmake -project "QT += testlib"
diff --git a/src/testlib/doc/src/qttest-best-practices.qdoc b/src/testlib/doc/src/qttest-best-practices.qdoc
index 7143e644fd1..788380652ac 100644
--- a/src/testlib/doc/src/qttest-best-practices.qdoc
+++ b/src/testlib/doc/src/qttest-best-practices.qdoc
@@ -182,6 +182,13 @@
     tested even when earlier ones fail. It also encourages systematic and
     uniform testing, because the same tests are applied to each data sample.
 
+    When a test is data-driven, you can specify its data-tag along with the
+    test-function name, as \c{function:tag}, on the command-line of the test to
+    run the test on just one specific test-case, rather than all test-cases of
+    the function. This can be used for either a global data tag or a local tag,
+    identifying a row from the function's own data; you can even combine them as
+    \c{function:global:local}.
+
     \section2 Use Coverage Tools
 
     Use a coverage tool such as \l {Froglogic Coco Code Coverage} or \l {gcov}
diff --git a/src/testlib/doc/src/qttestlib-manual.qdoc b/src/testlib/doc/src/qttestlib-manual.qdoc
index e940f3cae37..79974371030 100644
--- a/src/testlib/doc/src/qttestlib-manual.qdoc
+++ b/src/testlib/doc/src/qttestlib-manual.qdoc
@@ -228,9 +228,9 @@
     \snippet code/doc_src_qtestlib.qdoc 4
 
     Runs the \c toUpper test function with all available test data,
-    and the \c toInt test function with the test data called \c
+    and the \c toInt test function with the test data row called \c
     zero (if the specified test data doesn't exist, the associated test
-    will fail).
+    will fail and the available data tags are reported).
 
     \snippet code/doc_src_qtestlib.qdoc 5
 
@@ -485,7 +485,7 @@
 
     You can define \c{initTestCase_data()} to set up a global test data table.
     Each test is run once for each row in the global test data table. When the
-    test function itself \l{Chapter 2: Data-driven Testing}{is data-driven},
+    test function itself \l{Chapter 2: Data Driven Testing}{is data-driven},
     it is run for each local data row, for each global data row. So, if there
     are \c g rows in the global data table and \c d rows in the test's own
     data-table, the number of runs of this test is \c g times \c d.
@@ -508,6 +508,32 @@
     each locale provided by \c {initTestCase_data()}:
 
     \snippet code/src_qtestlib_qtestcase_snippet.cpp 31
+
+    On the command-line of a test you can pass the name of a function (with no
+    test-class-name prefix) to run only that one function's tests. If the test
+    class has global data, or the function is data-driven, you can append a data
+    tag, after a colon, to run only that tag's data-set for the function. To
+    specify both a global tag and a tag specific to the test function, combine
+    them with a colon between, putting the global data tag first. For example
+
+    \snippet code/doc_src_qtestlib.qdoc 6
+
+    will run the \c zero test-case of the \c roundTripInt() test above (assuming
+    its \c TestQLocale class has been compiled to an executable \c testqlocale)
+    in each of the locales specified by \c initTestCase_data(), while
+
+    \snippet code/doc_src_qtestlib.qdoc 7
+
+    will run all three test-cases of \c roundTripInt() only in the C locale and
+
+    \snippet code/doc_src_qtestlib.qdoc 8
+
+    will only run the \c zero test-case in the C locale.
+
+    Providing such fine-grained control over which tests are to be run can make
+    it considerably easier to debug a problem, as you only need to step through
+    the one test-case that has been seen to fail.
+
 */
 
 /*!
@@ -591,8 +617,8 @@
 
     Now that we finished writing our test, we want to execute
     it. Assuming that our test was saved as \c testqstring.cpp in an
-    empty directory, we build the test using qmake to create a project
-    and generate a makefile.
+    empty directory, we build the test using \c qmake to create a
+    project and generate a makefile.
 
     \snippet code/doc_src_qtestlib.qdoc 9
 
diff --git a/src/testlib/qtestcase.cpp b/src/testlib/qtestcase.cpp
index c1b9e31d803..caaf22508db 100644
--- a/src/testlib/qtestcase.cpp
+++ b/src/testlib/qtestcase.cpp
@@ -327,7 +327,7 @@ public:
     static QMetaMethod findMethod(const QObject *obj, const char *signature);
 
 private:
-    bool invokeTest(int index, const char *data, WatchDog *watchDog) const;
+    bool invokeTest(int index, QLatin1String tag, WatchDog *watchDog) const;
     void invokeTestOnData(int index) const;
 
     QMetaMethod m_initTestCaseMethod; // might not exist, check isValid().
@@ -1148,7 +1148,7 @@ public:
     If the function was successfully called, true is returned, otherwise
     false.
 */
-bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) const
+bool TestMethods::invokeTest(int index, QLatin1String tag, WatchDog *watchDog) const
 {
     QBenchmarkTestMethodData benchmarkData;
     QBenchmarkTestMethodData::current = &benchmarkData;
@@ -1168,6 +1168,18 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
         return globalDataCount ? gTable->testData(index)->dataTag() : nullptr;
     };
 
+    const auto dataTagMatches = [](QLatin1String tag, QLatin1String local, QLatin1String global) {
+        if (tag.isEmpty()) // No tag specified => run all data sets for this function
+            return true;
+        if (tag == local || tag == global) // Equal to either => run it
+            return true;
+        // Also allow global:local as a match:
+        return tag.startsWith(global) && tag.endsWith(local) &&
+               tag.size() == global.size() + 1 + local.size() &&
+               tag[global.size()] == ':';
+    };
+    bool foundFunction = false;
+
     /* For each entry in the global data table, do: */
     do {
         if (!gTable->isEmpty())
@@ -1180,7 +1192,6 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
                 break;
         }
 
-        bool foundFunction = false;
         int curDataIndex = 0;
         const int dataCount = table.dataCount();
         const auto dataTag = [&table, dataCount](int index) {
@@ -1188,22 +1199,18 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
         };
 
         // Data tag requested but none available?
-        if (data && !dataCount) {
-            // Let empty data tag through.
-            if (!*data)
-                data = nullptr;
-            else {
-                fprintf(stderr, "Unknown testdata for function %s(): '%s'\n", name.constData(), data);
-                fprintf(stderr, "Function has no testdata.\n");
-                return false;
-            }
+        if (!tag.isEmpty() && !dataCount && !globalDataCount) {
+            fprintf(stderr, "Unknown test data tag for function %s(): '%s'\n"
+                    "Function has no testdata.\n", name.constData(), tag.data());
+            return false;
         }
 
         /* For each entry in this test's data table, do: */
         do {
             QTestResult::setSkipCurrentTest(false);
             QTestResult::setBlacklistCurrentTest(false);
-            if (!data || !qstrcmp(data, table.testData(curDataIndex)->dataTag())) {
+            if (dataTagMatches(tag, QLatin1String(dataTag(curDataIndex)),
+                               QLatin1String(globalDataTag(curGlobalDataIndex)))) {
                 foundFunction = true;
 
                 QTestPrivate::checkBlackLists(name.constData(), dataTag(curDataIndex),
@@ -1219,24 +1226,31 @@ bool TestMethods::invokeTest(int index, const char *data, WatchDog *watchDog) co
                 if (watchDog)
                     watchDog->testFinished();
 
-                if (data)
+                if (!tag.isEmpty() && !globalDataCount)
                     break;
             }
             ++curDataIndex;
         } while (curDataIndex < dataCount);
 
-        if (data && !foundFunction) {
-            fprintf(stderr, "Unknown testdata for function %s: '%s()'\n", name.constData(), data);
-            fprintf(stderr, "Available testdata:\n");
-            for (int i = 0; i < table.dataCount(); ++i)
-                fprintf(stderr, "%s\n", table.testData(i)->dataTag());
-            return false;
-        }
-
         QTestResult::setCurrentGlobalTestData(nullptr);
         ++curGlobalDataIndex;
     } while (curGlobalDataIndex < globalDataCount);
 
+    if (!tag.isEmpty() && !foundFunction) {
+        fprintf(stderr, "Unknown testdata for function %s(): '%s'\n", name.constData(), tag.data());
+        if (table.dataCount()) {
+            fputs("Available test-specific data tags:\n", stderr);
+            for (int i = 0; i < table.dataCount(); ++i)
+                fprintf(stderr, "\t%s\n", table.testData(i)->dataTag());
+        }
+        if (globalDataCount) {
+            fputs("Available global data tags:\n", stderr);
+            for (int i = 0; i < globalDataCount; ++i)
+                fprintf(stderr, "\t%s\n", gTable->testData(i)->dataTag());
+        }
+        return false;
+    }
+
     QTestResult::finishedCurrentTestFunction();
     QTestResult::setSkipCurrentTest(false);
     QTestResult::setBlacklistCurrentTest(false);
@@ -1557,7 +1571,7 @@ void TestMethods::invokeTests(QObject *testObject) const
                 const char *data = nullptr;
                 if (i < QTest::testTags.size() && !QTest::testTags.at(i).isEmpty())
                     data = qstrdup(QTest::testTags.at(i).toLatin1().constData());
-                const bool ok = invokeTest(i, data, watchDog.data());
+                const bool ok = invokeTest(i, QLatin1String(data), watchDog.data());
                 delete [] data;
                 if (!ok)
                     break;