CROWDSOURCING VERSUS

MOBILE NETWORK TESTING

The need for an objective evaluation of mobile network quality and performance drives two fundamentally
different approaches, crowdsourcing and mobile network testing. The crowdsourcing market is highly satu-
rated. More than 20 companies have a significant focus on crowdsourcing mobile network measurements.
To get measurement results, they use different concepts that serve different purposes. This application card
describes the structure of the crowdsourcing space and analyzes the pros and cons of both approaches.

Your task
Mobile network operators and regulators have a growing ► Marketing/customer experience management
interest in getting deeper insights into network quality and (CEM) – assesses end-user experience of network
competition in specific markets. This interest is driven by quality
three key requirements: Crowdsourcing and mobile network testing (MNT) are two
► Benchmarking – provides a view of the relative fundamentally different approaches to addressing these
performance of different mobile networks requirements. Mobile network operators and regulators
► Operations optimization – improves operational need to evaluate these two approaches and map them
efficiency for the network operator (including customer against their needs. This application card investigates the
service, service assurance and network engineering) pros and cons of both concepts.

Application Card | Version 01.00

Crowdsourcing solution
Crowdsourcing solutions can be grouped into three ► Operator app – embeds a measurement SDK in an
categories: operator branded app
► Syndication – allows unrelated apps to integrate a ► Own app – places a branded app in app stores
measurement SDK for monetary compensation (similar specifically targeting users interested in network
to in-app ad placements). Often end-users do not know performance
that data is collected in the background and sent to
cloud storage for further analysis, which can be a legal
gray area.

Crowdsourcing market segmentation, based on company positioning

Crowdsourcing market segmentation, based on company positioning
Benchmarking

Own app

Syndication

Operations optimization

Operator app

Figure 1: Segmentation of crowdsourcing

solutions by requirement and application
Marketing/CEM category ­(examples, non-exhaustive).

Crowdsourcing analysis

Pros Cons
► Crowdsourcing data is collected continuously and ► Conditions of crowdsourcing data collection are unpredictable and mostly unknown:
autonomously without a dedicated measurement — T emperature: smartphone in the user’s pocket is heated to body temperature; heating due to applications
campaign. running in parallel
► Massive amount of data provides very high resolution
— Location: smartphone in the basement or 1st or 10th or 20th floor of a house
—  Device dependency: older phones with limited UE capabilities in use for long time – network performance
analysis. You get data from wherever people are. But measurement might be affected by limited UE capabilities
that does not ensure complete geographical coverage. —  Customer traffic plan: data volume might be expended and throughputs throttled by the network
► Mobile network coverage and quality statistics are ► Measurements are not reproducible (crowdsourcing analysis can be repeated or done on a newer data set, but it
available across a country or region. cannot be reproduced). If there is a dispute on a reported event, it cannot be reproduced for validation.
► Identification of sites, locations and operators with ► Crowdsourcing data cannot be used for network optimization. Limited or no information on root causes.
highest data traffic can offer insights for new site ► Crowdsourcing solutions mostly follow a proprietary approach with an undisclosed scoring method for the
planning or capacity expansion. results. The scores or results from different solutions are not comparable with each other.
► Provides an understanding of how the available ► Crowdsourcing solutions typically limit their active tests to the following quality KPIs: UL/DL throughput,
spectrum is used in different geographical areas. average latency, average jitter and packet loss.
► Massive amount of data can be filtered and used ► Smartphone operating systems are increasingly restrictive and limit the available parameters that
for drill-down analysis according to various criteria crowdsourcing apps (running in the background) may use.
(operators, location, time, etc.). ► Dedicated crowdsourcing apps typically show a negative bias (technically savvy subscribers in particular trigger
these tests when they experience a network issue).

2
Mobile network testing solution
Drive and walk tests are dedicated mobile network test Applications are tested in line with the following three
concepts. Dedicated smartphones (commercial UEs) and categories:
network scanners are used in a controlled test setup (tem- ► Accessibility ­– service availability, e.g. dropped calls
perature and RF shielding as far as possible) and perform ► Waiting time – call setup time
a set of test cases reflecting end-user behavior. Voice ser- ► Media quality – picture quality or speech quality
vices (telephony) and data services (data transfer, video
streaming, social media and web browsing) are the most The results are mapped to a point scale and an aggregated
commonly used services and account for the vast majority score is generated reflecting end-user perceived quality
of traffic volume in mobile networks. (a streamed video that cannot be started is more severe
than slightly reduced video quality and gets a higher
ETSI TR 103 559 describes best practices for robust net- weighting).
work QoS benchmark testing and scoring. The t­ echnical
report provides a recommendation for how to test the For further details and explanations, go to:
quality of mobile networks based on end-user perception. www.rohde-schwarz.com/nps

The Rohde & Schwarz network performance score imple-

ments the ETSI method.

Highway test campaign

Telephony

KPIs, e.g. Data transfer

score for web browsing
Video streaming Highway
Max.
Success rate Data score
0 Social media
Average Max.
Web browsing
duration 0
Web browsing Overall network
Max. score score
Duration > 15 s
0

Figure 2: Rohde & Schwarz network performance score (NPS) covers the full range of end-user applications.
City
score

Mobile network testing analysis

Pros Cons
► Data is collected under defined conditions and in a controlled test setup (temperature, RF shielding). You always ► Dedicated measurement campaigns have to be
know under which conditions the data is collected (outdoor, indoor, measurement device position, temperature, planned and executed which requires effort. They
etc.). are time-consuming and expensive compared
► Measurement campaigns are planned with latest smartphones and UE capabilities to ensure that the performance to crowdsourcing (although the crowdsourcing
measurement is not impacted by UE deficiencies. All applications that subscribers have running in the background reports are also not available for free).
contribute to the overall assessment of network quality. ► Network performance results are only available
► Using defined and harmonized test methods makes mobile network testing comparable and reproducible for areas where the data has been collected.
► Measurement campaigns are clearly defined (routes, teams, times, measurement setups, etc.) and can be easily A coverage map for a whole country cannot
repeated after 3, 6 or 12 months to measure performance improvement over time. efficiently be created based on mobile network
► The test scope is much wider compared to crowdsourcing. Professional mobile network testing solutions can also testing.
calculate advanced metrics such as mean opinion score (MOS) for voice and video content.
► The network performance score using the ETSI method provides insights for network optimization due to the
standardized measurement and scoring approach. All details are available for further analysis. When problems
are reported, context is available (RF conditions, detailed L3 messages from the network and applications) for
troubleshooting purposes.
► Its comparability due to the harmonized test method means that mobile network testing can be used to compare
Open RAN (O-RAN) with a single vendor RAN architecture.
► Agreements with mobile device and chipset manufacturers are in place to overcome the limitation of restrictive
smartphones operating systems.

Rohde & Schwarz Crowdsourcing versus mobile network testing 3

Recommendations ► Network optimization: If network optimization is to
Analysis and comparison of the pros and cons of both be based on data, the collected data must be reliable,
approaches shows that crowdsourcing and mobile net- collected under known, defined conditions and be
work testing are more complementary than competitive reproducible. The data also needs to be available with
and that one can never replace the other. context (RF conditions, detailed L3 messages from the
network and applications) in order to match identified
Depending on the target or the requirement, one approach problems with RAN events so that root causes for these
has advantages over the other. Here are some examples: problems can be identified. Only mobile network testing
► Network quality benchmarking/ranking of operator fulfills these requirements, whereas crowdsourcing data
networks in a country: For a rough overview, does not provide this context.
crowdsourcing data might be sufficient. But if the aim is ► Network usage analysis: To identify locations or
to make a fair comparison and draw solid conclusions, a geographical areas with high data and spectrum use,
comparable test method like ETSI TR 103 559 for mobile crowdsourcing is a viable solution because mass data
network testing is highly recommended. can be analyzed and the reliability of the data is not
► Coverage map of a whole country: If a coverage critical for this purpose.
map for a whole country is the goal, crowdsourcing is ► Crowdsourcing can be a good complement to a
the best solution. Basic RF data such as reference signal traditional drive test: Crowdsourcing can be used to
received power (RSRP) is sufficient in most cases, detect areas with low coverage or network problems.
although discussions are ongoing in the European Drive test teams can then be dispatched to collect data.
Conference of Postal and Telecommunications Crowdsourcing can help justify national benchmarking
Administrations (CEPT) to redefine the term “coverage”, campaigns (e.g. NPS campaigns), such as campaign
in particular in light of 5G beamforming. Mobile network and route planning to determine whether a campaign
testing cannot cover a whole country with each and is really covering the relevant parts of the country.
every street in every city, town, village at a reasonable
cost. Additional information
For more information on the mobile network testing solu-
tions for benchmarking discussed in this application card,
please contact your Rohde & Schwarz sales representative
or visit our website:
www.rohde-schwarz.com/mnt/network-benchmarking

5216420492
5216.4204.92 01.00 PDP/PDW 1 en

Rohde & Schwarz GmbH & Co. KG

www.rohde-schwarz.com R&S® is a registered trademark of Rohde & Schwarz GmbH & Co. KG
Trade names are trademarks of the owners
Rohde & Schwarz training PD 5216.4204.92 | Version 01.00 | September 2022 (jr)
www.training.rohde-schwarz.com Crowdsourcing versus mobile network testing
Rohde & Schwarz customer support Data without tolerance limits is not binding | Subject to change
www.rohde-schwarz.com/support © 2022 Rohde & Schwarz GmbH & Co. KG | 81671 Munich, Germany