SAP Knowledge Base Article

2100040 - FAQ: SAP HANA CPU

Component: HAN-DB-PERF (SAP HANA > SAP HANA Database > SAP HANA Database Performance), Version:
256, Released On: 2024/06/02

Symptom
You want to understand the CPU consumption of SAP HANA.

Environment
SAP HANA

Cause
1. Which indications exist for SAP HANA CPU problems?
2. Where do I find more information related to SAP HANA CPU analysis?
3. Is a high CPU consumption generally an issue?
4. How can I collect information about the CPU consumption?
5. How can additional CPU resource tracking be activated?
6. Is it possible to capture SQL statements with a particularly high CPU consumption?
7. What is the difference between system, user and I/O wait CPU consumption?
8. How can the CPU consumption of SAP HANA hosts be optimized?
9. How can CPU intensive operations in SAP HANA be identified and optimized?
10. Is it possible to limit the amount of consumed CPU?
11. How can I determine physical CPU details on SAP HANA side?
12. How can I trigger automatic follow-up actions in case of a high CPU consumption?
13. Is the CPU consumption shown by SAP HANA always correct?
14. Can the available CPU resources be limited on OS level?
15. Is it recommended to use hyperthreading in SAP HANA environments?
16. Which thread states and call stacks indicate CPU consumption?
17. How can the CPU performance of the SAP HANA database server be tested and compared?
18. What are reasons for slowness in combination with a limited CPU consumption?
19. What can be done in case of high CPU consumption in SAP HANA Cloud environments?

Resolution

1. Which indications exist for SAP HANA CPU problems?

The administration overview screen or the load graph of SAP HANA Studio shows a high current or historic CPU consumption.
The following SAP HANA alerts indicate problems in the CPU area:

Aler Name SAP Not Description

t e

5 Host CPU Usa 1909670 Determines the percentage CPU idle time on the host and therefore whether or not CPU resources
ge are running low.

SQL: "HANA_Configuration_MiniChecks" (SAP Notes 1969700, 1999993) returns a potentially critical issue (C = 'X') for one
of the following individual checks:

Check ID Details

M0210 Minimum CPU rate (MHz)

M0211 Hosts with varying CPU rates

M0212 CPU discrepancy host info vs. host agent

 M0215 Hosts with outdated CPU type

M0220 Current CPU utilization (%)

M0221 Peak CPU utilization (%, last day)

M0222 Time since CPU utilization > 95 % (h)

M0226 Peak system CPU utilization (%, last day)

M0227 External CPU utilization (%, last hour)

M0228 Erroneous system CPU calculation

M0229 Discrepancy between NUMA nodes and sockets

M0232 Hyperthreading active in critical context

M0233 NUMA nodes with high CPU utilization

M0234 CPU steal share (%)

M0893 Users with high thread limit

SQL: "HANA_Threads_Callstacks_MiniChecks" (SAP Notes 1969700, 2313619) reports critical check IDs.
The HDBCPUUsage alert is raised in SAP HANA Cloud environments.
A high amount of database threads are active with status "Running" (see SAP Note 2000000).

2. Where do I find more information related to SAP HANA CPU analysis?

The SAP HANA Troubleshooting and Performance Analysis Guide contains - among others - information about analyzing and
optimizing the SAP HANA CPU consumption.

3. Is a high CPU consumption generally an issue?

A high CPU consumption can be normal and acceptable, because SAP HANA uses available CPUs efficiently by parallelizing
complex operations. Therefore, a high CPU consumption can be intended and a consequence of a massively parallelized
processing of complex operations. If on the other hand simple database requests or other processes consume a high amount of
CPU, this can result in a critical CPU bottleneck situation which can be avoided. This SAP Note provides information how to
analyze and optimize unnecessary CPU consumption.

4. How can I collect information about the CPU consumption?

The following options exist to check the current CPU utilization of the SAP HANA database server:
• SAP HANA Studio -> Administration -> Overview -> CPU Usage
• SAP HANA Studio -> Administration -> Performance -> Load -> [System] CPU
The following SQL statements are available via SAP Note 1969700 in order to understand the CPU consumption of the SAP
HANA hosts:

SQL statement name Description

SQL: "HANA_Resources Current CPU utilization per core, breakdown to NUMA node possible
_CPU_UtilizationPerCo
re"

SQL: "HANA_Resources Current CPU information based on Host Agent information

_CPUAndMemory_Curr
entDetails"

SQL: "HANA_Resources Among others this SQL statement is able to check for current and historic CPU consumption on a granular
_CPUAndMemory_Hist basis. It displays the CPU utilization (in %), the number of CPUs busy with user activities, the number of CP
ory" Us busy with system activities and the number of idle CPUs.

SQL: "HANA_Configura This command provides information about the used CPUs including frequency and number.
tion_InfrastructureRep
ort"
 SQL: "HANA_LoadHist Host wide CPU consumption (SAP Note 2222110)
ory_Hosts"

SQL: "HANA_LoadHist Indexserver CPU information (total, system) (SAP Note 2222110)
ory_Services"

Additionally you should use operating system tools (e.g. top) in order to understand which processes are responsible for the
highest CPU consumption.

5. How can additional CPU resource tracking be activated?

An exact measurement of CPU consumption of a SAP HANA request isn't feasible because a lot of often parallel and short-
running threads can contribute to the execution and activating a precise measurement would be a significant overhead.
As of SAP HANA 1.0 SPS 12 a CPU approximation based on the states of active threads is used. When an active thread doesn't
have to wait (e.g. for locks, I/O or network), it is supposed to consume CPU. See SAP Note 2114710 for more information
related to SAP HANA threads. As a consequence, you always have CPU figures available without having to activate specific
measurements. Indeed, there is no other way to activate CPU time measurement mode with SAP HANA 1.0 SPS 12 and higher.
The following monitor view columns contain CPU information:
• M_SERVICE_THREADS.CPU_TIME_SELF
• M_SERVICE_THREADS.CPU_TIME_CUMULATIVE
• M_EXPENSIVE_STATEMENTS.CPU_TIME

6. Is it possible to capture SQL statements with a particularly high CPU consumption?

You can capture SQL statements exceeding a defined CPU consumption via expensive statements trace (see SAP Note 2119087)
using the following parameters:

Parameter Defau Uni Details

lt t

false Main switch, 'true' activates the expensive statement tra

global.ini -> [expensive_statement] -> enable
ce

us Threshold for minimum CPU utilization in micro second

global.ini -> [expensive_statement] -> thresh
old_cpu_time s (Rev. >= 94)

7. What is the difference between system, user and I/O wait CPU consumption?
CPU utilization is typically separated in system CPU, user CPU and I/O wait CPU consumption:

CPU utilization type Description

User CPU consumption originating from coding outside of the operating system kernel, e.g.:

• Expensive operation in SAP HANA like scan of large table or counting a high amount of distinct values
• Expensive operation of non-SAP HANA product running on same host

System CPU consumption originating from the operating system kernel, e.g.:

• Operating system background activities like memory defragmentation

• Implicit OS calls of SAP HANA like active waits for locks or I/O requests

I/O Wait CPUs waiting for I/O requests to complete, can usually be considered as idle time

8. How can the CPU consumption of SAP HANA hosts be optimized?

The following approaches exist to optimize CPU consumption and reduce the risk of running into CPU bottlenecks:

Possible sym Optimiza Optimization Details

ptoms tion
High system C Optimize Make sure that the OS recommendations of SAP Note 2000003 ("How can the configuration and perfor
PU consumpti OS config mance of the SAP HANA hardware, firmware and operating system be checked?") are implemented.
on caused by O uration
A more detailed analysis of activities involving system CPU consumption can be performed via OS kerne
S processes
l stack backtrace as described in SAP Note 2166414. Additionally the Linux perf package can be used to t
race operating system activities, e.g. "perf top ..." for generating and displaying a performance counter p
rofile in real time.
High system C Optimize System CPU consumption related to SAP HANA traces is typically caused by the following scenarios:
PU consumpti OS calls tr
• Active (yield) waits
on caused by S iggered by
• Frequent executions of expensive operating system calls
AP HANA proc SAP HAN
• Implicit triggering of undesired low-level activities like TLB shootdowns
esses A
• External software using Linux kernel taints, e.g. Cortex XDR, Crowdstrike Falcon, Imperva, Micro
soft Dependency Agent or Trend Micro (tmhook)
• External software using Linux kprobe interface (see Kernel Probes for details), e.g. Dynatrace One
agent (oneagentnetwork) or the SentinelOne agent or the system-probe component of Datadog Ag
ent

See "How can CPU intensive operations in SAP HANA be identified and optimized?" for more details.

For kernel taints and kprobe instrumentation the call stacks are often network related (methods like __l
ibc_recv, __libc_send, __writev, __writev_nocancel) and the threads frequently show "no SQL (NoAct
ion)" (SAP Note 2114710).

In case the SAP HANA call stacks don't contain relevant information, a more detailed analysis of activiti
es involving system CPU consumption can be performed via OS kernel stack backtrace as described in S
AP Note 2166414. Additionally the Linux perf package can be used to trace operating system activities, e
.g. "perf top ..." for generating and displaying a performance counter profile in real time or "sar -P all 1".

Linux kernel taints

Linux kernel taints can be identified by sapsysinfo.sh available via SAP Note 618104. Kernel taints or oft
en no problem, but taints flagged with N (unsupported), E (unsigned) or P (proprietary) should be chec
ked more in detail. A closer look at the OS kernel stack backtrace is useful to identify suspicious taints. F
or example, the following call stack indicates a spin lock (_raw_spin_lock) being called by a module call
ed krg_622832_imSLE15sp3_KS_SMP64:

_raw_spin_lock+0x1b/0x20
pseudolock_trylock_read+0x32/0x1a0 [krg_622832_imSLE15sp3_KS_SMP64]
? memory_manager_alloc+0x70/0x4f0 [krg_622832_imSLE15sp3_KS_SMP64]
? readv_pre_code+0xec0/0xec0 [krg_622832_imSLE15sp3_KS_SMP64]
? writev_pre_code+0xd1/0xeb0 [krg_622832_imSLE15sp3_KS_SMP64]
? cshook_network_ops_inet6_sockraw_recvmsg+0x185a9/0x21b10 [falcon_lsm_serviceab
le]
? cshook_systemcalltable_pre_ia32_pread64+0x23eb7/0x2cf40 [falcon_lsm_serviceabl
e]
? indexes_bit_mask_is_index_set+0xb8/0x280 [krg_622832_imSLE15sp3_KS_SMP64]
? _generic_fd_write_postcode+0x2e0/0x2e0 [krg_622832_imSLE15sp3_KS_SMP64]
? _orig_write+0x210/0x210 [krg_622832_imSLE15sp3_KS_SMP64]
? readv_pre_code+0xec0/0xec0 [krg_622832_imSLE15sp3_KS_SMP64]
? syscall_wrappers_generic_flow_with_param+0x6ee/0x830 [krg_622832_imSLE15sp3_KS
_SMP64]
? syscall_wrappers_generic_flow_with_param+0x6ee/0x830 [krg_622832_imSLE15sp3_KS
_SMP64]
? SYS_writev_common_wrap+0x86/0x100 [krg_622832_imSLE15sp3_KS_SMP64]
? SYS_writev_common_wrap+0x86/0x100 [krg_622832_imSLE15sp3_KS_SMP64]
? unload_network_ops_symbols+0x52f3/0x70f0 [falcon_lsm_pinned_13707]
? do_syscall_64+0x5b/0x1e0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9

Spin locks in the Linux kernel consume system CPU. The sapsysinfo.sh output helps to map the module
to an external software of company Imperva. After having deactivated the related tool, the system CPU c
onsumption went down to reasonable levels. With this result the external vendor can be approached, as
king for an optimization.

As another example, the following call stack indicates spin locking in context of the Microsoft Dependen
cy Agent:

queued_spin_lock_slowpath+0x7/0xe
_raw_spin_lock_irqsave+0x36/0x40
bs_acquire_spinlock+0xe/0x20 [microsoft_dependency_agent]
hash_table_get+0x26/0x90 [microsoft_dependency_agent]
bs_is_flow_tracing_disabled+0xa/0x30 [microsoft_dependency_agent]
bs_handle_tcp_xfer+0xb5/0x190 [microsoft_dependency_agent]
? tcp_sendmsg_locked+0x314/0xb50
inet_sendmsg_rethook+0xa0/0xc0 [microsoft_dependency_agent]
This specific issue was resolved after an upgrade of the Microsoft Dependency Agent from version 9.3 to
version 9.10.

Yet another example call stack, this time for spin locks caused by the Trend Micro hook (tmhook):

_raw_spin_lock+0x1e/0x30
tmhook_invoke_posthooks+0x5a/0xb0 [tmhook]
tmhook_handler+0x1be/0x2a0 [tmhook]
? do_syscall_64+0x5b/0x1e0
? tmhook_invoke+0xa0/0xa0 [tmhook]
do_syscall_64+0x5b/0x1e0
entry_SYSCALL_64_after_hwframe+0x61/0xc6

Linux kprobe interface

Tools using the Linux kprobe interface don't show up in the OS kernel stack backtrace. Instead "kprobe"
or "kretprobe" is visible. For example, the following call stacks indicate spin locks related to kprobe:

_raw_spin_lock_irqsave+0x36/0x50
pre_handler_kretprobe+0xfc/0x1c0
? tcp_sendmsg+0x5/0x40
aggr_pre_handler+0x40/0x80
? tcp_sendmsg+0x1/0x40
kprobe_ftrace_handler+0xab/0x100
ftrace_ops_assist_func+0xb1/0x150
? tcp_sendmsg_locked+0xbd0/0xbd0
? tcp_sendmsg+0x1/0x40
tcp_sendmsg+0x5/0x40

_raw_spin_lock+0x20/0x30
__pcpu_freelist_pop+0x82/0xb0
alloc_htab_elem+0x1f9/0x250
htab_map_update_elem+0x2a2/0x350
? tcp_sendmsg+0x5/0x40
? tcp_sendmsg+0x5/0x40
? trace_call_bpf+0x6d/0x90
? kprobe_perf_func+0x17b/0x230
? tcp_sendmsg+0x5/0x40
? kprobe_dispatcher+0x2f/0x50
? aggr_pre_handler+0x40/0x80
? tcp_sendmsg+0x1/0x40
? kprobe_ftrace_handler+0xab/0x100
? ftrace_ops_assist_func+0xb1/0x150
? tcp_sendmsg_locked+0xbd0/0xbd0
? tcp_sendmsg+0x1/0x40
? tcp_sendmsg+0x5/0x40

_raw_spin_lock+0x20/0x30
recycle_rp_inst+0x46/0xa0
? copy_oldmem_page_encrypted+0x20/0x20
? sock_sendmsg+0x3a/0x50
trampoline_handler+0x1f9/0x230
kretprobe_trampoline+0x25/0x50
? tcp_sendmsg+0x31/0x40

Apart from kprobe and the spin lock these call stacks of this example also have in common that they are
related to method tcp_sendmsg.

Active kernel probes can be identified via:

cat /sys/kernel/debug/kprobes/list

Now you can check for tracing instrumentation in directory /sys/kernel/debug/tracing/events/kprobes.

Among others, the following files related to tcp_sendmsg were found:
 /sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/format
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/hist
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/trigger
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/filter
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/id
/sys/kernel/debug/tracing/events/kprobes/r_tcp_sendmsg_net_14442/enable

The trailing number 14442 is the Linux process ID of the tool that uses this tracing. In this specific case t
his process ID pointed to /opt/datadog-agent/... (identified via "ps -ef | grep 14442"). After having disab
led the Datadog Agent, the system CPU consumption reduced significantly. As a next step, the tool confi
guration may be adjusted to work in a less intrusive manner and the provider of this tool can be involve
d in order to reduce the spin lock contention.

Spin locks in the following call stack can indicate overhead in interrupt processing:

<IRQ>
_raw_spin_lock+0x25/0x30
tpacket_rcv+0xa30/0xc70 [af_packet aae77b834dcc1a04b129ed1b495e8972008ea008]

Another symptom is high CPU consumption of Linux processes of type ksoftirqd. This scenario has been
observed in context of Dynatrace Oneagent (oneagentnetwork) after an upgrade to SLES 15.5.

For supportability considerations when running 3rd party solutions see SAP Note 3359552.

High system C Optimize Identify why application is responsible for the processes with the high system CPU utilization and optim
PU consumpti OS calls of ize or reduce these calls.
on caused by o other proc
A typical reason for high system CPU consumption in this context is the configuration of Transparent H
ther processes esses
uge Pages (see SAP Note 2131662). Make sure that you disable this feature based on the provided recom
mendations.

A more detailed analysis of activities involving system CPU consumption can be performed via OS kerne
l stack backtrace as described in SAP Note 2166414.

High CPU cons Optimize Reduce unnecessary high CPU requirements of non SAP HANA software running on SAP HANA hosts. I
umption of no non SAP f required, get in touch with the vendor of the software in order to ask for assistence.
n SAP HANA p HANA sof
rocesses tware

High SAP HAN Analyze ex See "How can CPU intensive operations in SAP HANA be identified and optimized?" for more details.
A user CPU co pensive S
nsumption AP HANA
operations

9. How can CPU intensive operations in SAP HANA be identified and optimized?
In order to identify CPU intensive operations within SAP HANA you can use SQL:
"HANA_Threads_ThreadSamples_FilterAndAggregation" (SAP Note 1969700) in the following way:
• If there is a specific time of high resource consumption, restrict BEGIN_TIME and END_TIME accordingly.
• Set the filter THREAD_STATE = 'Running', because typically the threads in state 'Running' are responsible for CPU
consumption.
• Set AGGREGATE_BY to 'HASH', so that the statement hashs of the top SQL statements are displayed.
• If no statement has exists for most samples, you can set AGGREGATE_BY to 'THREAD_TYPE, THREAD_METHOD,
THREAD_DETAIL' (or a subset) to understand which types of threads are responsible.
If you have identified top statement hashs, you can check if these SQL statements can be tuned. See SAP Note 2000002 for
more information related to SQL optimization.
For more detailed information like the call stacks you can collect additional runtime information (SAP Note 2313619):
• SQL: "HANA_Threads_Callstacks" (SAP Note 1969700) can be used to retrieve the call stacks of the currently active
threads.
• A runtime dump with call stacks and further runtime information can be created (SAP Note 1813020).
• Even more details can be retrieved with a kernel profiler trace (SAP Note 2800030).
Additionally make sure that the SAP HANA parameters are set based on the standard recommendations (see SAP
Note 2186744). For example, SAP Note 2207501 describes CPU issues if a specific undocumented parameter is activated.
Typical scenarios of high SAP HANA CPU consumption are:
Symptom / Call CP Details
stack U
Ty
pe

ptime::DeletedPag Sys Due to a bug with SAP HANA Rev. 102.01 to 102.02 a blocked garbage collection can result after some time
eList::recycle te in a high CPU consumption during recycling of the deleted page list. Therefore you have to put a particular f
m ocus on garbage collection issues (SAP Note 2169283) with these Revisions in order to avoid the escalation.
See SAP Note 2253017 for more information.

__GI___sched_yi Sys Up to SAP HANA 1.0 SPS 10 the row store uses active lock waits at different locations. Several of these activ
eld te e waits were removed as of SAP HANA 1.0 SPS 11.
m
With SAP HANA 1.0 <= SPS 11 it can happen that ptime::PageHeader::latch is responsible for __GI___sch
ed_yield waits, This problem is fixed starting with SAP HANA 1.0 SPS 12.

__GI___sched_yi Sys This specific yield wait is linked to the function cache (SAP Note 2502256). The lock holder is typically activ
eld te e in LinkHash::find. Entries in the function cache are removed when the related SQL cache entries are evict
ptime::Futex::lock m ed. In most cases contention is a consequence of too many SQL cache evictions, so you need to make sure th
expr::Function at evictions remain on a reasonable level (SAP Note 2124112).
Map::_matchID
You can use SQL: "HANA_SQL_SQLCache_Overview" (SAP Note 1969700) to check for evictions. Check I
Ds M1129 ("SQL cache evictions / h (short-term)") and M1130 ("SQL cache evictions / h") of the SAP HANA
Mini Checks (SAP Note 1999993) return a potentially critical rating in case the evictions are higher than ex
pected.

Starting with SAP HANA 2.00.048.00 you can disable the function cache using the following SAP HANA pa
rameter:

indexserver.ini -> [sql] -> enable_function_cache = false

Starting with SAP HANA 2.0 SPS 06 the function cache implementation no longer exists (issue number 236
643) and the parameter has no effect and can be removed.

Network related o Sys If you observe increased system CPU calls in context of network related call stack methods and increased st
perating system ca te atement load, Linux kernel taints or Linux kprobe instrumentation by external tools can be responsible. See
lls like: m Linux kernel taints and kprobe instrumentation for more information.

__libc_recv
__libc_send
__writev
__writev_nocance
l
Execution::Contex Sys This scenario can happen if CPU tracking is activated on SAP HANA <= 1.0 SPS 11:
t::getExecutionIns te
trumentationSnap m global.ini -> [resource_tracking] -> cpu_time_measurement_mode
shot
Execution::Contex Deactivate CPU tracking if many threads are working in the mentioned call stacks.
tStatistics::getCurr
entExecutionSnap
shot
Execution::Executi
onInstrumentatio
nSnapshot::takeM
easurement
Execution::Thread
::getSysExecution
Time
Execution::TraceC
ontextAggregatabl
e::continueKPI
TRexUtils::CPUTi
meMeasurement::
calculateCurrentTi
mes
TRexUtils::CPUTi
meMeasurement::
getAggregatedTim
es
TRexUtils::CPUTi
meMeasurement::
pauseMeasuremen
t
TRexUtils::CPUTi
meMeasurement::
resumeMeasurem
ent
TRexUtils::CPUTi
meMeasurement::
retrieveTimes
TRexUtils::CPUTi
meMeasurement::
startMeasurement
TRexUtils::KPIMa
nager::startMeasu
rement
TRexUtils::KPIMa
nager::stopMeasur
ement
__lll_lock_wait_p Sys These call stacks are typically linked to timezone conversions. You can consider the following aspects to red
rivate te uce the CPU consumption:
__lll_unlock_wak m
Make sure that the TZ environment variable (if defined at all) of the user starting SAP HANA is configured
e_private
with a proper POSIX value (SAP Note 2360909). Be aware that 'UTC' isn’t a proper value, instead you have
__tzfile_compute
to use e.g. 'UTC0'.
__tzset_parse_tz
__tz_convert For ABAP systems make sure that the following SAP HANA parameters point to the proper timezone tables
_L_lock and client 000 (SAP Notes 1791342):
_L_unlock
abapSysTimezone indexserver.ini -> [global] -> timezone_default_data_client_name = 000
indexserver.ini -> [global] -> timezone_default_data_schema_name = <abap_schema_wit
abapSysTimezone
h_TZ_tables>
_null
Evaluator::ThreeC
Additional settings required for SAP HANA <= 2.0 SPS 03:
odeProcessor::no
w xsengine.ini -> [global] -> timezone_default_data_client_name = 000
xsengine.ini -> [global] -> timezone_default_data_schema_name = <abap_schema_with_T
Z_tables>

Make sure that the configured timezone (M_HOST_INFORMATION -> timezone_name) is configured in t
able TTZZ. Switch to a different timezone (e.g. INDIA instead of IST) if possible. Be aware that M_HOST_I
NFORMATION may display a summer time while in TTZZ only the basis timezone is contained, this is okay
(e.g. CEST vs. CET, PDT vs. PST).

Adjust SQL statements responsible for a high amount of timezone conversion requests (e.g. by replacing cal
ls like now() with fix time values determined beforehand).

__cxa_throw Sys A high system CPU consumption in these call stacks is typically related to costly exception handling. See SA
__cxa_throw.cold te P Note 2313619 -> C0400 ("Exception unwinding") for more information and analysis steps.
__cxxabiv1::__cx m
a_throw
__dl_iterate_phd
r
__GI___pthread
_mutex_lock
__lll_lock_wait
__lll_unlock_wak
e
_L_lock_<id>
ltt::exception::do_
throw
ltt::impl::throw_c
heck
pthread_mutex_lo
ck
__pthread_mutex
_unlock_usercnt
_Unwind_Find
_FDE
_Unwind_GetText
RelBase
_Unwind_RaiseE
xception
_Unwind_Resum
e

Persistence garbag Sys If persistence garbage collectors (GCJob*) repeatedly wait for the FileIDMapping lock and consume signific
e collectors (GCJo te ant amount of system CPU, open a SAP case for further analysis.
b*) waiting for File m
In order to reduce the CPU consumption caused by parallel garbage collection you can set the following par
IDMapping locks
ameter to a lower value (details see further below):

global.ini -> [persistence] -> max_gc_parallelity

Translation lookas Sys If you observer a high system CPU consumption related to translation lookaside buffer (TLB) shootdowns o
ide buffer shootdo te n operating system level, you should check the following SAP Notes:
wns m
• SAP Note 2040144: Collecting Diagnosis Information from the SAP HANA Studio can cause TLB shoo
tdowns (Rev. <= 1.00.74.04)
• SAP Note 2206354: High System CPU Consumption Caused by Plan Trace

DataContainer::Vi Sys A high system CPU consumption on secondary sites of system replication environments (SAP Note 199988
rtualFileStatsProx te 0) with replication mode logreplay can be caused by a high number of LogRecoveryQueue<id> threads. See
y::getOrCreateVirt m SAP Note 1999998 ("vfLOBStatsMapLock") for more details and consider reducing the number of log recov
ualFileLOBStatisti ery threads with the following parameter (SAP Note 2222250):
cs
DataContainer::Vi global.ini -> [persistence] -> recovery_queue_count
rtualFileStatsProx
y::removeVirtualF
ileLOBStatistics

ptime::SparsePage Sys As a consequence of other problems (e.g. MemoryReclaim waits, SAP Note 1999998) a high system CPU co
Pool::Container::P te nsumption is possible in this module because of a spin lock implementation. This problem is fixed with Rev.
artition::Entry::ex m 1.00.122.07 and 2.00.002.
ePopForAlloc

__GI_kill Sys In context of allocation limit checks (SAP Note 1999997) a "kill -0" is sent to different SAP HANA services (
System::ProcessIn te module __GI_kill) in order to perform a process existence check. In specific scenarios this operation can be
formation::isProce m quite slow, consuming significant amounts of system CPU. This can slow down different operations queryin
ssAlive g the effective allocation limit, e.g. the nameserver ping (SAP Note 2222110). If you face this scenario you sh
MemoryManager:: ould check on operating system side why the system CPU consumption is so high, and eliminate the root ca
GlobalMemoryHa use. With SAP HANA >= 1.00.122.15, >= 2.00.012.04 and >= 2.00.24 the __GI_kill based existence check
ndler::processIsAli will be disabled in context of memory management.
ve
MemoryManager::
GlobalMemoryInf
os::getCurrentEffe
ctiveAllocationLim
itRespectingOther
Processes
syscall Sys When memory defragmentation happens, contention and spin locks are possible during munmap calls, resu
MemoryManager:: te lting in increased system CPU consumption. See SAP Note 1999997 ("What is memory garbage collection?"
munmapOverride m ) for more details.
System::UX::m
unmap
System::memFree
SystemPages
MemoryManager::
SystemPageMemo
ryReleaser::freeSy
stemPagesInternal
MemoryManager::
SystemPageMemo
ryReleaser::freeSy
stemPages
MemoryManager::
freeSystemPagesI
nternal
MemoryManager::
MemorySource::d
eallocateSystemM
emory
MemoryManager::
BigBlockAllocator:
:munmapGarbage
MemoryManager::
BigBlockAllocator:
:cleanupGarbage
MemoryManager::
BigBlockAllocator:
:collectGarbageFul
l
MemoryManager::
MemoryPool::coll
ectGarbageBBA
MemoryManager::
MemoryPool::coll
ectGarbage
MemoryManager::
PoolAllocator::coll
ectGarbage
MemoryManager::
ReclaimMemoryT
askHandler::Task
Base::executeTask
MemoryManager::
ReclaimMemoryT
askHandler::execu
teOneTasks
MemoryManager::
GlobalMemoryHa
ndler::pmParallelS
upportProvideMe
mory
MemoryManager::
GlobalMemoryHa
ndler::pmAcquireI
PMMLockOrSupp
ortProvideMemor
yHlp
MemoryManager::
GlobalMemoryHa
ndler::pmAcquireI
PMMLockOrSupp
ortProvideMemor
y

Linux call stack: Sys A Linux soft-lockup due to a high number of directory entries can result in high system CPU consumption a
te nd a hanging database (SAP Note 2753342).
evict
m
__dentry_kill
shrink_dentry_lis
t
shrink_dcache_pa
rent
proc_flush_task
release_task
do_exit
do_group_exit
get_signal_to_del
iver
do_signal
do_notify_resume
int_signal

Increased system a Sys The problem may be caused by a high number of changes to a specific table record in row store, which resul
nd user CPU cons te ts in version consolidation overhead.
umption m,
Check from application side if a particularly high amount of changes is executed against single table records
SQLExecutor thre Us
and try to reduce the change frequency. If the problem is caused by UPDATE operations on table QIWKTA
ads in status "Run er
B you should check for a proper RFC queue configuration. For example, SMQ2 entries in "STOP" state can r
ning" with slow ac
esult in a permanent re-execution of the QIWKTAB UPDATE operations and should be avoided.
cess to specific tab
le From a SAP HANA perspective the row store version consolidation of single records is further improved wit
MVCCGarbageColl h SPS 09 and will also reduce the probability of performance escalations.
ector thread active
or threads in Colle
ctVersion method

JoinEvaluator::Ta Sys In some cases there is an active wait for index handles (SAP Note 1999998) that can result in a massive incr
bleAccess::acquire te ease of system and user CPU consumption:
SearchLockAndRe m,
• When a delta merge has to wait for the index handle (TRexConfig_IndexMgrIndex_FastLock), it bloc
play Us
ks the index handle for all other requests to the table and other sessions may spin on CPU actively.
TRexAPI::Search:: er
• When an index handle request is started without a dedicated timeout, it checks once per microsecond
TableHandleProxy
for a cancellation signal.
::initHandle
TRexConfig::Index These scenarios are fixed starting with SAP HANA 1.00.122.08.
Handle::acquire
TRexConfig::Index
Handle::fastShare
dAcquire
TRexConfig::Index
Mgr::getBucket
Synchronization::
ReadWriteLock::ti
medWaitLockExcl
usive

Index handle relat

ed lock waits like:

TRexConfig_Inde
xMgrIndex_Lock
TRexConfig_Inde
xMgrIndex_FastL
ock
Execution::JobOp Sys These methods (often related to jx-pq* mutex waits, SAP Note 1999998) are prominent in case a high numb
enQueue::cleanup te er of concurrent, small jobs is executed. Analyze and optimize expensive database requests with a high Job
AfterGet m, Worker demand (SAP Note 2000002).
Execution::JobOp Us
With SAP HANA >= 2.00.066 the internal SAP HANA job handling is optimized (issue number 296880) w
enQueue::getNext er
hich reduces the user and system CPU overhead.
JobFrom
Execution::JobExe
cutorImpl::getNex
tSpecificFromNeig
hborNumaNodes

syscall Sys These syscalls are used to determine the current thread ID (getCurrentThreadID). In particular when the n
ptime::TcpRece te umber of threads is significant (e.g. due to lock contention or overload scenarios) the performance of these
iver::doWork m, calls can deteriorate and result in an increased CPU and system CPU consumption. Proceed as follows to op
ptime::TcpReceive Us timize this scenario:
r::runInner er
• Eliminate the main issue like lock contention or overload. Then also the risk of running into this follo
ptime::TcpReceive
w-up issue is reduced.
r::run
• Upgrade to SAP HANA >= 2.00.059.07 or >= 2.00.067 where the syscalls have been replaced with a
syscall more light-weight context ID retrieval (issue number 299078).
SessionLayer::C
This scenario typically results in several threads reported with method NoAction for a longer time (SAP Not
ommEventHold
e 2114710).
er::setPendingE
vent At the time of the issue call stack generation with runtime dumps or for the statistics server call stack histor
SessionLayer::Eve y is impacted.
ntIterator::reset
SessionLayer::Tcp
CommMgr::waitE
vents
ptime::TcpReceive
r::doWork
ptime::TcpReceive
r::runInner
ptime::TcpReceive
r::run

High user CPU co Us See SAP Note 2000002 and optimize the SQL statement.
nsumption caused er
by specific SQL sta
tement

ptime::TransToke Us If SAP HANA SPS <= 1.0 SPS 09 is used and threads with ptime::TransTokenMgr::checkTIDBufferFlush in
nMgr::checkTIDB er their call stacks are responsible for a high CPU consumption, the issue is linked to a transaction ID buffer fl
ufferFlush ush check. You can set the following parameter as a workaround:

indexserver.ini -> [transaction] -> maxtidvectorsizefortidbuffer = '0'

Starting with SAP HANA SPS 10 this problem no longer exists.

High CPU consum Us If you experience high CPU consumption in the context of Lumira, you can check if disabling the jobs as des
ption with Lumira er cribed in SAP Note 2106466 reduces the CPU utilization, and make sure that you only use supported Lumir
a functions (SAP Note 2195103).
__GI___sched_yi Us This call stack is linked to parsing activities (SAP Note 2332551). Check according to SAP Note 2124112 if th
eld er e SQL cache is configured and used optimally in order to minimize parsing activities.
Synchronization::i
mpl::SpinLock::lo
ck
ptime::Query::Pla
nHandle::validate
ptime::PreparedSt
atement::validate
_
ptime::Statement::
execute_

Synchronization m Sys The Synchronization::Mutex modules administer mutexes from a SAP HANA perspective, e.g. they record d
odules like te etails that are accumulated and displayed in monitoring view M_MUTEXES. See SAP Note 1999998 for mo
m, re information related to locks and mutexes in SAP HANA environments.
Synchronization::
Us
Mutex::lock A particularly high amount of lock / unlock activities can result in increased user and system CPU consump
er
Synchronization:: tion. Known scenarios are described in SAP Note 2313619:
Mutex::setOwner
• Check ID C1020 ("Unlocking of mutex")
Synchronization::
• Check ID C1021 ("Unlocking of non-recursive mutex")
Mutex::unlock
Synchronization::
NonrecursiveMute
x::unlock

INSERT / UPDAT Us A high CPU consumption in these modules can be a consequence of a high amount of accesses to the SAP H
E / DELETE threa er ANA resource container. In this context you can often see threads of type "Resource Load Wait" or waiting f
ds with modules li or UndoAnchor.
ke:
Most commonly this issue happens in context of undo file processing (module DataAccess::UndoFileAncho
PageAccess::Logic r::allocateFile). In this case the following optimizations exist:
alPageAccessImpl:
• Starting with SAP HANA 1.00.122.06 and SAP HANA 2 you can increase the size of the undo file cach
:loadPageInternal
e (default: 64 entries) by increasing the parameter indexserver.ini -> [persistence] -> undo_file_cach
PageAccess::Logic
e_size. This will eliminate the repeated creation and deletion of new undo files.
alPageControlBloc
• Starting with SAP HANA 1.00.122.12 the handling is partially improved, so that an upgrade to that Re
k::isIdentifiedBy
vision level or higher can improve the situation (SAP Note 2547543).
ResourceManager:
• Upgrade to SAP HANA 2.00.001 or higher where undo file creation is optimized.
:ResourceContain
erImpl::lookupRes
ourceInList
ResourceManager:
:ResourceContain
erImpl::putResour
ceHeaderIntoHas
hBucket
ResourceManager:
:ResourceContain
erImpl::readResou
rce
ResourceManager:
:ResourceContain
erImpl::testEquiv
TryAndIncreaseRe
fWithDisp
ResourceManager:
:ResourceHeader::
addReferenceIfMa
naged
ResourceManager:
:ResourceHeader::
releaseAndUpdate
Time
IndexingQueue / Us This can be a problem of the French lingware. See SAP Note 2428875 and implement SAP HANA Rev. 1.00.
Request threads w er 122.07 or 2.00.001.
ith thread method
indexing / prep/in
dexing are active i
n:

inxight::fsmindex:
:fst_lookup_forwa
rd
inxight::fsm_lexic
on::lookup
inxight::simple_fs
m_spec_lookup_f
or_compound::loo
kup
inxight::priority_u
nion_spec_lookup
::lookup
inxight::tagged_st
emming_impleme
ntation::word_tag
_lookup
inxight::tagged_st
emming_impleme
ntation::stem_tag
ged_word
inxight::tagged_st
emming_impleme
ntation::stem_tag
ged_sentence
inxight::tagged_st
emming_impleme
ntation::tagged_st
emming_impleme
ntation

TRexUtils::BitV Us A scan of an indexed column with SPARSE compression can consume significant CPU resources. This probl
ector::iterator:: er em is fixed with SAP HANA >= 1.00.122.10, >= 2.00.002.01 and >= 2.00.012. See SAP Note 2112604 and c
skip_unchecke onsider an uncompression of the related columns.
d
AttributeEngine::S
parseIndexScanVe
cOutHelper
AttributeEngine::S
pDocuments::scan
WithIndexInternal
AttributeEngine::S
pDocuments::scan
<AttributeEngine:
:ScanVectorBinSe
archPredicate
AttributeEngine::S
pAttribute<TrexTy
pes::StringAttribut
eValue, AttributeE
ngine::ValueDict
AttributeEngine::
AttributeApi::jeRe
adIndex
JoinEvaluator::Ta
bleAccess::jeReadI
ndex
MemoryManag Us Due to a bug there can be an overhead when updating reference counts of memory objects that can result in
er::PoolAllocat er high CPU consumption. The behavior is optimized with SAP HANA >= 1.00.122.11, >= 2.00.012.01 and >=
or::addReferen 2.00.020.
ce
As a workaround memory_tracking can be disabled (SAP Note 1999997):
TRexUtils::TRexK
PIMgrCB::createM global.ini -> [resource_tracking] -> memory_tracking = 'off'
emoryStatisticsCol
lector Be aware that disabling this feature also deactivates the statement memory limit and so you are at risk that
Execution::TraceC a single query consumes a very high amount of memory. Therefore you should additionally set the following
ontextAggregatabl parameter to a reasonable lower size of the resource container (SAP Note 1993128):
e::enableResource
Tracking indexserver.ini -> [memoryobjects] -> unload_lower_bound = '<min_size_of_resource_c
Execution::TraceC ontainer>'
ontextAggregatabl
e::resetAggregatio
ns

MemoryManag
er::PoolAllocat
or::release
TRexUtils::Memor
yStatistics::~Mem
oryStatistics>
TRexUtils::Memor
yStatisticsCollecto
r::~MemoryStatist
icsCollector
ltt::allocated_refc
ounted::destroyIm
p

JoinEvaluator::Lo Us This module indicates expensive join engine operations. It often correlates to a large size of heap allocator P
opJob::findJoinPa er ool/JoinEvaluator/JECreateNTuple (SAP Note 1999997). A related SAP HANA bug is fixed with Rev. 2.00.
irsTL_native 002 and higher. As a workaround the NO_GROUPING_SIMPLIFICATION hint (SAP Note 2142945) can b
e used. If triggered by BW / MDX, you can also disable the RSADMIN parameter MDX_F4_USE_SQL (SA
P Note 1865554).

Execution::Job Us This call stack indicates contention related to resource limitation settings via parameter default_statement
FinalObjectImp er _concurrency_limit (SAP Note 2222250). As a workaround you can disable the default_statement_concurr
l::initializeState ency_limit setting. This issue is fixed with SAP HANA >= 1.00.122.11, 2.00.002.02 and 2.00.010.
mentConcurren
cyLimit
Execution::JobFin
alObjectImpl::Job
FinalObjectImpl
Execution::JobExe
cutorImpl::execut
e
Execution::JobCo
ntextImpl::startEx
ecution

Stream::NetworkC Us A significant CPU consumption in SSL network modules can be caused by the SAP HANA bug described in
hannelSSLFilter::r er SAP Note 2385992.
eceiveRaw

QueryEntryPrunin Us This module indicates BW multiprovider pruning activities. As a workaround you can disable multiprovider
gEvaluator::doesP er pruning by setting the following parameter:
artproviderMatchI
nternal indexserver.ini -> [calcengine] -> disabled_patterns = 38

A SAP HANA optimization is available with >= 1.00.122.13, >= 2.00.012.02 and >= 2.00.021.
TRexCommonObj Us These call stack methods are linked to the itab leak trace that can be used to understand memory consumpti
ects::ItabLeakTrac er on in heap allocator Pool/itab (SAP Note 1999997). Make sure that the itab leak trace is only activated as ra
eWrapper::addCol rely and shortly as possible in order to avoid locking and performance overhead. The following related SAP
TRexCommonObj HANA parameters must not be set, otherwise the trace may be active:
ects::ItabLeakTrac
eWrapper::remove indexserver.ini -> [itab] -> trace_leaks
Col indexserver.ini -> [itab] -> trace_leakcallstack

AttributeEngine:: Us In general these modules indicate column scans, e.g. caused by an inappropriate application queries or insu
ClusterPredScanB er fficient index design.
vOutHelper
A unusual high thread activity in AttributeEngine::*Scan*OutHelper modules with SAP HANA 1.00.122.10
AttributeEngine::
- 1.00.122.11, 2.00.000 - 2.00.012.01 and 2.00.020 can be caused by the bug described in SAP Note 251680
ClusterPredScanV
7. As a workaround you can set the SAP HANA parameter
ecOutHelper
AttributeEngine:: indexserver.ini -> [optimize_compression] -> singleindex_consider_for_compressed_co
ClusterScanBvOut lumns = false
Helper
AttributeEngine:: and recompress the involved tables.
ClusterScanVecOu
tHelper
AttributeEngine::I
ndirectPredScanB
vOutHelper
AttributeEngine::I
ndirectPredScanV
ecOutHelper
AttributeEngine::I
ndirectScanBvOut
Helper
AttributeEngine::I
ndirectScanVecOu
tHelper
AttributeEngine::S
parseBvScanBvOu
tHelper
AttributeEngine::S
parseBvScanVecO
utHelper
AttributeEngine::S
parsePredScanBv
OutHelper
AttributeEngine::S
parsePredScanVec
OutHelper
AttributeEngine::S
parseRangeScanB
vOutHelper
AttributeEngine::S
parseRangeScanV
ecOutHelper
ltt::impl::casInt8 Us Active spinning in context of Execution::JobOpenFastUnsorted::getNextJobFrom can happen on systems w
ltt::impl::AtomicO er ith more than 8 CPU sockets in unspecific situations, particularly in cases of already increased CPU consum
peration<8ul>::ca ption. This problem is fixed starting with SAP HANA 1.00.122.15.
s
ltt::casPtr<Executi
on::Context>
Execution::Job
OpenFastUnsor
ted::getNextJob
From
Execution::JobExe
cutorImpl::getNex
tJobFromNumaN
odes
Metadata::Grante Us These call stack methods are linked to catalog authorization checks that are typically linked to SAP HANA
dPrivInfo::getPriv er monitoring and dictionary views that are defined with a HASANYPRIVILEGES privileges check. You can av
Id oid these expensive checks by assigning the CATALOG READ role to the executing user.
Metadata::Grante
HASANYPRIVILEGES is removed from the definition of certain views with the following SAP HANA Revisi
dPrivInfo::getPriv
on levels:
InfosOfObj
Metadata::Grante Views Optimized with
dPrivInfo::getPriv
FUNCTIONS >= 1.00.122.10
Infos
M_CS_ALL_COLUMNS >= 2.00.002.02
Metadata::Assigne M_CS_COLUMNS >= 2.00.012
dRoleInfo::search M_CS_TABLES
TransitiveClosure M_RS_TABLES
M_TABLES
ByGranteeId
OWNERSHIP
Authorization:: PROCEDURES
CatalogPrincipa SYNONYMS
lStorageManage TABLE_COLUMNS
TABLES
r::getTransitive
TRIGGERS
ClosureOfAssig VIEW_COLUMNS
nedRolesForGr VIEWS
antee
M_TEMPORARY_TABLES >= 1.00.122.14
Authorization::
>= 2.00.012.04
CatalogAuthori >= 2.00.023
zationStorageM
anager::isAutho
rized
Authorization::Pri
ncipal::readTransi
tiveRoleClosureFr
omStorage
Authorization::Pri
ncipal::getTransiti
veRoleClosure
Authorization::Aut
horizationManage
r::hasAnyPrivilege
Authorization::Aut
horizationManage
r::isAuthorizedReg
ardlessOfOwnersh
ip
Authorization::Aut
horizationManage
r::isAuthorized
Authorization::Aut
horizationManage
r::isAuthorizedSys
temPrivilege
__hasanyprivilege
s__String_BigInt
_String_String_St
ring
__hasanyprivil
eges__String_B
igInt_String_St
ring
ltt::allocator_stati Us Long runtimes in these call stacks on SAP HANA 1.00.122.14 and 2.00.012.03 can be caused by the SAP HA
stics::getCountInU er NA bug described in SAP Note 2580435.
se
MemoryManager::
PoolAllocator::che
ckDelayedCleanup
MemoryManager::
PoolAllocator::rele
ase

sigprocmask
sigjmp_save
MemoryManager::
PoolBlockEnumer
ator::moveToNext
Block
MemoryManager::
Allocator::dumpIn
UseBlocks
MemoryManager::
Allocator::dumpIn
UseBlocks
MemoryManager::
PoolAllocator::han
dleMemoryLeaks

Synchronization:: Us When late materialization is used (SAP Note 1975448) it can happen in rare scenarios that a high amount of
Mutex::lock er requests for mutex "LateMatColumnSource pageLock" are executed that negatively impact performance. In
TRexAPI::LateMat this case you see many threads in module Synchronization::Mutex::lock, but not in a waiting operating syst
ColumnSource::lo em call (syscall). As a temporary workaround you can consider increasing the late materialization threshold
ckLoadAndActivat so that less queries use late materialization (default: 20,000), e.g. to 1 million:
ePage
TRexAPI::LateMat indexserver.ini -> [search] -> late_materialization_threshold = 1000000
ColumnSource::ge
tNativeTEMP Be aware that this increase can result in higher memory requirements.

Diagnose::impl::T Us A high CPU consumption in these modules indicates massive generation of database trace information (SA
raceBuffer::sync er P Note 2380176). Check for what reason so many trace entries are written (activated trace, problem scenari
Diagnose::TraceO Sys o, ...) and take appropriate actions to reduce it.
utputFileHandlerI te
mpl m
Diagnose::TraceSe
gment::reserveEnt
ry
Diagnose::TraceSe
gment::waitReady
Diagnose::TraceSt
ream::flushTraceB
uffer
ptime::qo_Case::is Us These modules are traversed when the query optimizer is used to optimize the plan in calculation engine en
_same_exp er vironments. Long runtimes are typically caused by complex models. For testing purposes or as temporary w
ptime::qo_Comp:: orkarounds you can set the following calculation view execution hint:
is_same_pred
ptime::qo_Comp:: ce2qo_for_nested_views_with_sql_execution = 0
less_than
ptime::qo_Conj::is In order to disable it on global level, the following SAP HANA parameter can be set:
_same_pred
indexserver.ini -> [calcengine] -> ce2qo_for_nested_views_with_sql_execution = 0
ptime::qo_Func::i
s_same_exp
Be aware that this setting can result in performance degradations, so it should only be implemented with ca
ptime::qo_Func::i
re.
sSharable
ptime::qo_Normal Even with this setting the call stack can still show up in certain scenarios. In this case further simplification
izer::applyRule s like the hint OPTIMIZATION_LEVEL(MINIMAL) may be helpful (SAP Note 2142945).
ptime::qo_Normal
izer::case_normali
ze
ptime::qo_Normal
izer::eliminate_du
plicate_terms
ptime::qo_Normal
izer::exp_pred_no
rmalize
ptime::qo_Normal
izer::is_subtree_g
rouping_exp
ptime::qo_Normal
izer::normalize
ptime::qo_Normal
izer::normalize_di
sj
ptime::qo_Normal
izer::pred_normal
ize
ptime::qo_Normal
izer::push_down_
select
ptime::qo_Normal
izer::push_down_
select_one_rel
ptime::qo_Normal
izer::rel_pred_nor
malize
ptime::qo_Normal
izer::rewrite
ptime::qo_Normal
izer::select_thru_j
oin
ptime::qo_Normal
izer::simplify_com
parison
ptime::qo_Normal
izer::simplify_pre
d
ptime::qo_Predica
teDerivation::appl
y
ptime::qo_Predica
teNormalization::a
pply
ptime::qo_Rule::c
heckAndApply
ptime::qo_SelectP
ushDown::apply
ptime::qo_Visitor
Util::qo_check_rel
ations
ptime::qo_Visitor
Util::qo_visit_pre
ds
ptime::qo_Visitor
UtilImpl::check_al
l_internal
ptime::qo_Visitor
UtilImpl::visit_all
_internal
ptime::QueryOpti
mizer::optimize_p
lan
ptime::CommonA
piImpl::optimizeQ
oPlan
TrexCalculation
Engine::ceQOEx
ecutor::execute

Auditing::Comple Us These call stacks are used in context of SAP HANA auditing (SAP Note 2159014) when auditing is globally a
xObjectCallback::g er ctivated with the following SAP HANA parameter:
etAllUnderlyingOb
jectsWithOwner global.ini -> [auditing configuration] -> global_auditing_state = true
Auditing::ViewObj
ectHierarchyCallb Activation of this parameter is useful and required when auditing policies are defined, in other cases it shou
ack::getBaseObjec ld not be set to true in order to avoid unnecessary overhead in terms of CPU and memory (Pool/Auditing).
ts
Auditing::AuditEv
entFactory::getRel
atedEventList
Auditing::AuditEv
entFactory::genera
teEventsFromActi
onList
Auditing::AuditEv
entFactory::collect
DMLAuditRelevan
tCheckEntries

Diagnose::Module Us This call stack indicates the generation of call stacks (SAP Note 2313619) in context of configured traces like
Info::getModuleFo er allocation stack trace (SAP Note 2222218) or memory fencing (SAP Note 2222276).
rAddress
Check IDs M0661 ("Heap allocators with allocation stack trace"), M0662 ("Heap allocators with other mem
Diagnose::Unwind
ory traces") and M0663 ("Time since activated memory trace (days)") of the SAP HANA Mini Checks (SAP
Context::decodeU
Note 1999993) can be used to monitor for activated memory traces.
nwindInfo
Diagnose::Unwind SQL: "HANA_Memory_Heap_Flags" (ONLY_TRACE_FLAGS = 'X') available via SAP Note 1969700 can b
Context::step e used to check for heap allocators with activated trace flags.
Diagnose::CallStac
k::fastStackUnwin
d
Diagnose::CallStac
k::fillTrace
Diagnose::CallStac
kManager::saveSt
ackLimited
MemoryManag
er::CheckedBlo
ckHandler::cre
ateStacktrace
__cxa_free_excep Us This call stack is used in context of modification commands that change columns that are used for partitioni
tion er ng (SAP Note 2044468) - this imposes the possibility that the records need to be moved from one partition
TRexUtils::BitVect to the other and for that reason a significant overhead is possible.
or::resize
Among others this scenario can happen when you have partitioned a table by the complete primary key and
TrexStore::UdivLi
any of the primary key columns is adjusted. As suggested in SAP Note 2044468 it is usually helpful to partit
stContainerMVCC:
ion tables with HASH on the most selective primary key column. This reduces the risk that an updated colu
:reCreateUdivList
mn is part of the partition columns and so partition moves are less likely.
EqualSSN
TrexStore::Udiv The following optimizations exist in this scenario:
ListManager::g • Use SAP HANA >= 2.0 SPS 01 where this check is implemented more efficiently.
etEqualSSNUdi
• Partition the table based on a column that isn't modified.
vs
• Reduce the modification rate of columns being part of the partitioning criteria by changes on applicati
qo3::MethodInfo::
on side.
validUdivs
• Perform more frequent delta merges (SAP Note 2057046) on the underlying table to keep the critical
qo3::MethodInfo::
delta storage at a rather small size (trade-off between performance gain and delta merge overhead).
allDocs

[vgettimeofday] Us The operating system call __vdso_gettimeofday is a potentially slower fallback mechanism to retrieve time
__vdso_gettime er information when RDTSC / RDTSCP (Read Time Stamp Counter) isn't properly available for SAP HANA. T
ofday his call happens in user space, so you see user CPU, not system CPU. The trace file contains a warning like:
TimerImpl::gettim
• Fallback to system call for HR timer
eofday_MicroTim
• Usage of system timer is forced because the clocksource file '/sys/devices/system/clocksource/clockso
er
urce0/current_clocksource' has no 'tsc' clocksource
Basis::Timer::get
MicroTimer In this case check ID T0868 ("Fallback to system call for high resolution timer") may be reported by SQL: "
Basis::Timer::get HANA_TraceFiles_MiniChecks" (SAP Note 2380176).
HiPerfTimer
In order to avoid overhead caused by a slow timer you need to make sure that RDTSC / RDTSCP is properly
ptime::scoped_ti
configured on Linux level or that an efficient fallback mechanism is in place. See SAP Note 2753418 for mor
mer_for_result_t
e information.
_open::_end
ptime::scoped_ti For different environments problems are already known:
mer_for_result_t • AWS (SAP Note 2915232)
_open::~scoped_t • Azure (SAP Note 2791572)
imer_for_result_t • SLES 12.3 (SAP Note 2859234)
_open
• VMWare (SAP Note 2823760)

clock_gettime Us This is another call stack indicating overhead in context of retrieving time information. See SAP Note 27534
__clock_gettime_ er 18 and make sure that all timer recommendations like a proper clocksource configuration are in place.
2
__GI___clock_ge
ttime
fallback_ticks
ltt::timer::us
ltt::timer::ns_mon
otonic
rte::timer:ms
rte::timer:us

__wcscpy_chk Us This call stack can be a consequence of significant FDA READ activities in context of fast data access (FDA,
ptime::SessionCod er SAP Note 2399993) that is required to create and populate the internal table (itab). This bottleneck can be a
ecNewdb::eResult Sys reason of fluctuating response times of quick database requests. In the worst case, contention in OS call __
SetItab te wcscpy_chk can also impact other database requests that don't use FDA READ, but use memory operations
m like __wcscpy_chk in different contexts.
Thread method: E
xecQidItab As a temporary workaround you can consider to disable FDA READ by setting the following SAP ABAP prof
ile parameters (SAP Note 2399993):

• rsdb/supports_fda_prot = 0
• dbs/hdb/supports_fda_prot = 0

Starting with SAP HANA 2.00.024.04 and 2.00.32 the overhead of FDA READ is significantly reduced.
Catalyst::Transacti Us This contention can be caused by a SAP HANA bug with Revisions 2.00.024.07 - 2.00.024.08 and 2.00.034
onBrokerClient::a er - 2.00.035 (SAP Note 2748836).
cquireSnapshotTi
mestamp

businessDB::Quer Us These call stack methods indicate processing in context of currency conversion. You can reduce the overhea
yTools::linearizeDi er d by activating the currency conversion cache (SAP Note 2502256).
ctionaryOf
When businessDB::CacheInstanceImplementation appears in the call stack, the currency conversion cache i
businessDB::Quer
s already activated and used.
yTools::linearizeDi
ctionariesOf
businessDB::Si
mpleQuery::run
businessDB::Simp
leQuery::operator
businessDB::ERP
CurrencyConversi
on::fillConversion
RatesTable
businessDB::Conv
ersionMethodsReg
istry::createRatesT
able
businessDB::Conv
ersion::computeC
onversionRatesTa
ble
businessDB::Conv
ersion::convert
businessDB::Conv
ersion::operator

QueryMediator::F Us SAP Note 2785840 describes a problem with SAP HANA <= 1.00.122.16, <= 2.00.024.02 and 2.00.030 wh
ilterTransportOpt er ere filter transport estimations in the OLAP engine can be quite expensive in context of concurrent DML op
imization::apply erations. As a workaround it may help to disable precise filter transport estimations:
OlapEngine::OlapEn
gine::executeOLAPQ indexserver.ini -> [query_mediator] -> use_precise_filtertransport_estimations = fa
uery lse
TRexUtils::OutputVe Us This call stack indicates a full scan with HEX (SAP Note 2570371). On SAP HANA 2.00.040 it can be caused
c::fill er by the bug described in SAP Note 2793751. In general it can also be caused by an expensive database reques
TRexUtils::IndexVect t that can be analyzed and optimized based on SAP Note 2000002.
orGenericImpl::mget
SearchBv
TRexUtils::IndexVect
orRef::mgetSearch
AttributeEngine::Mai
nDataVectorScanRea
der
hex::cs::DataVectorS
canOrLookupBaseOp
hex::LeafOperator::xf
_run2
hex::PullOperator::p
ull
hex::cs::Conjunction
Op::run
hex::PullOperator::p
ull
hex::cs::PostFilterCo
njOp::run
hex::cs::FragmentSca
nOpWithRowPos::ru
n
hex::cs::TableSca
nControlOp::run
hex::impl::runNextI
mpl
hex::WsTaskJob::run

TransactionMana Us A high amount of metadata requests in scale-out systems isn't processed efficiently on the master node with
ger::MinReadTST er SAP HANA 1.0 and a high CPU utilization can be observed in module TransactionManager::MinReadTSTra
racker::registerMi cker::registerMinReadTS. Consider the following steps:
nReadTS
• Upgrade to SAP HANA 2.0
Newdb::DistMDCach
• Optimization of metadata requests (SAP Note 2222200 -> "What can be reasons for threads in networ
eServer::handle
k related states?" -> "Metadata access").

JoinEvaluator::Tuple Us This call stack is linked to cyclic join processing, the related thread method is ParallelLoopWithEqJob (SAP
List::partitionFind er Note 2114710). If the high CPU consumption can't be resolved differently (e.g. by adjusting application or S
JoinEvaluator::Tuple AP HANA configuration), you can disable cyclic joins via SAP HANA hint NO_CYCLIC_JOIN (SAP Note 21
List::partition_iterat 42945).
or::operator++
Issue number 299491 is used to check if offering a global parameter for controlling cyclic joins is possible.
JoinEvaluator::Lo
opWithEqJob::rec
urseWithEq
AttributeEngine::fnJ Us These call stack methods can become more dominant in case memory management is a bottleneck because
eReadIndexIndividu er they either take care for the memory management itself or they typically perform an increased number of m
al Sys emory allocations / deallocations. If you suspect that memory management is a bottleneck, you can check it
AttributeEngine::Val te by executing SQL: "HANA_Tests_MemoryOperations" (and optimally also SQL: "HANA_Tests_Arithmeti
ueArray::init m cOperations" and SQL: "HANA_Tests_StringOperations") a few times and evaluate the result using SQL: "
__cxa_free_exceptio HANA_Tests_Results" (SAP Note 1969700).
n
Example: (increased memory operations runtime)
Diagnose::StatisticsD
ata::addChild -----------------------------------------------------------------------------------
__GI___mremap -
IndexVectorAligned:: |TEST_STATEMENT |EXPECTATION |RESULT
get |
introsortLoopAux -----------------------------------------------------------------------------------
JoinEvaluator::Table -
Access::jeReadIndex |SQL: "HANA_Tests_ArithmeticOperations"|<= 500 ms / execution|307 ms / execution
|
JoinEvaluator::Trans
|SQL: "HANA_Tests_MemoryOperations" |<= 1000 ms / execution|41454 ms / execution
lationTableImplVect
|
or::resizeVector
|SQL: "HANA_Tests_StringOperations" |<= 1000 ms / execution|710 ms / execution
ltt_adp::vector::push
|
_back -----------------------------------------------------------------------------------
ltt::allocator::allocate -
ltt::allocate_raw_chu
nk You can use SQL: "HANA_Threads_Callstacks_History" (ONLY_MEMORY_MANAGEMENT_METHOD
ltt::impl::vectorFill S = 'X') available via SAP Note 1969700 in order to filter (and aggregate) historic call stacks linked to memo
ltt::impl::vectorResiz ry management. This can allow you to identify trends like an increase of reported samples for specific days:
e
Example: (AGGREGATE = 'TIME', TIME_AGGREGATE_BY = 'DAY')
ltt::string_base::strin
g_base ------------------------
ltt::uninitialized_fill |SAMPLE_TIME |COUNT|
memcpy_impl ------------------------
MemoryManager::Pr |2021/11/19 (FRI)| 3274|
echargeBase::remap |2021/11/18 (THU)| 9315|
Memory |2021/11/17 (WED)| 9886|
MemoryManager::Pr |2021/11/16 (TUE)|11087|
echargeBase::remap |2021/11/15 (MON)|17608|
|2021/11/14 (SUN)| 262|
BlocksFromLocalFre
|2021/11/13 (SAT)| 722|
eListWithoutUpdate
|2021/11/12 (FRI)| 6412|
Values
|2021/11/11 (THU)| 7372|
MemoryManager::Pr ------------------------
echargeBase::getMe
mory In this case some misconfigurations slowed down the database after a restart on 15th of November and a fix
MemoryManager::Pr was provided on 19th of November. During the time of the issue the increased amount of memory related o
echarge::prepare perations is visible.
MemoryManager::M
If you suspect a bottleneck in the memory management area, you can check SAP Note 1999997 and optimiz
emoryPool::reserveM
e the memory utilization and configuration, e.g.:
emoryAndAllocateBi
gOrHugeBlock • Make sure that the SAP HANA free memory cache doesn't have and adverse effect on performance an
MemoryManager::M d consider disabling it for SAP HANA 2.0 SPS 03 and SPS 04 (SAP Notes 2857553, 2600030). Also di
emoryPool::reserveM sabling it on SAP HANA 2.0 SPS 05 can sometimes be useful. Starting with SAP HANA 2.0 SPS 06 it i
emoryAndAllocateH s disabled per default.
ugeBlock • Optimize database request being responsible for a particularly high memory allocation rate.
MemoryManager::M • Take care for proper memory sizing and configuration to make sure that memory reclaims, compactio
emoryPool::allocateB ns and shrinks remain on a reasonable level.
igOrHugeBlock • Avoid frequent and aggressive manual memory compactions (e.g. "mm gc -f" with hdbcons) and confi
MemoryManager::M guration of parameters async_free_target or async_free_threshold.
emoryPool::allocate • Make sure that proactive reclaims (parameters gc_unused_memory_threshold_rel, gc_unused_mem
MemoryManager::Po ory_threshold_abs) are configured reasonably and in line with SAP Note 2600030.
olAllocator::allocate • SQL: "HANA_Threads_Callstacks_MiniChecks" (SAP Note 2313619) must not report check ID C010
NoThrowImpl 0 ("Proactive memory compaction") as potentially critical.
MemoryManager::Te
mpAllocator::allocate
NoThrowImpl
memset_impl
mremap
__open_nocancel
partitionFind
TRexCommonObject
s::VariableMem::sort
TRexUtils::BitVector:
:highestBitSet
TRexUtils::convertTo
Higher
TRexUtils::IndexVect
orRef::mset
TRexUtils::Mset::fill
TRexUtils::MsetLoop
::run
TRexUtils::MsetWor
d::run
TRexUtils::MultiColu
mnHashMap::init
UnifiedTable::NBitD
ataAccessor64::mset
ValueArray::init

Page faults

TRexAPI::UdivLo Us This call stack method indicates an access via inverted individual index (SAP Note 2600076). In addition to
okupWithoutConc er a normal expensive database request (SAP Note 2000002) this index type can be particularly expensive if t
at::lookupWithout he index column with the highest number of distinct values has some quite frequent values that are also use
Concat d when accessing the table. In this case you can consider to use an inverted value or inverted hash index ins
tead.

DataStatistics::Dat Us High CPU consumption in these modules indicates that the Data Statistics Adviser Cache is activated (SAP
aStatisticsAdviser er Note 2502256). This cache can be responsible for significant CPU overhead, so it should be only activated w
Cache::createNew ith care. Make sure that the following SAP HANA parameters aren't set to true to disable this adviser and its
Entries overhead:
DataStatistics::Dat
aStatisticsAdviser indexserver.ini -> [cache] -> datastatisticsadvisercache_enabled
Cache::createNew indexserver.ini -> [cache] -> datastatisticsadviser
Entry
Alternatively it can be configured in SAP HANA Cockpit (SAP Note 2800006) -> "Recommendations" -> "C
DataStatistics::Dat
onfigure" -> "Data Statistics Advisor".
aStatisticsAdviser
Cache::insert
DataStatistics::Dat
aStatisticsAdviser
Cache::lookup
DataStatistics::CS Us These call stacks are related to the maintenance of the column store access statistics cache (SAP Note 25022
AccessStatisticsCa er 56).
che::insert
See SAP Notes 2785533 and 2970271 and make sure that the column store access statistics cache is only act
DataStatistics::CS
ivated as short and targeted as possible. The following parameter controls the behavior (true -> activated):
AccessStatisticsCa
che::lookup indexserver.ini -> [cs_access_statistics] -> collection_enabled
AttributeEngine::
AccessCounterHel
pers::incrementCS
AccessStatisticsCa
cheValue
AttributeEngine::
AccessCounterHel
pers::incrementCS
AccessStatisticsCa
cheValueForKeyIn
ternal
AttributeEngine::
AccessCounterHel
pers::insertIntoCS
AccessStatisticsCa
che

AttributeEngine:: Us This call stack is related to a specific optimize compression phase (SAP Note 2112604) that can be quite exp
RowOrderOptimiz er ensive, using a high amount of concurrent JobWorkers. The parallelism can be controlled by the following S
er::determineValu AP HANA parameter:
eIdCounts
AttributeEngine indexserver.ini -> [optimize_compression] -> row_order_optimizer_threads
::SortRestRang
eJob::run See SAP Notes 2222250 for workload settings in general and SAP Note 2600030 for workload parameter re
commendations in particular to make sure that the concurrency settings (in particular row_order_optimize
r_threads, max_concurrency_hint) are in place based on the SAP best practices.

On top of the parallelism itself also NUMA effects can be triggered that can result in an overall increased us
er CPU consumption, looking similar to a slow-down of the CPU speed. In this case you need to make sure t
hat NUMA and also infrastructure details like firmware, BIOS and Intel specification updates are configure
d optimally (SAP Notes 2470289, 2632231). As a temporary workaround you can set the following paramet
er:

indexserver.ini -> [optimize_compression] -> numa_awareness_no_steal_jobs = true

Be aware that this parameter must only be set if there are severe NUMA effects beyond the actual SortRestR
angeJob activities.

Catalyst::Hash Us A high CPU consumption in Catalyst::HashMap coding can be caused by other threads getting stuck in glob
Map::Stripe:res er al deadlock detection (module TransactionManager::WaitGraph::checkGlobalDeadlock) for a long time, e.g.
erveElement due to waits for the "Transaction object latch" mutex (SAP Note 1999998). In the worst case (i.e. in case of a
Catalyst::Hash n internal deadlock with no progress) it is required to restart SAP HANA.
Map::Stripe::re
size
Catalyst::HashMa
p::reserve
ptime::Transactio
n::start_logging
ptime::Transactio
n::set_write_trans
ptime::Query::lock
_table_globally
MemoryManag Us A high CPU consumption in MemoryManager::StripedAllocator::allocateStriped is typically caused by facto
er::StripedAlloc er rs like:
ator::allocateSt
• Many client connections
riped
• Small ABAP statement cache size
ltt::allocator_stati
• global.ini -> [memorymanager] -> composite_statistics_striping set to NUMA
stics::allocator_sta
tistics In this case you the following parameter in order to deactivate internal striping:
Execution::Contex
tAllocator::initImp global.ini -> [memorymanager] -> statistics_type = 7
licitStatementMe
See SAP Note 1999997 -> "How can allocations in Pool/Statistics be analyzed and optimized?" for further d
moryBooking
etails about these aspects.
Execution::Statem
entContext::initIm
plicitStatementMe
moryBooking
MemoryManager::
PoolAllocator::Poo
lAllocator
MemoryManager::
PoolAllocator::cre
ateCompositeAlloc
ator
MemoryManager::
PoolAllocator::cre
ateSubAllocatorIn
ternal
ltt::allocator_stati Us Heuristic reordering in context of query rewrites can introduce significant overhead if on lower layers expen
stics::getCountAll er sive operations happen repeatedly, e.g. counting M_CONTEXT_MEMORY entries repeatedly like in the left
oc example call stack. You can check if disabling heuristic reordering with hint OPTIMIZATION_LEVEL(RUL
ltt::allocator_stati E_BASED) (SAP Note 2142945) can help to reduce the overhead. The SAP HANA Mini Checks (SAP Notes
stics::isActive 1969700, 1999993) were extended by this hint for that purpose.
ltt::allocator_stati
stics::isVisible
Diagnose::RowCo
untFunctor::execu
te
Diagnose::Statistic
sWrapper::travers
eNodesRecursiveI
mpl
Diagnose::Statistic
sWrapper::travers
eNodesRecursiveI
mpl
Diagnose::Statistic
sWrapper::travers
eNodesRecursiveI
mpl
Diagnose::Statistic
sWrapper::travers
eNodesRecursiveI
mpl
Diagnose::Statistic
sWrapper::travers
eNodesRecursive
TypedStatisticsWr
apper__M_CONT
EXT_MEMORY::t
raverseNodesImpl
Diagnose::Statistic
sWrapper::travers
eNodesBase
Diagnose::Statistic
sRegistry::traverse
Nodes
ptime::StatisticsM
onitorHandle::get
RowCountEstimat
ion
ptime::StatisticsA
PIWrapper::getMo
nitorViewRowCou
nt
ptime::TableSizeE
stimator::fetchHis
togramOfMonitor
View
ptime::TableSizeE
stimator::Estimate
ptime::qo_size_es
timation::estimate
_one_rel
ptime::qo_size_es
timation::head_es
timation
ptime::qo_size_es
timation::rel_esti
mation
ptime::qo::heuristi
c::heuristic_reord
er_join
ptime::qo::heur
istic::heuristic_
reorder
ptime::qo_Heurist
icJoinReorder::ap
ply
ptime::qo_Rule::c
heckAndApply
ptime::qo_Normal
izer::applyRule
ptime::qo_Normal
izer::rewrite

ptime::Monitor:: Us This call stack is linked to queries on monitoring view M_MERGED_TRACES. SAP Note 2088971 -> "M_
Monitor er MERGED_TRACES" describes possibilities to reduce the data available in the underlying trace files.
ptime::MergedT
racesMonitor::I
terator::fetch
ptime::Monitor_sc
an::fetch_with_itr
ptime::Monitor_sc
an::do_fetch
ptime::Table::fetc
h

Configuration::Par Us Large call stacks with repeated occurrences of these modules are linked to an inefficient evaluation of neste
ameterBase::id er d expressions in context of HEX (SAP Note 2570371). The following options exist to improve the situation:
Configuration::Par
• Upgrade to SAP HANA 2.0 >= SPS 05 where the problem is fixed.
ameter::value
• Disable HEX via hint or globally (SAP Note 2570371 -> "How can the usage of HEX be controlled"?).
ptime::qo::hex::En
• Reduce the amount of used expressions, e.g. simplify complex / nested CASE conditions
umerator::isHexC
ompatibleStorage
Type
ptime::qo::hex::En
umerator::hasUns
upportedColumnI
nUse
ptime::qo::hex::Ch
eckExpHexSuppor
tVisitor::operator
ptime::qo_Visitor
UtilImpl::check_al
l_internal
ptime::qo_Visit
orUtil::qo_chec
k_all_exprs_pr
e_order
ptime::qo::hex::
CheckExpHexS
upportVisitor::c
heckExpression
Diagnose::Statistic Us Expensive read write lock maintenance in context of AbapSysTimeZoneSingleCache is related to lock ptime:
sData::addChild er :AbapTimeZone (SAP Note 1999998). It can e.g. happen in context of huge aggregations via STRING_AGG.
Diagnose::Statistic A fix is available with SAP HANA >= 2.00.048.00.
sData::StatisticsD
ata
Synchronization::
ReadWriteLockSta
tistics::ReadWrite
LockStatistics
Synchronization::
ReadWriteLock::R
eadWriteLock
ptime::Env::Ab
apSysTimeZone
SingleCache::A
bapSysTimeZon
eSingleCache

AttributeEngine:: Us This call stack indicates index lookups in context of HEX (SAP Note 2570371). The following options may re
MainInvertedInde er duce the overhead:
xLookupReader::l
• SQL statement tuning (SAP Note 2000002)
ookup
• SAP HANA >= 2.00.046: Adjustment of predicate term sampling (SAP Note 2124112 -> "What kind of
hex::cs::IndexLoo
advanced parsing features exist?" -> "Predicate term sampling")
kupOp::run
• As a temporary workaround: Deactivation of HEX (SAP Note 2570371), optimally only on statement l
evel via hint NO_USE_HEX_PLAN (SAP Note 2142945)
rte::diag_topic::ch Us This call stack is linked to a trace activated on user level, e.g. a user-specific trace or an application_user set
ange_level_user er ting in contexts like the expensive statements trace. Make sure that you activate user related traces only as f
rte::diag_topic::co ocused and as short as possible. See SAP Note 2119087 for more details related to SAP HANA traces.
ntext_levels::set_l
evel
rte::diag_topic::co
ntext_levels::reset
Diagnose::Coun
tedUserTraceCo
ntext::operator
=
Diagnose::Coun
tedUserTraceCo
ntext::removeU
serTrace
Execution::Stat
ementContext::
setRootStateme
ntHash
Execution::Stat
ementContext::
setStatementHa
sh
Execution::Trac
eContext::~Trac
eContext

rte::diag_topic::ch
ange_level_user
rte::diag_topic::co
ntext_levels::set_l
evel
rte::diag_topic::co
ntext_levels::cont
ext_levels
Diagnose::Coun
tedUserTraceCo
ntext::addUser
Trace

TRexUtils::BitVect Us These call stacks indicate BitVector processing. A BitVector is a low level memory structure that is used in d
or::BitVector er ifferent contexts. The related heap allocator is Pool/BitVector. See SAP Note 1999997 -> "Pool/BitVector" f
TRexUtils::BitVect or typical reasons of heavy BitVector processing.
or::numSet_AVX5
12
TRexUtils::BitVect
or::resize
Configuration::Par Us When a thread waits for a packet to arrive on a network socket, it uses the epoll call with a timeout defined
ameterBase::id er with the following parameter (default: 100, unit: ms):
ptime::TcpReceive
r::doWork indexserver.ini -> [session] -> event_wait_timeout
ptime::TcpReceive
r::run When no packet arrived within the timeout period, some checks are done and with SAP HANA >= 2.00.059
TrexThreads::Pool .07 and >= 2.00.067 among others the current value for the following parameter is determined (default: 50
Thread::run 0, unit: ms):
TrexThreads::Pool
indexserver.ini -> [session] -> tcp_receiver_idle_check_threshold
Thread::run
Execution::Thread
With SAP HANA 2.0 SPS 05 the parameter check is expensive to a certain extent and in cases where many t
::staticMainImp
hreads are active in network receive, the regular 100 ms check of this parameter can become dominant in te
Execution::pthrea
rms of CPU consumption (issue number 311538). Thus, general contention scenarios blocking many thread
dFunctionWrappe
s (e.g. ConsistentChangeLock, slow commits) can over time result in a CPU bottleneck when more and more
r
threads get stuck in network receive.
start_thread
__GI___clone As a workaround the event_wait_timeout parameter can be increased so that the tcp_receiver_idle_check_
threshold parameter access happens less frequently, e.g. to 1000 ms:
gstack (normal cal
l stacks may not w indexserver.ini -> [session] -> event_wait_timeout = 1000
ork):
An optimization is available with SAP HANA >= 2.00.059.10 but the Configuration::ParameterBase::delega
set<Configurati
te call stack can still be responsible for CPU overhead. In this case eliminate the root cause for the high num
on::ParameterI
ber of threads piling up (e.g. avoid starting ABAP application servers while SAP HANA is still loading the co
D> () at smart_pt
lumn store).
r.hpp:319
smart_ptr () at sm
art_ptr.hpp:527
id () at Basis/Confi
guration/impl/Par
ameter.cpp:138
value () at Basis/C
onfiguration/Para
meter.hpp:241
doWork () at ptim
e/session/tcp_re
ceiver.cc:272
run () at ptime/ses
sion/tcp_receiver.
cc:492

Configuration::
ParameterBase:
:delegate
Configuration::Del
egatingParameter
ValueManager::ge
tValue
ptime::TcpReceive
r::doWork

10. Is it possible to limit the amount of consumed CPU?

See SAP Note 2222250 ("How can workload management be configured for CPU and threads?") for an overview of parameters
and settings that can be used to influence parallelism and CPU consumption.

11. How can I determine physical CPU details on SAP HANA side?
General CPU information can be retrieved via monitoring view M_HOST_INFORMATION or with the related command SQL:
"HANA_Hosts_Overview" (SAP Note 1969700).
Example:
 ---------------------------------------------------------
|CPU_DETAILS|CPU_MHZ|CPU_MODEL |
---------------------------------------------------------
|60*2294MHz | 2294|Intel Xeon CPU E7-8880 v3 @ 2.30GHz|
---------------------------------------------------------

If hyperthreading is used, the number of logical CPU cores is included in brackets after the CPU number (e.g.
"60(120)*2294MHz").
Further details like the number of CPUs per socket can't be determined via SQL. On SAP HANA side you can alternatively use
the following hdbcons command (SAP Note 2222218):

hdbcons 'jexec info'

Example:

Using 2 numa nodes

numa_features: 1
max_concurrency: 20 (cfg=0, dyn=20)
max_concurrency_hint: 20 (cfg=0)
min_concurrency_hint: 10 (cfg=0)
0 statement limiters
System info:
2 CPU sockets, 20 physical cores, 20 logical cores, 20 total possibles cores
Using global restriction to a subset of cores: [11111111 11111111 1111]
Numa node 0: usable cores=10
workers: [11111111 11000000 0000]
others: [00000000 00000000 0000]
full: [11111111 11000000 0000]
max_concurrency: 10, dyn=10
Numa node 1: usable cores=10
workers: [00000000 00111111 1111]
others: [00000000 00000000 0000]
full: [00000000 00111111 1111]
max_concurrency: 10, dyn=10
[OK]

In this example you can see that 20 physical cores are available in two sockets, so each socket contains 10 cores.
For more information related to NUMA in SAP HANA environments see SAP Note 2470289.

12. How can I trigger automatic follow-up actions in case of a high CPU consumption?
For an optimal root cause analysis it is important to have as much analysis data available for times of high CPU consumption
as possible. In order to support the collection of analysis data (e.g. via runtime dump, SAP Note 2400007) you can schedule
SAP HANASitter (SAP Note 2399979).
Example:

Command Details

python hanasitter.py -cpu Standard behavior and additional trigger of actions when the CPU consumption is at least 95 % during 5 c
0,5,2,95 hecks with an interval of 2 seconds

13. Is the CPU consumption shown by SAP HANA always correct?

In most cases you can rely on the displayed CPU figures, but for the following reasons it can be wrong:

Scenario SAP Details

Not
e
 Wrong system CPU number 222 If the system CPU values are identical to the user CPU values, you most see wrong values. In this
s in load history 2110 case you can check the real system CPU consumption in a different way, e.g. via SQL: "HANA_R
esources_CPUAndMemory_History" (SAP Note 1969700).

NULL values for system CP 252 Due to a bug in the view definition of HOST_RESOURCE_UTILIZATION_STATISTICS NULL v
U in HOST_RESOURCE_U 7122 alues can be reported for system CPU consumption. This problem is fixed with SAP HANA >= 2.
TILIZATION_STATISTICS 00.020. Alternatively you can adjust the view definition based on the instructions in the SAP Not
e.

Estimated CPU time measur Starting with SAP HANA 1.0 SPS 12 CPU times are estimated rather than computed, so there can
ement be some deviations from the actual CPU consumption.

14. Can the available CPU resources be limited on OS level?

In SAP HANA environments you should generally avoid to limit CPU resources on OS level, because this limitation is often
intransparent and certain SAP HANA tasks can significantly suffer in case of insufficient CPU resources.
Examples for undesired CPU limitations introduced on OS level:

Feature SAP Details

Note

CPU power 25196 If a CPU power-save mode is active, the frequency of CPUs is reduced dynamically in case of reduced workload.
-save mode 30

CPU foldin 2205 CPU folding dynamically deactivates CPU in case of reduced workload. The related cpufoldd is part of the pseries
g 917 -energy package available on IBM on Power systems. You can use the following OS command to make sure that i
2292 t is not installed:
690
rpm -q pseries-energy

15. Is it recommended to use hyperthreading in SAP HANA environments?

In general hyperthreading can be used in SAP HANA environments, but on Intel based systems with more than 8 CPUs it
should usually be deactivated. See SAP Note 2711650 for more information.

16. Which thread states and call stacks indicate CPU consumption?
Only thread state "Running" represents CPU consumption. All other thread states indicate waiting (e.g. for locks or network
communications). You can use SQL: "HANA_Threads_ThreadSamples_FilterAndAggregation" (THREAD_STATE =
'Running') available via SAP Note 1969700 to display thread activities consuming CPU.
SAP Note 2929927 describes an exception where a network activity with the following call stack is accounted as running
instead of "Network Poll", thus confusing the JobWorker concurrency calculations:

__poll_nocancel
ldap_int_poll
ldap_int_tls_start
ldap_start_tls_s
Authentication::LDAP::Session::bind
Authentication::LDAP::SessionFactory::createNewSession

Call stacks working in __libc_recv are also mapped to thread state "Running" although the CPU consumption is usually minor
(issue number 290193).
As described in SAP Note 1999998, there can be some "hidden" lock waits of threads in status "Running". Related call stacks
(SAP Note 2313619) contain:
• __GI___pthread_mutex_lock
• __GI___sched_yield
• __lll_lock_wait
• _L_lock_1008
• pthread_mutex_lock
• pthread_mutex_lock_internal
• ptime::Futex::lock
• Synchronization::impl::SpinLock::lock
 • Synchronization::impl::SpinLock::_lock
• Synchronization::SystemMutex::lock
For threads created outside of the normal SAP HANA context the thread state "Unknown" is used that can mean both CPU
consumption or wait states. See SAP Note 2114710 for more details.
In call stacks (SAP Note 2313619) the first module is of interest. It is indicating CPU consumption as long as it doesn't indicate
an idle system call (e.g. syscall, do_futex_wait, io_queue_run, __nanosleep_nocancel, pthread_cond_wait) or network
operations (e.g. __GI_epoll*, __GI___poll*, __epoll*, __poll*, __read*, __write*). You can use SQL:
"HANA_Threads_Callstacks_Current" or SQL: "HANA_Threads_Callstacks_History" (ACTIVITY_TYPE = 'CPU') available
via SAP Note 1969700 in order to filter for call stacks indicating CPU consumption.

17. How can the CPU performance of the SAP HANA database server be tested and compared?
The performance of a CPU depends on various factors like the CPU type, the system load, the NUMA configuration, the power-
saving scenario or the activation of special features like mitigations for Spectre / Meltdown vulnerabilities. Thus, it can
sometimes be useful to measure certain basic CPU operations on SAP HANA side and compare it with usual expectations and
between systems. For that purpose test commands are available via SAP Note 1969700:
• SQL: "HANA_Tests_ArithmeticOperations": Executes arithmetic operations
• SQL: "HANA_Tests_StringOperations": Executes string operations
The runtimes of these commands can subsequently be evaluated with SQL: "HANA_Tests_Results" (SAP Note 1969700).
Example:

-------------------------------------------------------------------------------------------
--
|TEST_STATEMENT |EXECUTIONS|EXPECTATION |RESULT
|
-------------------------------------------------------------------------------------------
--
|SQL: "HANA_Tests_ArithmeticOperations"| 5|<= 500 ms / execution|226 ms / executio
n|
|SQL: "HANA_Tests_StringOperations" | 5|<= 1000 ms / execution|506 ms / executio
n|
-------------------------------------------------------------------------------------------
--

18. What are reasons for slowness in combination with a limited CPU consumption?
The following scenarios are most typical explanation why SAP HANA can be slow while at the same time the available CPU
resources are utilized only to a minor extent:
• Transactional or internal lock contention (SAP Note 1999998)
• Restrictive CPU workload settings like affinity, default_statement_concurrency_limit, max_concurrency or
max_concurrency_hint (SAP Note 2222250)
• NUMA issues (SAP Note 2470289)
• Memory management bottlenecks

19. What can be done in case of high CPU consumption in SAP HANA Cloud environments?
In case of high CPU consumption in SAP HANA Cloud (SHC) environments, e.g. reported via HDBCPUUsage alert, you can
proceed according to SAP Note 3123732 in order to analyze and optimize the CPU utilization or adjust the sizing.

Keywords
Performance
Alert
Host CPU Usage
threshold_cpu_time
M_SERVICE_THREADS.CPU_TIME_SELF
M_SERVICE_THREADS.CPU_TIME_CUMULATIVE
M_EXPENSIVE_STATEMENTS.CPU_TIME
CPUTimeMeasurement

Attributes
Key Value

Other Components SAP HANA > SAP HANA Database (HAN-DB)

Products
Products

SAP BusinessObjects BI 4.1, add-on for SAP Lumira

SAP HANA, platform edition all versions

SAP Lumira 1.0, server version for the SAP BusinessObjects BI platform

SAP Lumira, desktop edition 1.0

SAP Lumira, server edition 2.0

This document refers to

SAP Component Title
Note/
KBA

2915232 HAN-DB- HANA timer fallback alert on AWS due to incorrect configuration
MON

2823760 BC-OP-LNX- HANA Timer Fallback on VMware based servers

ESX

2800030 HAN-DB- FAQ: SAP HANA Kernel Profiler

PERF

2753342 BC-OP-LNX- BUG: soft lockup - CPU#x stuck for xxs!

SUSE

2600076 HAN-DB FAQ: SAP HANA Inverted Individual Indexes

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2570371 HAN-DB- FAQ: SAP HANA Execution Engine (HEX)

ENG

2519630 HAN-DB Recommendation: Check whether power save mode is active

2502256 HAN-DB- FAQ: SAP HANA Caches

ENG

2470289 HAN-DB FAQ: SAP HANA Non-Uniform Memory Access (NUMA)

2400007 HAN-DB- FAQ: SAP HANA Runtime Dumps

PERF

2399993 HAN-DB FAQ: SAP HANA Fast Data Access (FDA)

2399979 HAN-DB- How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter
PERF

2380176 HAN-DB FAQ: SAP HANA Database Trace

2313619 HAN-DB- How-To: Generating and Evaluating SAP HANA Call Stacks
MON

2222276 HAN-DB How-To: Activating SAP HANA Memory Fencing

2222250 HAN-DB- FAQ: SAP HANA Workload Management

PERF

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2222200 HAN-DB FAQ: SAP HANA Network

2222110 HAN-DB- FAQ: SAP HANA Load History

PERF
2195103 BI-LUM-SRV SAP Lumira server version for SAP HANA and SAP Lumira Cloud are retiring

2186744 HAN-DB FAQ: SAP HANA Parameters

2169283 HAN-DB- FAQ: SAP HANA Garbage Collection

ENG

2159014 HAN-DB- FAQ: SAP HANA Security

SEC

2142945 HAN-DB- FAQ: SAP HANA Hints

PERF

2135965

2124112 HAN-DB- FAQ: SAP HANA Parsing

PERF

2119087

2116157 HAN-DB FAQ: SAP HANA Consistency Checks and Corruptions

2114710 HAN-DB- FAQ: SAP HANA Threads and Thread Samples

MON

2112604 HAN-DB- FAQ: SAP HANA Compression

ENG

2106466

2088971 HAN-DB- How-To: Controlling the Amount of Records in SAP HANA Monitoring Views
MON

2081591 HAN-DB- FAQ: SAP HANA Table Distribution

PER

2057046 HAN-DB- FAQ: SAP HANA Delta Merges

ENG

2044468 HAN-DB- FAQ: SAP HANA Partitioning

ENG

2000002 HAN-DB- FAQ: SAP HANA SQL Optimization

PERF

1999998 HAN-DB- FAQ: SAP HANA Lock Analysis

PERF

1999997 HAN-DB FAQ: SAP HANA Memory

1999993 HAN-DB- How-To: Interpreting SAP HANA Mini Check Results

MON

1999880 HAN-DB-HA FAQ: SAP HANA System Replication

1909670 HAN-DB- How to handle HANA Alert 5: ‘Host CPU Usage'

MON

1813020 HAN-DB How to generate a runtime dump on SAP HANA

784391 BC-OP-LNX SAP support terms and 3rd-party Linux kernel drivers

618104 BC-OP-LNX "sapsysinfo" - Collecting system information on Linux

3359552 BC-OP-LNX Linux: Supportability considerations when running 3rd-party software

3123732 HAN-CLS- Analyze Reason for Alert HDB CPU Usage

HC

2929927 HAN-DB- When Jobworkers Are Occupied to Make LDAP Connection But Fail,Then Other Operations May Not Function
ENG Properly

2859234 BC-OP-LNX- SAP HANA Timer Fallback alert on SLES 12 SP3

SUSE

2857553 HAN-DB Overall System Performance Degrading After HANA or OS Kernel Upgrade

2793751 HAN-DB High CPU Consumption With Many Threads Waiting in hex::impl::WsAllTasksRefState::waitForAllTasks

2791572 HAN-DB Performance Degradation Because of Missing VDSO Support in Microsoft Azure VM
2785840 HAN-DB Degraded Performance of the Select Statement Running on the Table Heavily Affected by DML Queries

2753418 HAN-DB Potential Performance Degradation Due to Timer Fallback

2748836 HAN-DB Indexserver Busy With High CPU on Catalyst::TransactionBrokerClient::acquireSnapshotTimestamp

2711650 HAN-DB SAP HANA on Intel server systems with more than 8 CPU´s / sockets

2632231 HAN-DB SAP HANA and Non-Uniform Memory Access (NUMA)

2580435 HAN-DB Memory Leak in Pool/RowEngine/Session

2547543 HAN-DB High CPU Used by DataContainer::VarSizeEntryUserDataHandler::getUnusedEntryFromFreeList

2527122 HAN-DB- Values in

MON _SYS_STATISTICS.HOST_RESOURCE_UTILIZATION_STATISTICS.TOTAL_CPU_SYSTEM_TIME_DELTA
are NULL

2516807 HAN-DB jeReadIndex on Compressed Column of Index Type FULL is Slow

2385992 HAN-DB SAP HANA DB: Hanging Connection Requests When Trying to Open Secure Connections to HANA in Parallel

2292690 HAN-DB SAP HANA DB: Recommended OS settings for RHEL 7

2205917 HAN-DB SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12

1993128 HAN-DB SAP HANA: column store table unloads and unloading behavior of Memory Objects Container

1975448 HAN-DB Problems due to Late Materialization and chunk-wise Data Processing

1969700 HAN-DB SQL Statement Collection for SAP HANA

1865554 BW-BEX- MDX: Access type F4 help / improved error update

OT-MDX

SAP HANA Troubleshooting and Performance Analysis Guide

Kernel Probes

This document is referenced by

SAP Note/ Component Title
KBA

2222200 FAQ: SAP HANA Network

2999990 HAN-DB-PERF How-To: SAP HANA Performance Analysis

3273961 HAN-DB-PERF The Orphan JobWorker threads utilize 99% CPU.

3257190 HAN-DB-PERF High CPU usage caused by column store access statistics cache

3065607 CA-LT-MC Performance tips & tricks for SAP S/4HANA Migration Cockpit: Migrate Data Using Staging
Tables

3011192 HAN-DB-PERF Analysing High User CPU in SAP HANA DB

3024031 HAN-DB-PERF CPU spikes every hour.

2915232 HAN-DB-MON HANA timer fallback alert on AWS due to incorrect configuration

2800008 HAN-DB-ENG- FAQ: SAP HANA Fulltext Indexes

TXT

2975817 HAN-DB-PERF Locks caused by enabled Data Statistics Adviser Cache

2970921 HAN-DB-MON HANA CPU spikes to 80-100%

2952074 HAN-DB High CPU Caused by Optimize Compression

2313619 HAN-DB-MON How-To: Generating and Evaluating SAP HANA Call Stacks

2092196 HAN-DB-ENG How-To: Terminating Sessions in SAP HANA

2918507 HAN-DB-MON HANA CPU usage is higher than expected at intermittent intervals based on configured CPU
Workload Parameters

2399990 HAN-DB How-To: Analyzing ABAP Short Dumps in SAP HANA Environments

2908925 HAN-DB-HA Takeover slow during optimized compression phase

1999997 HAN-DB FAQ: SAP HANA Memory

2502256 HAN-DB-ENG FAQ: SAP HANA Caches

2843506 HAN-DB-PERF 100% CPU usage on Time Stamp Counter (TSC)

1616401 BC-UPG-TLS-TLA Parallelism in the Upgrades, EhPs and Support Packages implementations

2785814 HAN-DB-BAC HANA high CPU utilization caused by large backup catalog

2699939 HAN-DB SAP HANA Emergency Suitcase

2399993 HAN-DB FAQ: SAP HANA Fast Data Access (FDA)

2380176 HAN-DB FAQ: SAP HANA Database Trace

2498739 HAN-DB-PERF How-To: Analyzing Runtime Dumps with SAP HANA Dump Analyzer

2000000 HAN-DB-PERF FAQ: SAP HANA Performance Optimization

2600030 HAN-DB Parameter Recommendations in SAP HANA Environments

2494336 HAN-DB Cannot turn off CPU Measurement in HANA 1.0 SPS12

2142945 HAN-DB-PERF FAQ: SAP HANA Hints

2112604 HAN-DB-ENG FAQ: SAP HANA Compression

2470289 HAN-DB FAQ: SAP HANA Non-Uniform Memory Access (NUMA)

2400007 HAN-DB-PERF FAQ: SAP HANA Runtime Dumps

2044468 HAN-DB-ENG FAQ: SAP HANA Partitioning

2400024 HAN-DB-MON How-To: SAP HANA Administration and Monitoring

2399979 HAN-DB-PERF How-To: Configuring automatic SAP HANA Data Collection with SAP HANASitter

2222218 HAN-DB FAQ: SAP HANA Database Server Management Console (hdbcons)

2000002 HAN-DB-PERF FAQ: SAP HANA SQL Optimization

2222250 HAN-DB-PERF FAQ: SAP HANA Workload Management

1999998 HAN-DB-PERF FAQ: SAP HANA Lock Analysis

2222110 HAN-DB-PERF FAQ: SAP HANA Load History

2116157 HAN-DB FAQ: SAP HANA Consistency Checks and Corruptions

2186744 HAN-DB FAQ: SAP HANA Parameters

2180165 HAN-DB FAQ: SAP HANA Expensive Statements Trace

2169283 HAN-DB-ENG FAQ: SAP HANA Garbage Collection

2154870 HAN-DB How-To: Understanding and defining SAP HANA Limitations

2114710 HAN-DB-MON FAQ: SAP HANA Threads and Thread Samples

1999993 HAN-DB-MON How-To: Interpreting SAP HANA Mini Check Results

2000003 HAN-DB-ENG FAQ: SAP HANA