Redis server crash when adding lots of hash keys

[Kortenbach]

My Redis server is crashing at seemingly random points. Mostly when I'm setting a lot of Hash keys, but it could be something else. So far the server has crashed 5 times. Below is the report I copied from the server console:
[14676] 21 Mar 14:21:37.249 # fork operation complete
[14676] 21 Mar 14:21:37.250 * Background saving terminated with success
[14676] 21 Mar 14:21:51.051 * Starting automatic rewriting of AOF on 100% growth

[14676] 21 Mar 14:21:51.052 * Background append only file rewriting started by p
id 13768

=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 4.0.2.1
[13768] 21 Mar 14:21:51.130 # --- EXCEPTION_ACCESS_VIOLATION
[13768] 21 Mar 14:21:51.131 # --- STACK TRACE
redis-server.exe!UnhandledExceptiontHandler(d:\dev\github\redis\src\win32_intero
p\win32_stacktrace.cpp:185)(0x00000001, 0x00000000, 0x00000001, 0x1400497A2)
kernel32.dll!UnhandledExceptionFilter(d:\dev\github\redis\src\win32_interop\win3
2_stacktrace.cpp:185)(0x0012E530, 0x00000006, 0x00000000, 0x100000001)
ntdll.dll!EtwEventSetInformation(d:\dev\github\redis\src\win32_interop\win32_sta
cktrace.cpp:185)(0x0012F170, 0x0334D8F8, 0x140170830, 0x00000000)
ntdll.dll!C_specific_handler(d:\dev\github\redis\src\win32_interop\win32_stackt
race.cpp:185)(0x00130000, 0x0012FF90, 0x0012FF90, 0x778EF818)
ntdll.dll!RtlDecodePointer(d:\dev\github\redis\src\win32_interop\win32_stacktrac
e.cpp:185)(0x00130000, 0x7779DD88, 0x000127EC, 0x001C0080)
ntdll.dll!RtlUnwindEx(d:\dev\github\redis\src\win32_interop\win32_stacktrace.cpp
:185)(0x0012F170, 0x0012EC80, 0x00000000, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(d:\dev\github\redis\src\win32_interop\win32
stacktrace.cpp:185)(0x14019AEA4, 0x1400DA5E7, 0x14019AEA4, 0x00000002)
redis-server.exe!rioWriteBulkObject(d:\dev\github\redis\src\aof.c:796)(0x0000000
0, 0x0000000E, 0x00000002, 0x14007DB77)
redis-server.exe!rioWriteHashIteratorCursor(d:\dev\github\redis\src\aof.c:979)(0
x0012F56E, 0x0012F520, 0x00000002, 0x0012F5D0)
redis-server.exe!rewriteHashObject(d:\dev\github\redis\src\aof.c:998)(0x0012F5D0
, 0x0012F5C8, 0x14019AEA6, 0x7E112AE6CC0)
redis-server.exe!rewriteAppendOnlyFile(d:\dev\github\redis\src\aof.c:1100)(0x025
70000, 0x02570000, 0x00000000, 0x02570000)
redis-server.exe!QForkChildInit(d:\dev\github\redis\src\win32_interop\win32_qfor
k.cpp:350)(0x0039B1E0, 0x00000178, 0x003A78E0, 0x1401B8D18)
redis-server.exe!QForkStartup(d:\dev\github\redis\src\win32_interop\win32_qfork.
cpp:525)(0x0039B1E0, 0x00000000, 0x00000005, 0x00000006)
redis-server.exe!main(d:\dev\github\redis\src\win32_interop\win32_qfork.cpp:1284
)(0x00000000, 0x00000000, 0x1401BAD98, 0x00000000)
redis-server.exe!__scrt_common_main_seh(f:\dd\vctools\crt\vcstartup\src\startup
exe_common.inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
kernel32.dll!BaseThreadInitThunk(f:\dd\vctools\crt\vcstartup\src\startup\exe_com
mon.inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\vcstartup\src\startup\exe_common.
inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\vcstartup\src\startup\exe_common.
inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[13768] 21 Mar 14:21:51.153 # --- INFO OUTPUT
[14676] 21 Mar 14:22:04.652 # fork operation failed
[14676] 21 Mar 14:22:04.658 # Background AOF rewrite terminated by signal 1
[14676] 21 Mar 14:22:04.759 * Starting automatic rewriting of AOF on 110% growth

[14676] 21 Mar 14:22:04.759 * Background append only file rewriting started by p
id 14272

=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 4.0.2.1
[14272] 21 Mar 14:22:04.798 # --- EXCEPTION_ACCESS_VIOLATION
[14272] 21 Mar 14:22:04.798 # --- STACK TRACE
redis-server.exe!UnhandledExceptiontHandler(d:\dev\github\redis\src\win32_intero
p\win32_stacktrace.cpp:185)(0x00000001, 0x00000000, 0x00000001, 0x1400497A2)
kernel32.dll!UnhandledExceptionFilter(d:\dev\github\redis\src\win32_interop\win3

This repeats until I close the server

[tporadowski]

Can you observe and share some statistics about usage of Redis - in terms of number of keys, data ingestion rate, and so on? I'm looking for some clues how to reproduce this :)

[Kortenbach]

I could send you the exe that does the trick. I will need your mail address... Op 21 mrt. 2018 14:47 schreef "Tomasz Poradowski" <notifications@github.com

…

: Can you observe and share some statistics about usage of Redis - in terms of number of keys, data ingestion rate, and so on? I'm looking for some clues how to reproduce this :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATFzjxeyDNSjCkxhLraP0umu5pQd4Akgks5tglnVgaJpZM4Sz2Ur> .

[tporadowski]

tomasz@poradowski.com (it's also in my profile)

[tporadowski]

@Kortenbach please ping me here when you send it as it might go to spam by accident

[Kortenbach]

ping

[tporadowski]

Pong! As expected, landed in spam folder due to suspicious attachment ;)

[tporadowski]

@Kortenbach I've just sent you fixed version for testing via email. If it also works fine for you - I will commit the changes and release version 4.0.2.2. Thanks!

[Kortenbach]

Thanks ! Will test next Monday.

[wywdstc]

=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 4.0.2.2
[3468] 11 Apr 15:53:59.670 # --- EXCEPTION_ACCESS_VIOLATION
[3468] 11 Apr 15:53:59.675 # --- STACK TRACE
redis-server.exe!UnhandledExceptiontHandler(d:\dev\github\redis\src\win32_intero
p\win32_stacktrace.cpp:185)(0x00000000, 0x00000000, 0x00000001, 0x140049903)
KERNELBASE.dll!UnhandledExceptionFilter(d:\dev\github\redis\src\win32_interop\wi
n32_stacktrace.cpp:185)(0x00000003, 0x7FFB84541AD0, 0x00000000, 0x7FFB86FE8736)
ntdll.dll!memset(d:\dev\github\redis\src\win32_interop\win32_stacktrace.cpp:185)
(0x0013EE70, 0x7FA71903A9AA, 0x140170830, 0x7FA600000000)
ntdll.dll!_C_specific_handler(d:\dev\github\redis\src\win32_interop\win32_stackt
race.cpp:185)(0x00000000, 0x0013E3B0, 0x0013EE70, 0x0013EE70)
ntdll.dll!chkstk(d:\dev\github\redis\src\win32_interop\win32_stacktrace.cpp:185
)(0x00000001, 0x7FFB86FB0000, 0x00000000, 0x7FFB870E6204)
ntdll.dll!RtlRaiseException(d:\dev\github\redis\src\win32_interop\win32_stacktra
ce.cpp:185)(0x7FA69AED1614, 0x7FA64A505D17, 0x000003E0, 0x7FA6981ABD3A)
ntdll.dll!KiUserExceptionDispatcher(d:\dev\github\redis\src\win32_interop\win32
stacktrace.cpp:185)(0x03C07040, 0x7FA6D3E36290, 0x0013F240, 0x7FA6D3E1D923)
redis-server.exe!dictNext(d:\dev\github\redis\src\dict.c:600)(0x7FA6D3E36290, 0x
0000000F, 0x0013F20F, 0x0000039E)
redis-server.exe!rdbSaveObject(d:\dev\github\redis\src\rdb.c:787)(0x0013F240, 0x
00000004, 0x0013F240, 0x0013F240)
redis-server.exe!rdbSaveRio(d:\dev\github\redis\src\rdb.c:958)(0x001A6C30, 0x01A
A0000, 0x01EA2760, 0x00000D8C)
redis-server.exe!rdbSave(d:\dev\github\redis\src\rdb.c:1038)(0x01AA0000, 0x00000
005, 0x01AA0000, 0x00000010)
redis-server.exe!QForkChildInit(d:\dev\github\redis\src\win32_interop\win32_qfor
k.cpp:344)(0x00000000, 0x001AB500, 0x001A6AB0, 0x1401B8E70)
redis-server.exe!QForkStartup(d:\dev\github\redis\src\win32_interop\win32_qfork.
cpp:525)(0x001AB500, 0x00000000, 0x00000005, 0x00000006)
redis-server.exe!main(d:\dev\github\redis\src\win32_interop\win32_qfork.cpp:1284
)(0x140045300, 0x00000000, 0x1401BAEF8, 0x7FF5FFFF4000)
redis-server.exe!__scrt_common_main_seh(f:\dd\vctools\crt\vcstartup\src\startup
exe_common.inl:283)(0x140045328, 0x7FF5FFFF4000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(f:\dd\vctools\crt\vcstartup\src\startup\exe_com
mon.inl:283)(0x7FFB851213B0, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\vcstartup\src\startup\exe_common.
inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\vcstartup\src\startup\exe_common.
inl:283)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[3468] 11 Apr 15:53:59.990 # --- INFO OUTPUT
[3316] 11 Apr 15:54:00.299 # Error writing to socket fd

Hi,version 4.0.2.2 still has this bug。

[tporadowski]

@wywdstc can you provide some steps to reproduce it?

[wywdstc]

@tporadowski
Also when adding lots of hash key..

[tporadowski]

Could you please be a little more specific? :) What is "lots of hash key"? Are you adding new hashes or setting many entries in one or more hashes?