PostgreSQL regression tests hanging on Windows? Check path depth.
I just confirmed the cause an extremely weird problem that’s been frustrating me for days. I want to share it so nobody else has to waste their time on this.
It appears that – at least on my build machine, a Windows 7 SP1 x64 box with Windows SDK 7.1, Visual Studio 2010 Express SP1 and Visual Studio Express 2012 on it – vcregress check will hang indefinitely with a postgres.exe process sitting at 100% cpu if the regression tests are run in a path that is too deep. This seems to happen with both x64 and x86 builds.
git.exe seems to have a similar problem, where a git clean -fdx in a deep directory tree will sit at 100% cpu forever, making no progress.
In both git.exe‘s and postgres.exe's cases, Process Monitor shows a steam of QueryNameInformationFile events with result BUFFER OVERFLOW. QueryNameInformationFile is the IRP_MJ_QUERY_INFORMATION operation of ZwQueryInformationFile as documented in MSDN here. It’s a kernel-level operation.
I’m yet to determine the root cause of the issue. To work around the problem, build in a shallower directory tree.
I’ve included a bunch of details after the cut, primarily to help anyone else with this problem find this post.
This works:
cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\BT\release\SL_OS\windows\TA\x86\TO\2008\src\tools\msvc "perl" vcregress.pl check ============== creating temporary installation ============== ============== initializing database system ============== ============== starting postmaster ============== running on port 57532 with PID 100 ============== creating database "regression" ============== CREATE DATABASE ALTER DATABASE ============== running regression test queries ============== test tablespace ... ok
This, with the exact same code, fails, hanging indefinitely on test tablespace:
cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\tools\msvc "perl" vcregress.pl check ============== creating temporary installation ============== ============== initializing database system ============== ============== starting postmaster ============== running on port 57532 with PID 128 ============== creating database "regression" ============== CREATE DATABASE ALTER DATABASE ============== running regression test queries ============== test tablespace ...
The postgres.exe backend the test is running on sits at 100% cpu (or rather, 50% because it’s 100% of one core on a dual core box). It cannot be terminated via Task Manager or Process Explorer – End Task appears to succeed without error, but doesn’t actually kill the process. Neither does taskkill.exe even with the /F flag. Only a reboot seems to get rid of it. Sometimes subsequent attempts to kill it fail with a “permission denied” error.
When they’re hung, a backtrace from Process Explorer shows, for a 32-bit Pg running on a 64-bit host:
ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a ntdll.dll!NtCreateFile+0xa wow64.dll!Wow64EmulateAtlThunk+0xe697 wow64.dll!Wow64SystemServiceEx+0xd7 wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d wow64.dll!Wow64SystemServiceEx+0x1ce wow64.dll!Wow64LdrpInitialize+0x429 ntdll.dll!RtlUniform+0x6e6 ntdll.dll!RtlCreateTagHeap+0xa7 ntdll.dll!LdrInitializeThunk+0xe ntdll.dll!NtCreateFile+0x12 kernel32.dll!CreateFileW+0x4a kernel32.dll!CreateFileA+0x36 postgres.exe!pgwin32_open+0xbc postgres.exe!BasicOpenFile+0x1e postgres.exe!GetNewRelFileNode+0xfa postgres.exe!heap_create_with_catalog+0x1df postgres.exe!DefineRelation+0x44e postgres.exe!standard_ProcessUtility+0x425 postgres.exe!PortalRunUtility+0xa2 postgres.exe!PortalRunMulti+0x11b postgres.exe!PortalRun+0x176 postgres.exe!exec_simple_query+0x3d1 postgres.exe!PostgresMain+0x5b5 postgres.exe!BackendRun+0x179 postgres.exe!SubPostmasterMain+0x31d postgres.exe!main+0x1f4 postgres.exe!__tmainCRTStartup+0x122 kernel32.dll!BaseThreadInitThunk+0x12 ntdll.dll!RtlInitializeExceptionChain+0x63 ntdll.dll!RtlInitializeExceptionChain+0x36
In Process Monitor, you see a long stream of QueryNameInformationFile events:
Date & Time: 16/01/2013 9:31:17 PM Event Class: File System Operation: QueryNameInformationFile Result: BUFFER OVERFLOW Path: C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\data TID: 4908 Duration: 0.0000015 Name: \jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\da
with stack traces like this, again from a 32-bit Pg on a 64-bit host:
0 0xfffff88001420067 0xfffff88001420067 1 0xfffff88001421329 0xfffff88001421329 2 0xfffff8800141f6c7 0xfffff8800141f6c7 3 0xfffff80002589636 0xfffff80002589636 4 0xfffff800026b2bbe 0xfffff800026b2bbe 5 0xfffff800026b2f2f 0xfffff800026b2f2f 6 0xfffff800026b3947 0xfffff800026b3947 7 0xfffff800026b3a4b 0xfffff800026b3a4b 8 0xfffff800025fc3ef 0xfffff800025fc3ef 9 0xfffff800025d8162 0xfffff800025d8162 10 0xfffff800025d95f6 0xfffff800025d95f6 11 0xfffff800025daefc 0xfffff800025daefc 12 0xfffff800025e5b54 0xfffff800025e5b54 13 0xfffff800022e1253 0xfffff800022e1253 14 ntdll.dll ZwCreateFile + 0xa 0x773c186a C:\Windows\SYSTEM32\ntdll.dll 15 wow64.dll whNtCreateFile + 0x10f 0x7388bff7 C:\Windows\SYSTEM32\wow64.dll 16 wow64.dll Wow64SystemServiceEx + 0xd7 0x7387cf87 C:\Windows\SYSTEM32\wow64.dll 17 wow64cpu.dll TurboDispatchJumpAddressEnd + 0x2d 0x73802776 C:\Windows\SYSTEM32\wow64cpu.dll 18 wow64.dll RunCpuSimulation + 0xa 0x7387d07e C:\Windows\SYSTEM32\wow64.dll 19 wow64.dll Wow64LdrpInitialize + 0x429 0x7387c549 C:\Windows\SYSTEM32\wow64.dll 20 ntdll.dll LdrpInitializeProcess + 0x17e4 0x773b4956 C:\Windows\SYSTEM32\ntdll.dll 21 ntdll.dll ?? ::FNODOBFM::`string' + 0x29220 0x773b1a17 C:\Windows\SYSTEM32\ntdll.dll 22 ntdll.dll LdrInitializeThunk + 0xe 0x7739c32e C:\Windows\SYSTEM32\ntdll.dll 23 ntdll.dll ZwCreateFile + 0x12 0x775700a6 C:\Windows\SysWOW64\ntdll.dll 24 KERNELBASE.dll CreateFileW + 0x35e 0x7628c5ef C:\Windows\syswow64\KERNELBASE.dll 25 kernel32.dll CreateFileWImplementation + 0x69 0x76a83f86 C:\Windows\syswow64\kernel32.dll 26 kernel32.dll CreateFileA + 0x37 0x76a853e4 C:\Windows\syswow64\kernel32.dll 27 postgres.exe pgwin32_open + 0xbc, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp \src\port\open.c(77) 0x13eba5c C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src \test\regress\tmp_check\install\bin\postgres.exe 28 postgres.exe BasicOpenFile + 0x1e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\x p\src\backend\storage\file\fd.c(560) 0x12eec6e C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_ TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 29 postgres.exe GetNewRelFileNode + 0xfa, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_ os\xp\src\backend\catalog\catalog.c(578) 0x11a836a C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_ TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 30 postgres.exe heap_create_with_catalog + 0x1df, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg _target_os\xp\src\backend\catalog\heap.c(1073) 0x11adf8f C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH \x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 31 postgres.exe DefineRelation + 0x44e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\commands\tablecmds.c(636) 0x11fb98e C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 32 postgres.exe standard_ProcessUtility + 0x425, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\utility.c(535) 0x130f425 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 33 postgres.exe PortalRunUtility + 0xa2, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1191) 0x130d022 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 34 postgres.exe PortalRunMulti + 0x11b, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1320) 0x130d16b C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 35 postgres.exe PortalRun + 0x176, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(815) 0x130da06 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 36 postgres.exe exec_simple_query + 0x3d1, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(1053) 0x130ba61 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 37 postgres.exe PostgresMain + 0x5b5, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(3969) 0x130c345 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 38 postgres.exe BackendRun + 0x179, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(3987) 0x12c9459 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 39 postgres.exe SubPostmasterMain + 0x31d, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(4491) 0x12cd08d C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 40 postgres.exe main + 0x1f4, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\main\main.c(175) 0x123fc24 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 41 postgres.exe __tmainCRTStartup + 0x122, f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c(554) 0x13f7366 C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe 42 kernel32.dll BaseThreadInitThunk + 0xe 0x76a833aa C:\Windows\syswow64\kernel32.dll 43 ntdll.dll __RtlUserThreadStart + 0x70 0x77589ef2 C:\Windows\SysWOW64\ntdll.dll 44 ntdll.dll _RtlUserThreadStart + 0x1b 0x77589ec5 C:\Windows\SysWOW64\ntdll.dll
Leave a Reply
Want to join the discussion?Feel free to contribute!