PostgreSQL regression tests hanging on Windows? Check path depth.

I just confirmed the cause an extremely weird problem that’s been frustrating me for days. I want to share it so nobody else has to waste their time on this.

It appears that – at least on my build machine, a Windows 7 SP1 x64 box with Windows SDK 7.1, Visual Studio 2010 Express SP1 and Visual Studio Express 2012 on it – vcregress check will hang indefinitely with a postgres.exe process sitting at 100% cpu if the regression tests are run in a path that is too deep. This seems to happen with both x64 and x86 builds.

git.exe seems to have a similar problem, where a git clean -fdx in a deep directory tree will sit at 100% cpu forever, making no progress.

In both git.exe‘s and postgres.exe's cases, Process Monitor shows a steam of QueryNameInformationFile events with result BUFFER OVERFLOW. QueryNameInformationFile is the IRP_MJ_QUERY_INFORMATION operation of ZwQueryInformationFile as documented in MSDN here. It’s a kernel-level operation.

I’m yet to determine the root cause of the issue. To work around the problem, build in a shallower directory tree.

I’ve included a bunch of details after the cut, primarily to help anyone else with this problem find this post.

This works:

	cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\BT\release\SL_OS\windows\TA\x86\TO\2008\src\tools\msvc
	"perl" vcregress.pl check
============== creating temporary installation        ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 57532 with PID 100
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test tablespace               ... ok

This, with the exact same code, fails, hanging indefinitely on test tablespace:

	cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\tools\msvc
	"perl" vcregress.pl check
============== creating temporary installation        ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 57532 with PID 128
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test tablespace               ... 

The postgres.exe backend the test is running on sits at 100% cpu (or rather, 50% because it’s 100% of one core on a dual core box). It cannot be terminated via Task Manager or Process Explorer – End Task appears to succeed without error, but doesn’t actually kill the process. Neither does taskkill.exe even with the /F flag. Only a reboot seems to get rid of it. Sometimes subsequent attempts to kill it fail with a “permission denied” error.

When they’re hung, a backtrace from Process Explorer shows, for a 32-bit Pg running on a 64-bit host:

ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntdll.dll!NtCreateFile+0xa
wow64.dll!Wow64EmulateAtlThunk+0xe697
wow64.dll!Wow64SystemServiceEx+0xd7
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!NtCreateFile+0x12
kernel32.dll!CreateFileW+0x4a
kernel32.dll!CreateFileA+0x36
postgres.exe!pgwin32_open+0xbc
postgres.exe!BasicOpenFile+0x1e
postgres.exe!GetNewRelFileNode+0xfa
postgres.exe!heap_create_with_catalog+0x1df
postgres.exe!DefineRelation+0x44e
postgres.exe!standard_ProcessUtility+0x425
postgres.exe!PortalRunUtility+0xa2
postgres.exe!PortalRunMulti+0x11b
postgres.exe!PortalRun+0x176
postgres.exe!exec_simple_query+0x3d1
postgres.exe!PostgresMain+0x5b5
postgres.exe!BackendRun+0x179
postgres.exe!SubPostmasterMain+0x31d
postgres.exe!main+0x1f4
postgres.exe!__tmainCRTStartup+0x122
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36

In Process Monitor, you see a long stream of QueryNameInformationFile events:

Date & Time:    16/01/2013 9:31:17 PM
Event Class:    File System
Operation:      QueryNameInformationFile
Result: BUFFER OVERFLOW
Path:   C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\data
TID:    4908
Duration:       0.0000015
Name:   \jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\da

with stack traces like this, again from a 32-bit Pg on a 64-bit host:

0              0xfffff88001420067      0xfffff88001420067      
1              0xfffff88001421329      0xfffff88001421329      
2              0xfffff8800141f6c7      0xfffff8800141f6c7      
3              0xfffff80002589636      0xfffff80002589636      
4              0xfffff800026b2bbe      0xfffff800026b2bbe      
5              0xfffff800026b2f2f      0xfffff800026b2f2f      
6              0xfffff800026b3947      0xfffff800026b3947      
7              0xfffff800026b3a4b      0xfffff800026b3a4b      
8              0xfffff800025fc3ef      0xfffff800025fc3ef      
9              0xfffff800025d8162      0xfffff800025d8162      
10             0xfffff800025d95f6      0xfffff800025d95f6      
11             0xfffff800025daefc      0xfffff800025daefc      
12             0xfffff800025e5b54      0xfffff800025e5b54      
13             0xfffff800022e1253      0xfffff800022e1253      
14      ntdll.dll       ZwCreateFile + 0xa      0x773c186a      C:\Windows\SYSTEM32\ntdll.dll
15      wow64.dll       whNtCreateFile + 0x10f  0x7388bff7      C:\Windows\SYSTEM32\wow64.dll
16      wow64.dll       Wow64SystemServiceEx + 0xd7     0x7387cf87      C:\Windows\SYSTEM32\wow64.dll
17      wow64cpu.dll    TurboDispatchJumpAddressEnd + 0x2d      0x73802776      C:\Windows\SYSTEM32\wow64cpu.dll
18      wow64.dll       RunCpuSimulation + 0xa  0x7387d07e      C:\Windows\SYSTEM32\wow64.dll
19      wow64.dll       Wow64LdrpInitialize + 0x429     0x7387c549      C:\Windows\SYSTEM32\wow64.dll
20      ntdll.dll       LdrpInitializeProcess + 0x17e4  0x773b4956      C:\Windows\SYSTEM32\ntdll.dll
21      ntdll.dll        ?? ::FNODOBFM::`string' + 0x29220      0x773b1a17      C:\Windows\SYSTEM32\ntdll.dll
22      ntdll.dll       LdrInitializeThunk + 0xe        0x7739c32e      C:\Windows\SYSTEM32\ntdll.dll
23      ntdll.dll       ZwCreateFile + 0x12     0x775700a6      C:\Windows\SysWOW64\ntdll.dll
24      KERNELBASE.dll  CreateFileW + 0x35e     0x7628c5ef      C:\Windows\syswow64\KERNELBASE.dll
25      kernel32.dll    CreateFileWImplementation + 0x69        0x76a83f86      C:\Windows\syswow64\kernel32.dll
26      kernel32.dll    CreateFileA + 0x37      0x76a853e4      C:\Windows\syswow64\kernel32.dll
27      postgres.exe    pgwin32_open + 0xbc, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp
\src\port\open.c(77)     0x13eba5c       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src
\test\regress\tmp_check\install\bin\postgres.exe
28      postgres.exe    BasicOpenFile + 0x1e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\x
p\src\backend\storage\file\fd.c(560)     0x12eec6e       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
29      postgres.exe    GetNewRelFileNode + 0xfa, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_
os\xp\src\backend\catalog\catalog.c(578) 0x11a836a       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
30      postgres.exe    heap_create_with_catalog + 0x1df, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg
_target_os\xp\src\backend\catalog\heap.c(1073)   0x11adf8f       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH
\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
31      postgres.exe    DefineRelation + 0x44e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\commands\tablecmds.c(636)        0x11fb98e       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
32      postgres.exe    standard_ProcessUtility + 0x425, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\utility.c(535)     0x130f425       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
33      postgres.exe    PortalRunUtility + 0xa2, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1191)     0x130d022       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
34      postgres.exe    PortalRunMulti + 0x11b, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1320)      0x130d16b       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
35      postgres.exe    PortalRun + 0x176, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(815)    0x130da06       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
36      postgres.exe    exec_simple_query + 0x3d1, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(1053) 0x130ba61       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
37      postgres.exe    PostgresMain + 0x5b5, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(3969)      0x130c345       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
38      postgres.exe    BackendRun + 0x179, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(3987)        0x12c9459       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
39      postgres.exe    SubPostmasterMain + 0x31d, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(4491) 0x12cd08d       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
40      postgres.exe    main + 0x1f4, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\main\main.c(175)   0x123fc24       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
41      postgres.exe    __tmainCRTStartup + 0x122, f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c(554) 0x13f7366       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
42      kernel32.dll    BaseThreadInitThunk + 0xe       0x76a833aa      C:\Windows\syswow64\kernel32.dll
43      ntdll.dll       __RtlUserThreadStart + 0x70     0x77589ef2      C:\Windows\SysWOW64\ntdll.dll
44      ntdll.dll       _RtlUserThreadStart + 0x1b      0x77589ec5      C:\Windows\SysWOW64\ntdll.dll