.NET Zone is brought to you in partnership with:

Sasha Goldshtein is a Senior Consultant for Sela Group, an Israeli company specializing in training, consulting and outsourcing to local and international customers.Sasha's work is divided across these three primary disciplines. He consults for clients on architecture, development, debugging and performance issues; he actively develops code using the latest bits of technology from Microsoft; and he conducts training classes on a variety of topics, from Windows Internals to .NET Performance. You can read more about Sasha's work and his latest ventures at his blog: http://blogs.microsoft.co.il/blogs/sasha. Sasha writes from Herzliya, Israel. Sasha is a DZone MVB and is not an employee of DZone and has posted 204 posts at DZone. You can read more from them at their website. View Full User Profile

P/Invoke with C++ bool Return Values

  • submit to reddit

I encountered an interesting gotcha today, which I thought would be interesting to share with you. The symptoms were the following: I wrote a simple C++ function exported from a DLL, and tried to invoke it using a P/Invoke wrapper. The C++ function returns a bool, and was declared as follows (the code was simplified for expository purposes):

extern "C" __declspec(dllexport) bool IsPrime(int n) { 
    if (n <= 1) return false; 
    if (n == 2) return true; 
    if (n % 2 == 0) return false; 
    for (int i = 3; i < n; ++i) { 
        if (n % i == 0) return false; 
    return true; 

The managed counterpart was:

static extern bool IsPrime(int n);

At runtime, for some values of n, the output would be inconsistent. Specifically, when n=0 or n=1, the managed wrapper would return true – although the C++ function clearly returns false. For other values, however, the wrapper returned the right values.

Next, I disassembled the C++ function, to find the following code responsible for the first branch:

cmp dword ptr[n], 1 
jg BranchNotTaken 
xor al, al 
jmp BailOut 

In other words, to return false, the function clears out the AL register (which is the lowest byte of the EAX register), and then returns. What of the rest of the EAX register? In the debug build, it’s likely to contain 0xCCCCCCCC, because of this part of the function’s prologue:

lea edi,[ebp-0CCh]  
mov ecx,33h  
mov eax,0CCCCCCCCh  
rep stos dword ptr es:[edi]

To conclude, the function returns 0xCCCCCC00 in the EAX register, which is then interpreted as true by the managed wrapper. Why? Because by default, P/Invoke considers bool parameters to be four-byte values, akin to the Win32 BOOL, and then any non-zero value is mapped to true.

To fix this, I had to convince P/Invoke to marshal back the return value as a single byte. There’s no dedicated type for that in the UnmanagedType enumeration, but U1 (unsigned byte) or I1 (signed byte) will do the trick:

[return: MarshalAs(UnmanagedType.U1)] 
static extern bool IsPrime(int n);

I am posting short links and updates on Twitter as well as on this blog. You can follow me: @goldshtn

Published at DZone with permission of Sasha Goldshtein, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)