Introduction to Windows Syscalls

What Are System Calls, Really?

Every time your program does something that touches the outside world, opening a file, allocating memory, talking to the network, waiting on another thread, it’s asking the operating system to do that work on its behalf. Your program doesn’t get to poke the disk controller or the network card directly. It can’t, by design. The CPU itself enforces this separation through privilege rings: user-mode code (ring 3 on x86/x64) runs with a restricted set of permissions, while the kernel (ring 0) runs with full access to hardware and memory. A system call is the formal, controlled doorway between those two worlds.

Think about why this separation exists in the first place. If any application could directly manipulate hardware or arbitrary physical memory, one buggy or malicious program could take down the whole machine, read another process’s secrets, or corrupt the filesystem out from under everyone else. So the CPU and the OS conspire to make sure that the only way into kernel mode is through a narrow, well-defined gate: the syscall. Your code sets up some registers or a stack with the operation you want and its arguments, then executes a special instruction that traps into the kernel (historically int 0x2e on older x86 Windows, and syscall/sysenter on modern x64 and x86 respectively). The CPU switches privilege levels, the kernel validates what you’re asking for, does the work, and hands control back to you in user mode with a result.

That transition isn’t free, though, and this is worth dwelling on because it explains a lot of design decisions elsewhere in the OS. Switching between user mode and kernel mode means saving and restoring CPU state, flushing or otherwise dealing with pipeline and cache effects, and running through validation logic in the kernel that has to assume nothing about the caller. Do this too often, in a tight loop, and you’ll feel it. That’s why performance-sensitive code tries to batch operations, use buffering, or avoid syscalls in hot paths wherever possible, and it’s part of why Windows gives you both a high-level, friendly API and a lower-level, closer-to-the-metal one, so you can choose the right tool depending on whether you care more about convenience or raw speed.

A few threads run through everything else in this post, so it’s worth naming them up front. First, privilege: syscalls are the mechanism by which user code is allowed to do privileged things, but only in ways the kernel explicitly permits and validates. Second, error handling: because the kernel can’t just throw an exception across the mode boundary the way you might in ordinary code, Windows APIs report failure through return values plus a secondary mechanism (GetLastError() for WinAPI, NTSTATUS codes for the Native API) that you have to actively check. Third, the tradeoff between performance and portability: dropping down to raw syscalls can shave overhead and dodge some safety nets, but it ties your code tightly to the internals of a specific Windows version. And fourth, that user-to-kernel transition cost means syscall efficiency isn’t just an academic concern, it shows up in real profiling data for anything I/O heavy.

Two Layers, One Kernel: WinAPI vs the Native API

Here’s the part that trips people up when they first start reading about Windows internals: there isn’t just one API for reaching the kernel, there are (at least) two layers stacked on top of each other, and understanding why both exist tells you a lot about how Windows is put together.

At the top is the Windows API, usually just called WinAPI, or Win32 for historical reasons even though it now covers 64-bit systems too. This is the API most Windows programmers actually use: functions like CreateFile, ReadFile, CreateProcess, and so on, declared in headers like windows.h and implemented in DLLs like kernel32.dll and advapi32.dll. WinAPI is deliberately friendly. It has sensible defaults, does argument checking, translates flags into whatever the kernel actually expects, and generally tries to save you from yourself. Microsoft documents it extensively and treats it as a stable, supported contract: if you write code against WinAPI today, you have strong guarantees it’ll keep working on future Windows versions.

But WinAPI doesn’t talk to the kernel directly. Underneath it sits the Native API, sometimes called the NT API, implemented in ntdll.dll. This is the layer that actually performs the ring 3 to ring 0 transition and issues the real system calls into the NT kernel (ntoskrnl.exe). Functions here have names like NtCreateFile, NtReadFile, NtAllocateVirtualMemory, and they’re the direct, thin wrappers around what the kernel exposes. When you call CreateFile, what’s actually happening under the hood is that kernel32.dll validates and translates your arguments, then calls NtCreateFile in ntdll.dll, which does the actual mode transition into the kernel.

Why bother with two layers instead of just exposing NtCreateFile to everyone directly? A few reasons, and they’re all still relevant today. The Native API is explicitly undocumented and unsupported by Microsoft. Function signatures, structure layouts, and even the existence of specific Nt functions can change between Windows versions and builds without warning, because Microsoft only guarantees compatibility for the public WinAPI surface. WinAPI exists precisely to give application developers a stable target that insulates them from that churn. There’s also a usability gap: Native API calls deal in raw kernel concepts, OBJECT_ATTRIBUTES structures, IO_STATUS_BLOCK, NTSTATUS codes, that are more expressive but a lot more fiddly than the string-and-flag interfaces WinAPI gives you.

So who actually reaches for the Native API directly? Mostly people who need capabilities WinAPI doesn’t expose at all, or who need to avoid WinAPI’s own instrumentation. Security researchers and malware authors are the classic examples: calling NtCreateFile directly via GetProcAddress, rather than going through CreateFile, means you skip whatever hooks or monitoring might be sitting in kernel32.dll, which matters if you’re trying to evade an EDR product that hooks the WinAPI layer but not ntdll.dll (or vice versa, in the case of tools that hook ntdll.dll itself, which is why some malware goes even further and issues the syscall instruction manually with hardcoded syscall numbers, bypassing ntdll.dll entirely). Driver developers and low-level tooling authors also use the Native API when they need functionality that simply has no WinAPI equivalent. For everyone else, writing ordinary applications, sticking with WinAPI is almost always the right call: it’s documented, stable, and does a lot of useful validation for you.

A Map of Common Syscalls

It helps to have a rough map of what kinds of operations route through the syscall boundary, because it’s not just file I/O, it’s essentially everything that touches shared or privileged OS state. The table below groups the common categories along with their most familiar WinAPI entry points.

Notice what these categories have in common: files, processes, memory, synchronization primitives, sockets, registry keys. Every one of them is a resource that’s either shared across the system or requires the kernel to arbitrate access to it. A file lives on a device the kernel manages. A process is a kernel object with its own address space and security context that only the kernel can create or destroy safely. Memory allocation touches the page tables, which are kernel-owned data structures. Synchronization objects like mutexes coordinate across threads and potentially across processes, which again means the kernel has to be the referee. Even the registry, conceptually just a hierarchical settings database, is implemented as a kernel-managed hive with its own access control. None of this is optional plumbing you could push into user mode; it’s exactly the set of things that has to be centrally controlled if the OS is going to remain stable and secure with dozens of processes running concurrently.

It’s also worth noticing that WaitForSingleObject shows up in two rows. That’s not a mistake, it’s a good illustration of how Windows syncs different kinds of kernel objects (process handles, thread handles, mutexes, events) under one unified waiting mechanism. The kernel doesn’t need a different “wait” primitive for every object type; anything that’s “waitable” plugs into the same dispatcher wait logic. That kind of unification is a recurring theme in the NT kernel design, and it’s part of why the object manager (the subsystem that tracks handles, files, processes, and so on as generic kernel objects) is one of the more elegant pieces of Windows internals.

Making System Calls: The WinAPI Path

Let’s actually look at code, starting with the friendly layer. Here’s the canonical example: creating a file.

#include <windows.h>
#include <stdio.h>

int main() {
    HANDLE hFile = CreateFile(
        "example.txt", GENERIC_WRITE, 0, NULL, CREATE_NEW,
        FILE_ATTRIBUTE_NORMAL, NULL
    );

    if (hFile == INVALID_HANDLE_VALUE) {
        printf("Error: %lu\n", GetLastError());
        return 1;
    }

    CloseHandle(hFile);
    return 0;
}

There’s more going on here than the seven-argument function call suggests. CreateFile isn’t just “open a file”, it’s a general-purpose factory for opening or creating almost any kind of I/O handle on Windows: files, but also pipes, serial ports, and various device objects, which is part of why the signature is so wide. GENERIC_WRITE tells the kernel what access rights you’re requesting up front, which matters because Windows checks and grants access rights once, at open time, rather than re-checking permissions on every subsequent read or write. That’s a meaningful design choice: it means the cost of security enforcement is paid once per handle, not once per operation, which keeps ReadFile/WriteFile calls cheap. CREATE_NEW says “fail if this file already exists” rather than silently overwriting or opening it, which is the kind of flag you want to get right, because the difference between CREATE_NEW, CREATE_ALWAYS, OPEN_EXISTING, and OPEN_ALWAYS is exactly the difference between “safe” and “accidentally clobbering a file that was already there.”

Notice the error handling pattern too: CreateFile doesn’t throw an exception on failure, it returns a sentinel value (INVALID_HANDLE_VALUE, not NULL, which is a common gotcha for people used to other APIs) and you’re expected to call GetLastError() immediately afterward to find out why. That “immediately” matters. GetLastError() reflects thread-local state that gets overwritten by the next API call, so if you do anything else in between (even something that looks harmless, like a logging call that itself makes a WinAPI call) you can lose the real error code. This pattern exists because it’s cheap: no exception machinery, no unwinding, just a return value plus a place to stash more detail if the caller wants it.

Reading is the natural next step once you’ve got a handle:

HANDLE hFile = CreateFile("example.txt", GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
char buffer[100];
DWORD bytesRead;

if (!ReadFile(hFile, buffer, sizeof(buffer)-1, &bytesRead, NULL)) {
    printf("Read error: %lu\n", GetLastError());
}
buffer[bytesRead] = '\0';
printf("Data: %s\n", buffer);
CloseHandle(hFile);

The detail worth internalizing here is bytesRead. ReadFile doesn’t promise to fill your buffer completely, it promises to tell you how much it actually read, and a well-behaved caller uses that count rather than assuming the buffer got fully populated. This matters more than it looks like it should: reads from disks, network-backed drives, or pipes can legitimately return short, and code that ignores bytesRead and just treats the buffer as fully written is a classic source of bugs (and, in security-sensitive code, of information disclosure, since you might print or transmit whatever garbage was previously sitting in the unfilled part of the buffer). The manual null-termination on the next line is a reminder that Windows file APIs work with raw bytes, not C strings, so it’s on you to make the data safe to treat as text.

Writing follows the mirror-image pattern:

const char *data = "Hello, World!";
DWORD bytesWritten;
WriteFile(hFile, data, strlen(data), &bytesWritten, NULL);

Same idea in reverse: bytesWritten tells you how much actually made it out, which you should check against what you asked to write, especially on things like named pipes or sockets-as-handles where partial writes are a normal, expected occurrence rather than an edge case.

Dropping Down to the Native API

Now let’s go one layer deeper and look at what’s actually happening underneath CreateFile. ntdll.dll exports the raw NT function, and because it’s not part of the documented public API surface, you typically can’t just #include a header and link against it the normal way. Instead, the common pattern is to resolve it dynamically at runtime:

#include <windows.h>
#include <winternl.h>

typedef NTSTATUS (NTAPI *NtCreateFilePtr)(
    PHANDLE, ACCESS_MASK, POBJECT_ATTRIBUTES, PIO_STATUS_BLOCK,
    PLARGE_INTEGER, ULONG, ULONG, ULONG, ULONG, PVOID, ULONG
);

HMODULE ntdll = GetModuleHandleA("ntdll.dll");
NtCreateFilePtr NtCreateFile = (NtCreateFilePtr)GetProcAddress(ntdll, "NtCreateFile");

Walk through what this snippet is really doing. GetModuleHandleA grabs a handle to ntdll.dll, which is always already loaded in every Windows process (you can’t run without it, since it’s the layer that bootstraps process startup and hosts the actual syscall stubs). GetProcAddress then looks up the exported symbol NtCreateFile by name and hands you a raw function pointer, which you cast to a function pointer type matching the (undocumented, but well-reverse-engineered) NT signature. This is exactly the mechanism by which you can call an undocumented function without Microsoft ever having to publish a header for it: as long as you know the export name and can guess or find the calling convention and argument layout, GetProcAddress doesn’t care whether the function is “supported.”

That signature is worth pausing on because it’s a good example of how much more the kernel actually wants to know compared to what CreateFile asks for. POBJECT_ATTRIBUTES bundles up things like the object name, a root directory handle for relative opens, and security attributes into a single structure, rather than passing a bare path string; that’s part of a more general object-manager convention that spans far more than just files. PIO_STATUS_BLOCK is where the kernel writes back detailed status information about how the I/O request went, split into a status code and an operation-specific “information” field, which is more granular than the single DWORD return you’d get from GetLastError(). And the return type is NTSTATUS, not a boolean or a handle: NT status codes are a much richer error-reporting scheme than the WinAPI error codes, with distinctions (like informational versus warning versus error severities) that get flattened away by the time WinAPI translates them into a GetLastError() value.

That richness is exactly why CreateFile exists: most callers don’t want to build an OBJECT_ATTRIBUTES structure and parse an IO_STATUS_BLOCK just to open a file. But if you’re doing something WinAPI doesn’t support, opening a file relative to a directory handle rather than a full path, or using access modes and sharing semantics that don’t map cleanly onto the WinAPI flag set, dropping to NtCreateFile directly is the way to get at the kernel’s full feature set. It’s more work, and it requires a real understanding of kernel structures like OBJECT_ATTRIBUTES and IO_STATUS_BLOCK, but it’s also a look at what the kernel is actually capable of before WinAPI simplifies it down.

Beyond Files: Processes, Memory, Synchronization, and Networking

Files are the easiest category to reason about because everyone’s used a file API before, but the same user-mode-to-kernel-mode story plays out across the rest of the syscall surface. It’s worth walking through a few more examples, because each one highlights a different reason the kernel has to be involved.

Process Management

STARTUPINFO si = { sizeof(si) };
PROCESS_INFORMATION pi;

CreateProcess(NULL, "notepad.exe", NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);
WaitForSingleObject(pi.hProcess, INFINITE);
CloseHandle(pi.hProcess);
CloseHandle(pi.hThread);

Creating a process is a heavier operation than it might look, and it’s a good example of why user-mode code can’t just be handed the power to spin up new processes without kernel involvement. The kernel has to build a brand-new address space, set up the initial thread and its stack, load the executable image and its dependent DLLs, establish a security token derived from (or explicitly overriding) the parent’s, and register the new process as a kernel object that other code can reference by handle. CreateProcess returns two handles, one for the process and one for its initial thread, packed into that PROCESS_INFORMATION structure, and both need to be closed explicitly. This is a detail people forget: handles are a reference-counted kernel resource, and leaking them (by not calling CloseHandle) is a real, if slow-burning, resource leak, exactly the same category of bug as leaking file descriptors on Unix.

WaitForSingleObject here is doing double duty as a demonstration of that unified wait mechanism mentioned earlier: pi.hProcess is waitable because a process object becomes “signaled” when it terminates, so waiting on it is literally how you block until a child process exits, no separate “wait for process exit” API required.

Memory Management

LPVOID pMemory = VirtualAlloc(NULL, 1024, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
ZeroMemory(pMemory, 1024);
VirtualFree(pMemory, 0, MEM_RELEASE);

VirtualAlloc is the syscall-backed workhorse underneath higher-level allocators like the C runtime’s malloc or the WinAPI heap functions (HeapAlloc and friends usually carve memory out of a heap that was itself obtained via VirtualAlloc, and only fall back to the kernel when the heap needs to grow). The MEM_RESERVE/MEM_COMMIT distinction is a genuinely useful idea worth understanding on its own: reserving address space just claims a range of virtual addresses without backing it with physical memory or page-file space, while committing actually asks the kernel to guarantee that memory will be available (backed by RAM or the page file) when you touch it. You can reserve a huge range cheaply and commit only the parts you end up needing, which is exactly how things like growable stacks and sparse data structures avoid wasting memory up front. PAGE_READWRITE sets the page protection, and this is one of the more security-relevant knobs in the whole API: getting page protections wrong (say, allocating memory as both writable and executable) is one of the classic ingredients in code-injection techniques, because it lets you write shellcode into memory and then jump to it.

Synchronization

HANDLE hMutex = CreateMutex(NULL, FALSE, NULL);
WaitForSingleObject(hMutex, INFINITE);
// Critical section
ReleaseMutex(hMutex);
CloseHandle(hMutex);

A mutex is a kernel object precisely because mutual exclusion across threads (and potentially across processes, since named mutexes can be shared between them) needs a referee that isn’t itself subject to the race condition it’s trying to prevent. If two threads both tried to implement locking purely in user-mode memory without kernel help, you’d need atomic CPU instructions at minimum, and you’d still lack a way to make a thread sleep efficiently while waiting rather than burning CPU in a spin loop. WaitForSingleObject(hMutex, INFINITE) does exactly that: it blocks the calling thread, taking it off the CPU entirely, until the mutex becomes available, at which point the scheduler wakes it back up. That’s a real syscall-mediated context switch, not a busy-wait, and it’s part of why kernel-provided synchronization primitives are worth using over hand-rolled ones even though they cost more per acquisition than a pure user-mode spinlock: they scale far better when contention means actual waiting rather than brief spins.

Networking (TCP Server)

WSADATA wsaData;
WSAStartup(MAKEWORD(2,2), &wsaData);

SOCKET listenSocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
bind(listenSocket, (struct sockaddr*)&serverAddr, sizeof(serverAddr));
listen(listenSocket, SOMAXCONN);
SOCKET clientSocket = accept(listenSocket, (struct sockaddr*)&clientAddr, &clientAddrSize);
closesocket(clientSocket);
closesocket(listenSocket);
WSACleanup();

Winsock is a slightly different beast from the rest of this post because it isn’t a thin wrapper over NT syscalls in the same direct way file and process APIs are, it’s Windows’ implementation of the Berkeley sockets model, layered on top of the kernel’s networking stack, with WSAStartup/WSACleanup handling library-level initialization and teardown of the sockets DLL itself (a step that has no real equivalent for file or process APIs, and that trips up people porting Unix networking code). But the underlying shape of the interaction is the same story as everything else in this post: socket asks the kernel to allocate a socket object, bind asks it to associate that socket with a local address and port (which requires the kernel to check whether you actually have permission to bind there, since low port numbers are historically privileged), listen flips the socket into a mode where the kernel will start queuing incoming connection attempts on your behalf, and accept blocks until the kernel hands you a fresh socket for a completed incoming connection. Every one of those steps needs kernel involvement because the network stack, packet queues, port bindings, and connection state machines are shared, security-sensitive resources, exactly like files, processes, and memory. It’s the same underlying pattern wearing a different API skin.

Wrapping Up

Once you see it, the pattern repeats everywhere in Windows: a friendly, documented WinAPI function validates your arguments and calls down into a leaner, undocumented NT function in ntdll.dll, which executes the actual instruction that crosses from user mode into kernel mode. WinAPI trades a bit of raw performance and flexibility for stability and ease of use; the Native API trades that safety net for direct access to what the kernel can really do. Most code should live in WinAPI. But knowing what’s underneath it, and knowing how to get there with GetProcAddress when you genuinely need to, is what separates “I can call Windows functions” from actually understanding how Windows works.