Buffer Overflow - Stack and Heap Overflow Explained

Summary: A buffer overflow occurs when a program writes more data to a buffer than it can hold, causing adjacent memory areas to be overwritten. Buffer overflows enable stack-based overflows (overwriting the return address → arbitrary code), heap-based overflows, and format string attacks. Protection mechanisms: ASLR, stack canaries, NX bit/DEP, PIE, Fortify Source. Buffer overflows are one of the oldest classes of vulnerabilities and form the basis of many exploits.

Buffer overflows are over 40 years old - and still relevant today. The Morris Worm (1988) exploited a stack overflow in fingerd. OpenSSL Heartbleed (2014) was a heap buffer over-read. Log4Shell was facilitated by a buffer overflow in Java deserialization. C/C++ code without modern protection mechanisms remains vulnerable: embedded systems, network equipment, SCADA/OT systems, browser engines. A buffer overflow in the right place enables remote code execution with system privileges.

Stack-Based Buffer Overflow

Memory layout (x86, simplified) - the stack grows from high to low address:

[high address]
| Function arguments     |
| Return Address (RET)   |  ← Return address - critical!
| Saved Base Pointer     |
| Local variables       |  ← Buffer is located here
| ...                    |
[low address]

Normal function call (vulnerable C code):

void vulnerable(char *input) {
    char buffer[64];       // 64-byte stack buffer
    strcpy(buffer, input); // UNSAFE: no length check!
}

If input > 64 bytes: buffer overflows, overwriting Saved EBP and the return address.

Exploit flow (classic)

# Total payload:
python3 -c "import sys; sys.stdout.buffer.write(
    b'A' * 64 +         # Fill buffer
    b'B' * 4 +          # Overwrite Saved EBP
    b'\xef\xbe\xad\xde' +  # Return Address (little endian!)
    b'\x90' * 20 +      # NOP sled
    shellcode           # Actual shellcode
)" | ./vulnerable_program

Unsafe C Functions (always avoid)

Function	Problem
gets()	no length check, always unsafe
strcpy()	no length check
strcat()	no length check
sprintf()	when format string is abused
scanf()	%s without length specification

Safe Alternatives:

fgets(buffer, sizeof(buffer), stdin)  // with length check!
strncpy(dest, src, sizeof(dest)-1)    // with length check
strlcpy(dest, src, sizeof(dest))      // BSD extension
snprintf(buf, sizeof(buf), ...)       // safe

Protection Mechanisms

1. Stack Canaries

• Compiler inserts a random value before the return address
• Upon returning from a function: Check the value
• If altered: Terminate the program
• GCC: -fstack-protector-all

# Check if Canary is active:
checksec --file=./program
# Output: STACK CANARY: Enabled

Workaround: Format string → Stack leak → Read canary value → Insert into overflow → Passes check.

2. ASLR (Address Space Layout Randomization)

• Operating system: Stack/heap/libraries different at every startup
• Attacker does not know target address → Exploit more difficult

# Linux - Check status:
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 2 = fully enabled

# Enable:
echo 2 > /proc/sys/kernel/randomize_va_space

Workarounds:

• Information leak: Read program address via another bug
• Brute force (32-bit: only 16-bit entropy → 65,536 attempts)
• ret2libc: libc always at a predictable location (without PIE)

3. NX Bit / DEP (No-Execute / Data Execution Prevention)

• Memory pages: either writable OR executable (not both!)
• Shellcode on stack → not executable
• Hardware: CPU NX bit (Intel XD, AMD NX)
• OS: W^X (Write XOR Execute)

checksec --file=./program
# NX: Enabled

Workaround: Return-Oriented Programming (ROP) - uses existing code fragments (gadgets) in legitimate libraries without injecting custom code.

4. PIE (Position-Independent Executable)

• Program loads itself at a random address (not just libraries)
• Enhances ASLR: all addresses are unknown
• GCC: -fPIE -pie

5. Fortify Source

• Compile-time + runtime: buffer size known → check
• GCC: -D_FORTIFY_SOURCE=2
• Replaces strcpy → __strcpy_chk (with size check!)

6. Control Flow Integrity (CFI)

• Validates allowed jump targets
• Return address must point to a known return location
• Clang CFI, Microsoft Control Flow Guard (CFG)
• Complicates ROP chains

> Stack Canaries + ASLR + NX + PIE + CFI together make exploit development very difficult (but not impossible!) - typically, multiple bugs are required: Info leak + Buffer overflow + CFI bypass.

Modern Variants

Heap-Based Buffer Overflow

Heap memory is dynamically allocated (malloc/new) - there is no return address on the heap, so other attack vectors are required.

Targets on the heap:

• Heap metadata (free list, chunk headers) → tcache poisoning
• Function pointers on the heap → overwrite → code execution
• C++ vtable pointer → overwrite → call any method

// Example (Use-After-Free + Heap Overflow):
char *buf = malloc(16);    // 16 bytes allocated
read(0, buf, 256);         // Writes 256 bytes → Heap Overflow!
// → Overwrites neighboring chunk header

Format string vulnerability

printf(user_input);           // WRONG: user_input as format string!
printf("%s", user_input);     // CORRECT: fixed format

Attack:

• user_input = "%x.%x.%x.%x" → reads stack values (Leak!)
• user_input = "%n" → writes to memory (Code Execution!)
• Attacker: reads canary value, addresses → ASLR bypass → classic buffer overflow with known addresses

Integer Overflow → Buffer Overflow

uint8_t size = user_provided_size;  // 0-255
size = size * 2;                    // Overflow! 200*2=400 → 400%256=144!
char *buf = malloc(size);           // Buffer too small!
memcpy(buf, src, user_provided_size * 2);  // Overflow!

Detection and Prevention

Static Analysis

flawfinder ./src/         # C/C++ vulnerability scanner
cppcheck --enable=all ./  # C++ static analysis
# CodeQL: github.com/github/codeql-action (CI/CD integration)

Dynamic Analysis

# AddressSanitizer (ASan):
# Compile with: -fsanitize=address → Detect heap/stack overflows

valgrind --tool=memcheck ./program  # Detect memory errors

# American Fuzzy Lop (AFL):
afl-fuzz -i inputs/ -o findings/ ./program

Compiler Flags Best Practices

CFLAGS="-Wall -Wextra -Werror \
        -fstack-protector-all \
        -D_FORTIFY_SOURCE=2 \
        -fPIE -pie \
        -Wl,-z,relro,-z,now \
        -fsanitize=address,undefined"

Flag	Effect
-Wall -Wextra -Werror	Treat all warnings as errors
-fstack-protector-all	Stack Canaries
-D_FORTIFY_SOURCE=2	Fortify
-fPIE -pie	PIE
-Wl,-z,relro,-z,now	RELRO
-fsanitize=address,undefined	ASan + UBSan