Since the debug output from the ld
dynamic linker/loader confirms that both the victim
and spy
programs load the correct input file, the next step would be to verify if the kernel has actually set up the physical pages where libmyl.so
is loaded in memory to be shared between the victim
and spy
.
In Linux this is possible to verify since kernel version 2.6.25 via the pagemap
interface in the kernel that allows userspace programs to examine the page tables and related information by reading files in /proc
.
The general procedure for using pagemap to find out if two processes share memory goes like this:
- Read
/proc/<pid>/maps
for both processes to determine which parts of the memory space are mapped to which objects.
- Select the maps you are interested in, in this case the pages to which
libmyl.so
is mapped.
- Open
/proc/<pid>/pagemap
. The pagemap
consists of 64-bit pagemap descriptors, one per page. The mapping between the page's address and it's descriptors address in the pagemap
is page address / page size * descriptor size. Seek to the descriptors of the pages you would like to examine.
- Read a 64-bit descriptor as an unsigned integer for each page from the
pagemap
.
- Compare the page frame number (PFN) in bits 0-54 of the page descriptor between the
libmyl.so
pages for victim
and spy
. If the PFNs match, the two processes are sharing the same physical pages.
The following sample code illustrates how the pagemap
can be accessed and printed from within the process. It uses dl_iterate_phdr()
to determine the virtual address of each shared library loaded into the processes memory space, then looks up and prints the corresponding pagemap
from /proc/<pid>/pagemap
.
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <inttypes.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <link.h>
#include <errno.h>
#include <error.h>
#define E_CANNOT_OPEN_PAGEMAP 1
#define E_CANNOT_READ_PAGEMAP 2
typedef struct __attribute__ ((__packed__)) {
union {
uint64_t pmd;
uint64_t page_frame_number : 55;
struct {
uint64_t swap_type: 5;
uint64_t swap_offset: 50;
uint64_t soft_dirty: 1;
uint64_t exclusive: 1;
uint64_t zero: 4;
uint64_t file_page: 1;
uint64_t swapped: 1;
uint64_t present: 1;
};
};
} pmd_t;
static int print_pagemap_for_phdr(struct dl_phdr_info *info,
size_t size, void *data)
{
struct stat statbuf;
size_t pagesize = sysconf(_SC_PAGESIZE);
char pagemap_path[BUFSIZ];
int pagemap;
uint64_t start_addr, end_addr;
if (!strcmp(info->dlpi_name, "")) {
return 0;
}
stat(info->dlpi_name, &statbuf);
start_addr = info->dlpi_addr;
end_addr = (info->dlpi_addr + statbuf.st_size + pagesize) & ~(pagesize-1);
printf("\n%10p-%10p %s\n\n",
(void *)start_addr,
(void *)end_addr,
info->dlpi_name);
snprintf(pagemap_path, sizeof pagemap_path, "/proc/%d/pagemap", getpid());
if ((pagemap = open(pagemap_path, O_RDONLY)) < 0) {
error(E_CANNOT_OPEN_PAGEMAP, errno,
"cannot open pagemap: %s", pagemap_path);
}
printf("%10s %8s %7s %5s %8s %7s %7s\n",
"", "", "soft-", "", "file /", "", "");
printf("%10s %8s %7s %5s %11s %7s %7s\n",
"address", "pfn", "dirty", "excl.",
"shared anon", "swapped", "present");
for (unsigned long i = start_addr; i < end_addr; i += pagesize) {
pmd_t pmd;
if (pread(pagemap, &pmd.pmd, sizeof pmd.pmd, (i / pagesize) * sizeof pmd) != sizeof pmd) {
error(E_CANNOT_READ_PAGEMAP, errno,
"cannot read pagemap: %s", pagemap_path);
}
if (pmd.pmd != 0) {
printf("0x%10" PRIx64 " %06" PRIx64 " %3d %5d %8d %9d %7d\n", i,
(unsigned long)pmd.page_frame_number,
pmd.soft_dirty,
pmd.exclusive,
pmd.file_page,
pmd.swapped,
pmd.present);
}
}
close(pagemap);
return 0;
}
int main()
{
dl_iterate_phdr(print_pagemap_for_phdr, NULL);
exit(EXIT_SUCCESS);
}
The output of the program should look similar to the following:
$ sudo ./a.out
0x7f935408d000-0x7f9354256000 /lib/x86_64-linux-gnu/libc.so.6
soft- file /
address pfn dirty excl. shared anon swapped present
0x7f935408d000 424416 1 0 1 0 1
0x7f935408e000 424417 1 0 1 0 1
0x7f935408f000 422878 1 0 1 0 1
0x7f9354090000 422879 1 0 1 0 1
0x7f9354091000 43e879 1 0 1 0 1
0x7f9354092000 43e87a 1 0 1 0 1
0x7f9354093000 424790 1 0 1 0 1
...
where:
address
is the virtual address of the page
pfn
is the pages page frame number
soft-dirty
indicates the if soft-dirty bit is set in the pages Page Table Entry (PTE).
excl.
indicates if the page is exclusively mapped (i.e. page is only mapped for this process).
file / shared anon
indicates if the page is a file pages or a shared anonymous page.
swapped
indicates if the page is currently swapped (implies present
is zero).
present
indicates if the page is currently present in the processes resident set (implies swapped
is zero).
(Note: I run the example program with sudo
as since Linux 4.0 only users with the CAP_SYS_ADMIN
capability can get PFNs from /proc/<pid>/pagemap
. Starting from Linux 4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN
. The reason for this change is to make it more difficult to exploit another memory related vulnerability, the Rowhammer attack, using the information on the virtual-to-physical mapping exposed by the PFNs.)
If you run the example program several times, you should notice that the virtual address of the page should change (due to ASLR), but the PFN for shared libraries that are in use by other processes should stay the same.
If the PFNs for libmyl.so
match between the victim
and spy
program, I would start looking for a reason for why the attack fails in the attack code itself. If the PFNs don't match, the additional bits may give some hint why the pages are not set up to be shared. The pagemap
bits indicate the following:
present file exclusive state:
0 0 0 non-present
1 1 0 file page mapped somewhere else
1 1 1 file page mapped only here
1 0 0 anonymous non-copy-on-write page (shared with parent/child)
1 0 1 anonymous copy-on-write page (or never forked)
Copy-on-write pages in (MAP_FILE | MAP_PRIVATE)
areas are anonymous in this context.
Bonus: To obtain the number of times a page has been mapped, the PFN can be used to look up the page in /proc/kpagecount
. This file contains a 64-bit count of the number of times each page is mapped, indexed by PFN.
Method 2 and 3 (modifying ps, top, libraries), is prone to walk around breach. E.g. an attacker can just use there own ps
.
A better way would be to block these tools from accessing info on processes of other users. Then running the processes to be hidden as a different user.
For the 1st part, edit /etc/fstab
, to include
#protect /proc
proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=2,gid=admin 0 0
Best Answer
I believe that the really short answer is that Linux compilers arrange code into pieces, at least one of which is just pure code, and can therefore be memory mapped into more than one process' address space. Any globals get mapped such that each process gets its own copy.
You can see this using
readelf
, orobjdump
, butreadelf
gives a clearer picture, I think.Here's a piece of output from
readelf -e /usr/lib/libc.so.6
. That's the C library, probably mapped into almost every process. The relevant part ofreadelf
output (although all of it is interesting) is the Program Headers:The two LOAD lines are the only pieces of the file that get mapped directly into memory. The first LOAD header maps a piece of
/usr/lib/libc.so.6
into memory with R and E permissions: read and execute. That's the code. Hardware features keep a program from writing to that piece of memory, so all programs can share the same pages of real, physical memory. The kernel can set up the hardware to map the same physical memory into all processes.The second LOAD header is marked RW - read and write. This is the part with global variables that the C library uses. Each process gets its own copy in physical memory, mapped into that process' address space with the hardware permissions set to allow reading and writing. That section is not shared.
You can see these memory mappings in a running process using the
/proc
file system. A good command to illustrate:cat /proc/self/maps
. That lists all the memory mappings that thecat
process has, and from what files the kernel got them.As far as how much you have to do to ensure that your function is allocated to memory that gets mapped into different processes, it's pretty much all down to the flags you give to the compiler. Code intended for ".so" shared libraries is compiled "position independent". Position independent code does things like refer to memory locations of variables with offsets relative to the current instruction, and jumps or branches to locations relative to the current instruction, rather than loading from or writing to absolute addresses, and jumping to absolute addresses. That means the "RE" LOAD piece of
/usr/lib/libc.so
and the "RW" piece only have to have be loaded at addresses that are the same distance apart in each process. In your example code, thestatic
variable will always be at least a multiple of a page size apart from the code that references it, and it will always get loaded that distance apart in a process' address space due to the way the LOAD ELF headers are given.Just a note about the term "shared memory": There's a user-level shared memory system, associate with "System V interprocess communications system". That's a way for more than one process to very explicitly share a piece of memory. It's fairly complicated and obscure to set up and get correct. The shared memory that we're talking about here is more-or-less completely invisible to any user process. Your example code won't know the difference if it's running as position independent code shared between multiple processes, or if it's the only copy ever.