Linux – How to ensure that a shared library will have its memory pages shared by several processes

cachelinuxlinux-kernelshared library

Let me first describe what I want to do, followed by what I manage to do, and finally my issue.

Goal: implementing flush+flush cache attack in C

I am trying to implement in C the flush+flush cache attack (https://gruss.cc/files/flushflush.pdf). Basically, it exploits the fact that two different processes might share the same memory pages when using shared libraries. This results in a "shared usage" of the cache.

Let assume that we have a victim process, continually running and sometimes executing a function func imported from a shared library.

In parallel, we assume that we have a spy process, running on the same computer as the victim, whose goal is to spy when the victim calls func. The spy also has access to the same shared library. The spy process pseudo-code is the following:

i=0;
for (i = 0; i < trace_length ; i++)
{
    trace[i] = flush_time( address of function "func");
    i++;
}

where flush_time(<address>) is a function that returns the time it takes for the CPU to flush the memory pointed by address from all the cache levels. On Intel processor, this can be achieved through the assembly instruction clflush. One can observe that the execution of clflush is faster when the address is not present in the cache. As a result, the timing required to flush a memory address can directly be translated in its presence (or not) inside the cache.

The spy process returns a trace vector which contains the flush_time results over the time. From the previous observation, this trace will exhibit higher timings when the victim also calls the function func. The spy will thus deduct when the victim is calling func.

What I manage to do: making the attack working against the GSL shared library

I implemented the aforementioned attack, where the shared library is the GSL. Arbitrarily, I chose gsl_stats_mean (defined in gsl_statistics_double) to act as the function func I am willing to spy.

In that case, the spying works perfectly as I clearly can see a timing difference when the victim program makes calls to gsl_stats_mean

My issue: the attack doesn't work on a homemade shared library

I now want to create my own shared library and use it for the spy/victim test. Let assume . denotes the folder in which my spy.c and victim.c files are. I created two files myl.c and myl.h in a folder ./src/myl, which respectively contain the description of func and it's declaration. As previously, the goal of my spy is to detect the usage of func from the victim.

Both spy.c and victim.c contain the include line: 

 #include "src/myl/myl.h"

The creation of the shared library is done using the following commands:

gcc -c -fPIC src/myl/myl.c -o bin/shared/myl.o  #creation of object in ./bin/shared
gcc -shared bin/shared/myl.o -o bin/shared/libmyl.so #creation of the shared library in ./bin/shared
gcc -c spy.c -o spy.o #creation of spy's process object file
gcc -c victim.c -o victim.o #creation of victim's process object file
gcc spy.o -Lbin/shared -lmyl -o spy #creation of spy's executable
gcc victim.o -Lbin/shared -lmyl -o victim #creation of victim's executable

I then launch my victim and spy using the following lines:

LD_LIBRARY_PATH=$(pwd)/bin/shared ./victim
LD_LIBRARY_PATH=$(pwd)/bin/shared ./spy

However, as opposed to the case where I was using the GSL function, I cannot see any activity on the cache anymore. I guess this means that my spy and victim processes do not share the same memory page for my shared library (while however, it was the case when using the GSL). Note that when compiling this way, the spying still works when targeting a GSL function.

My main question is the following: how to ensure that a homemade compiled shared library will have the memory paging shared when being executed by several processes at the same time? It seems to be the case for "proper" library that I installed, such as the GSL, gmp, native libraries etc…. But not for the one I made myself.

Thanks in advance, and I apologize if the answer is straightforward.

EDIT: output of LD_DEBUG=libs and files for both spy and victim.
NOTE: victim is called pg2 and spy is called pg1 (sorry about that)

First, libs for victim, followed by files for victim (pg2). Then, libs for the spy, followed by files for the spy (pg1):

LD_DEBUG=libs LD_LIBRARY_PATH=$(pwd)/bin/shared ./pg2
 31714: find library=libmyl.so [0]; searching
 31714:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/x86_64:/home/romain/Documents/work/test/page_sharing/bin/shared/tls:/home/romain/Documents/work/test/page_sharing/bin/shared/x86_64:/home/romain/Documents/work/test/page_sharing/bin/shared      (LD_LIBRARY_PATH)
 31714:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/x86_64/libmyl.so
 31714:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/libmyl.so
 31714:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/x86_64/libmyl.so
 31714:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31714: 
 31714: find library=libc.so.6 [0]; searching
 31714:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared       (LD_LIBRARY_PATH)
 31714:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libc.so.6
 31714:  search cache=/etc/ld.so.cache
 31714:   trying file=/lib/x86_64-linux-gnu/libc.so.6
 31714: 
 31714: 
 31714: calling init: /lib/x86_64-linux-gnu/libc.so.6
 31714: 
 31714: 
 31714: calling init: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31714: 
 31714: 
 31714: initialize program: ./pg2
 31714: 
 31714: 
 31714: transferring control: ./pg2
 31714: 



LD_DEBUG=files LD_LIBRARY_PATH=$(pwd)/bin/shared ./pg2
 31901: 
 31901: file=libmyl.so [0];  needed by ./pg2 [0]
 31901: file=libmyl.so [0];  generating link map
 31901:   dynamic: 0x00007f5a3b34be48  base: 0x00007f5a3b14b000   size: 0x0000000000201028
 31901:     entry: 0x00007f5a3b14b580  phdr: 0x00007f5a3b14b040  phnum:                  7
 31901: 
 31901: 
 31901: file=libc.so.6 [0];  needed by ./pg2 [0]
 31901: file=libc.so.6 [0];  generating link map
 31901:   dynamic: 0x00007f5a3b144ba0  base: 0x00007f5a3ad81000   size: 0x00000000003c99a0
 31901:     entry: 0x00007f5a3ada1950  phdr: 0x00007f5a3ad81040  phnum:                 10
 31901: 
 31901: 
 31901: calling init: /lib/x86_64-linux-gnu/libc.so.6
 31901: 
 31901: 
 31901: calling init: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31901: 
 31901: 
 31901: initialize program: ./pg2
 31901: 
 31901: 
 31901: transferring control: ./pg2
 31901: 


LD_DEBUG=libs LD_LIBRARY_PATH=$(pwd)/bin/shared ./pg1
 31938: find library=libmyl.so [0]; searching
 31938:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/x86_64:/home/romain/Documents/work/test/page_sharing/bin/shared/tls:/home/romain/Documents/work/test/page_sharing/bin/shared/x86_64:/home/romain/Documents/work/test/page_sharing/bin/shared      (LD_LIBRARY_PATH)
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/x86_64/libmyl.so
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/tls/libmyl.so
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/x86_64/libmyl.so
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31938: 
 31938: find library=libgsl.so.23 [0]; searching
 31938:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared       (LD_LIBRARY_PATH)
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libgsl.so.23
 31938:  search cache=/etc/ld.so.cache
 31938:   trying file=/usr/local/lib/libgsl.so.23
 31938: 
 31938: find library=libgslcblas.so.0 [0]; searching
 31938:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared       (LD_LIBRARY_PATH)
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libgslcblas.so.0
 31938:  search cache=/etc/ld.so.cache
 31938:   trying file=/usr/local/lib/libgslcblas.so.0
 31938: 
 31938: find library=libc.so.6 [0]; searching
 31938:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared       (LD_LIBRARY_PATH)
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libc.so.6
 31938:  search cache=/etc/ld.so.cache
 31938:   trying file=/lib/x86_64-linux-gnu/libc.so.6
 31938: 
 31938: find library=libm.so.6 [0]; searching
 31938:  search path=/home/romain/Documents/work/test/page_sharing/bin/shared       (LD_LIBRARY_PATH)
 31938:   trying file=/home/romain/Documents/work/test/page_sharing/bin/shared/libm.so.6
 31938:  search cache=/etc/ld.so.cache
 31938:   trying file=/lib/x86_64-linux-gnu/libm.so.6
 31938: 
 31938: 
 31938: calling init: /lib/x86_64-linux-gnu/libc.so.6
 31938: 
 31938: 
 31938: calling init: /lib/x86_64-linux-gnu/libm.so.6
 31938: 
 31938: 
 31938: calling init: /usr/local/lib/libgslcblas.so.0
 31938: 
 31938: 
 31938: calling init: /usr/local/lib/libgsl.so.23
 31938: 
 31938: 
 31938: calling init: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31938: 
 31938: 
 31938: initialize program: ./pg1
 31938: 
 31938: 
 31938: transferring control: ./pg1
 31938: 
0: 322 # just some output of my spying program
1: 323 # just some output of my spying program
 31938: 
 31938: calling fini: ./pg1 [0]
 31938: 
 31938: 
 31938: calling fini: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so [0]
 31938: 
 31938: 
 31938: calling fini: /usr/local/lib/libgsl.so.23 [0]
 31938: 
 31938: 
 31938: calling fini: /usr/local/lib/libgslcblas.so.0 [0]
 31938: 
 31938: 
 31938: calling fini: /lib/x86_64-linux-gnu/libm.so.6 [0]
 31938: 




LD_DEBUG=files LD_LIBRARY_PATH=$(pwd)/bin/shared ./pg1
 31940: 
 31940: file=libmyl.so [0];  needed by ./pg1 [0]
 31940: file=libmyl.so [0];  generating link map
 31940:   dynamic: 0x00007fb3d8794e48  base: 0x00007fb3d8594000   size: 0x0000000000201028
 31940:     entry: 0x00007fb3d8594580  phdr: 0x00007fb3d8594040  phnum:                  7
 31940: 
 31940: 
 31940: file=libgsl.so.23 [0];  needed by ./pg1 [0]
 31940: file=libgsl.so.23 [0];  generating link map
 31940:   dynamic: 0x00007fb3d8582ac8  base: 0x00007fb3d8126000   size: 0x000000000046da60
 31940:     entry: 0x00007fb3d8180e30  phdr: 0x00007fb3d8126040  phnum:                  7
 31940: 
 31940: 
 31940: file=libgslcblas.so.0 [0];  needed by ./pg1 [0]
 31940: file=libgslcblas.so.0 [0];  generating link map
 31940:   dynamic: 0x00007fb3d8124df0  base: 0x00007fb3d7ee8000   size: 0x000000000023d050
 31940:     entry: 0x00007fb3d7eea120  phdr: 0x00007fb3d7ee8040  phnum:                  7
 31940: 
 31940: 
 31940: file=libc.so.6 [0];  needed by ./pg1 [0]
 31940: file=libc.so.6 [0];  generating link map
 31940:   dynamic: 0x00007fb3d7ee1ba0  base: 0x00007fb3d7b1e000   size: 0x00000000003c99a0
 31940:     entry: 0x00007fb3d7b3e950  phdr: 0x00007fb3d7b1e040  phnum:                 10
 31940: 
 31940: 
 31940: file=libm.so.6 [0];  needed by /usr/local/lib/libgsl.so.23 [0]
 31940: file=libm.so.6 [0];  generating link map
 31940:   dynamic: 0x00007fb3d7b1cd88  base: 0x00007fb3d7815000   size: 0x00000000003080f8
 31940:     entry: 0x00007fb3d781a600  phdr: 0x00007fb3d7815040  phnum:                  7
 31940: 
 31940: 
 31940: calling init: /lib/x86_64-linux-gnu/libc.so.6
 31940: 
 31940: 
 31940: calling init: /lib/x86_64-linux-gnu/libm.so.6
 31940: 
 31940: 
 31940: calling init: /usr/local/lib/libgslcblas.so.0
 31940: 
 31940: 
 31940: calling init: /usr/local/lib/libgsl.so.23
 31940: 
 31940: 
 31940: calling init: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so
 31940: 
 31940: 
 31940: initialize program: ./pg1
 31940: 
 31940: 
 31940: transferring control: ./pg1
 31940: 
0: 325 # just some output of my spying program
1: 327 # just some output of my spying program
 31940: 
 31940: calling fini: ./pg1 [0]
 31940: 
 31940: 
 31940: calling fini: /home/romain/Documents/work/test/page_sharing/bin/shared/libmyl.so [0]
 31940: 
 31940: 
 31940: calling fini: /usr/local/lib/libgsl.so.23 [0]
 31940: 
 31940: 
 31940: calling fini: /usr/local/lib/libgslcblas.so.0 [0]
 31940: 
 31940: 
 31940: calling fini: /lib/x86_64-linux-gnu/libm.so.6 [0]
 31940:

Best Answer

Since the debug output from the ld dynamic linker/loader confirms that both the victim and spy programs load the correct input file, the next step would be to verify if the kernel has actually set up the physical pages where libmyl.so is loaded in memory to be shared between the victim and spy.

In Linux this is possible to verify since kernel version 2.6.25 via the pagemap interface in the kernel that allows userspace programs to examine the page tables and related information by reading files in /proc.

The general procedure for using pagemap to find out if two processes share memory goes like this:

  1. Read/proc/<pid>/maps for both processes to determine which parts of the memory space are mapped to which objects.
  2. Select the maps you are interested in, in this case the pages to which libmyl.so is mapped.
  3. Open /proc/<pid>/pagemap. The pagemap consists of 64-bit pagemap descriptors, one per page. The mapping between the page's address and it's descriptors address in the pagemap is page address / page size * descriptor size. Seek to the descriptors of the pages you would like to examine.
  4. Read a 64-bit descriptor as an unsigned integer for each page from the pagemap.
  5. Compare the page frame number (PFN) in bits 0-54 of the page descriptor between the libmyl.so pages for victim and spy. If the PFNs match, the two processes are sharing the same physical pages.

The following sample code illustrates how the pagemap can be accessed and printed from within the process. It uses dl_iterate_phdr() to determine the virtual address of each shared library loaded into the processes memory space, then looks up and prints the corresponding pagemap from /proc/<pid>/pagemap.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <inttypes.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <link.h>
#include <errno.h>
#include <error.h>

#define E_CANNOT_OPEN_PAGEMAP 1
#define E_CANNOT_READ_PAGEMAP 2

typedef struct __attribute__ ((__packed__)) {
    union {
        uint64_t pmd;
        uint64_t page_frame_number : 55;
        struct {
            uint64_t swap_type: 5;
            uint64_t swap_offset: 50;
            uint64_t soft_dirty: 1;
            uint64_t exclusive: 1;
            uint64_t zero: 4;
            uint64_t file_page: 1;
            uint64_t swapped: 1;
            uint64_t present: 1;
        };
    };
} pmd_t;


static int print_pagemap_for_phdr(struct dl_phdr_info *info,
                                  size_t size, void *data)
{
    struct stat statbuf;

    size_t pagesize = sysconf(_SC_PAGESIZE);

    char pagemap_path[BUFSIZ];
    int pagemap;

    uint64_t start_addr, end_addr;

    if (!strcmp(info->dlpi_name, "")) {
        return 0;
    }

    stat(info->dlpi_name, &statbuf);

    start_addr = info->dlpi_addr;
    end_addr = (info->dlpi_addr + statbuf.st_size + pagesize) & ~(pagesize-1);

     printf("\n%10p-%10p %s\n\n",
            (void *)start_addr,
            (void *)end_addr,
            info->dlpi_name);

     snprintf(pagemap_path, sizeof pagemap_path, "/proc/%d/pagemap", getpid());

     if ((pagemap = open(pagemap_path, O_RDONLY)) < 0) {
         error(E_CANNOT_OPEN_PAGEMAP, errno, 
               "cannot open pagemap: %s", pagemap_path);
     }

     printf("%10s %8s %7s %5s %8s %7s %7s\n",
            "", "", "soft-", "", "file /", "", "");
     printf("%10s %8s %7s %5s %11s %7s %7s\n",
            "address", "pfn", "dirty", "excl.",
            "shared anon", "swapped", "present");

     for (unsigned long i = start_addr; i < end_addr; i += pagesize) {
          pmd_t pmd;

          if (pread(pagemap, &pmd.pmd, sizeof pmd.pmd, (i / pagesize) * sizeof pmd) != sizeof pmd) {
              error(E_CANNOT_READ_PAGEMAP, errno,
                    "cannot read pagemap: %s", pagemap_path);
          }

          if (pmd.pmd != 0) {
              printf("0x%10" PRIx64 " %06" PRIx64 " %3d %5d %8d %9d %7d\n", i,
                     (unsigned long)pmd.page_frame_number,
                     pmd.soft_dirty,
                     pmd.exclusive,
                     pmd.file_page,
                     pmd.swapped,
                     pmd.present);
          }
    }

    close(pagemap);

    return 0;
}

int main()
{
    dl_iterate_phdr(print_pagemap_for_phdr, NULL);

    exit(EXIT_SUCCESS);
}

The output of the program should look similar to the following:

$ sudo ./a.out

0x7f935408d000-0x7f9354256000 /lib/x86_64-linux-gnu/libc.so.6

                      soft-         file /                
   address      pfn   dirty excl. shared anon swapped present
0x7f935408d000 424416   1     0        1         0       1
0x7f935408e000 424417   1     0        1         0       1
0x7f935408f000 422878   1     0        1         0       1
0x7f9354090000 422879   1     0        1         0       1
0x7f9354091000 43e879   1     0        1         0       1
0x7f9354092000 43e87a   1     0        1         0       1
0x7f9354093000 424790   1     0        1         0       1
...

where:

  • address is the virtual address of the page
  • pfn is the pages page frame number
  • soft-dirty indicates the if soft-dirty bit is set in the pages Page Table Entry (PTE).
  • excl. indicates if the page is exclusively mapped (i.e. page is only mapped for this process).
  • file / shared anon indicates if the page is a file pages or a shared anonymous page.
  • swapped indicates if the page is currently swapped (implies present is zero).
  • present indicates if the page is currently present in the processes resident set (implies swapped is zero).

(Note: I run the example program with sudo as since Linux 4.0 only users with the CAP_SYS_ADMIN capability can get PFNs from /proc/<pid>/pagemap. Starting from Linux 4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN. The reason for this change is to make it more difficult to exploit another memory related vulnerability, the Rowhammer attack, using the information on the virtual-to-physical mapping exposed by the PFNs.)

If you run the example program several times, you should notice that the virtual address of the page should change (due to ASLR), but the PFN for shared libraries that are in use by other processes should stay the same.

If the PFNs for libmyl.so match between the victim and spy program, I would start looking for a reason for why the attack fails in the attack code itself. If the PFNs don't match, the additional bits may give some hint why the pages are not set up to be shared. The pagemap bits indicate the following:

present file exclusive state:
   0      0     0      non-present
   1      1     0      file page mapped somewhere else
   1      1     1      file page mapped only here
   1      0     0      anonymous non-copy-on-write page (shared with parent/child)
   1      0     1      anonymous copy-on-write page (or never forked)

Copy-on-write pages in (MAP_FILE | MAP_PRIVATE) areas are anonymous in this context.

Bonus: To obtain the number of times a page has been mapped, the PFN can be used to look up the page in /proc/kpagecount. This file contains a 64-bit count of the number of times each page is mapped, indexed by PFN.

Related Question