elf: do not access objects with preemption disable

program::lookup and program::lookup_addr were written for optimal
performance and look-freedom by using the RCU lock to protect the short
module iteration. However, with demand-paged file mapping, the actual
objects are demand-page, and we cannot assume that object memory can be
read without sleeping.

So switch these functions to use with_modules, the more general, and
somewhat slower, technique to iterate over modules. With with_modules,
the RCU lock is only used to copy the list of modules, and the protection
against the individual modules being deleted is done separately (by
incrementing a reference count preventing any module from being deleted).

This makes symbol lookup a bit slower, but symbol lookups are relatively
infrequent (usually occur during early application startup, the first time
each symbol is used), and measuring Java's startup time, as an example,
I didn't see any measureable slowdown.

Fixes #169.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

core/elf.cc

@@ -932,15 +932,16 @@ void program::del_debugger_obj(object* obj)
 symbol_module program::lookup(const char* name)
 {
     trace_elf_lookup(name);
-    SCOPE_LOCK(osv::rcu_read_lock);
-    for (auto module : _modules_rcu.read()->objects) {
-        // NOTE: We are in rcu_read_lock, so code below must not sleep or
-        // malloc. If it does, we'll need to switch to use with_modules()s
-        if (auto sym = module->lookup_symbol(name)) {
-            return symbol_module(sym, module);
+    symbol_module ret(nullptr,nullptr);
+    elf::get_program()->with_modules([&](const elf::program::modules_list &ml)
+    {
+        for (auto module : ml.objects) {
+            if (auto sym = module->lookup_symbol(name)) {
+                ret = symbol_module(sym, module);
+            }
         }
-    }
-    return symbol_module(nullptr, nullptr);
+    });
+    return ret;
 }
 
 void* program::do_lookup_function(const char* name)
@@ -958,16 +959,18 @@ void* program::do_lookup_function(const char* name)
 dladdr_info program::lookup_addr(const void* addr)
 {
     trace_elf_lookup_addr(addr);
-    SCOPE_LOCK(osv::rcu_read_lock);
-    for (auto module : _modules_rcu.read()->objects) {
-        // NOTE: We are in rcu_read_lock, so code below must not sleep or
-        // malloc. If it does, we'll need to switch to use with_modules()s
-        auto ret = module->lookup_addr(addr);
-        if (ret.fname) {
-            return ret;
+    dladdr_info ret;
+    elf::get_program()->with_modules([&](const elf::program::modules_list &ml)
+    {
+        for (auto module : ml.objects) {
+            ret = module->lookup_addr(addr);
+            if (ret.fname) {
+                return;
+            }
         }
-    }
-    return {};
+        ret = {};
+    });
+    return ret;
 }
 
 program* get_program()
@@ -1175,10 +1178,6 @@ void program::module_delete_enable()
         _modules_to_delete.clear();
     }
 
-    // Need to delay the deletion a bit more, as we might have a
-    // program::lookup() in progress.
-    osv::rcu_synchronize();
-
     for (auto ef : to_delete) {
         ef->unload_segments();
         delete ef;