Use raw pointers for potentially racy loads (#233)

Shared references assert immutability, so any concurrent access would be UB
disregarding data race concerns.