Programming Persistent Memory: A Comprehensive Guide for Developers

Index pmempool utility features, 92 TOIDs, 92, 93 libpmemlog, 69 libpmemobj-cpp, 68 libpmemobj, 141 bindings, 128 libpmemobj architecture definition, 111 goal, 111 ACID transactions, 328, 329 librpmem architecture atomic and transactional APIs, 315 active-passive replication internal allocator interface, 323, 324 memory allocation, 324–328 architecture, 359 Memory OID components, 360, 361 error handling, 364 ASLR, 316 Hello World, 369, 370 convert pointer, PMEMoid, 318 Hello World, pointer performance replication, 364, 366, 367 problems, 316, 317 latency, 362 pointer requirements, 316 libpmemobj library, 359 modules, 313, 314 poolset, 362 OID, 314 pull model, 363 PMEMoids, 315 RDMA architecture, 360 power fail atomicity remote_open routine, 368, 369 redo logging, 320, 321 remote persistent memory, 363 Undo logging, 321 rpmemd program, 359 store runtime (volatile) state, 330 synchronous replication model, 363 thread local storage, libsafec, 404 libvmem, 67 using lanes, 318, 319 libvmemcache, 66, 178 libpmemobj library, 359 application, 179 characteristics, 178 allocate memory, 93 design data APIs aspects, 180 atomic operations, 94, 95, 97 extent-based allocator, 180, 181 benefits, 104 noncontiguous allocations, 181 optinal flags, 104 functions, 183 reserve/publish, 97–100 working, 179 transactional, 100–102, 104 Linux Volume Manager (LVM), 384 debugging and error handling, 106–108 loadIndexTableFromPersistentMemory() defining, 81 function, 267 memory pools Low-Level Persistence Library (LLPL), 418 pmemobj_create(), 84, 86–88 pmempool utility, 83 read and display string, 88, 90 memory poolset, 90–91 pmemobj_set_funcs() function, 106 432

M Index Machine check exception (MCE), 334 capacity, 11 main() function, 213, 234 characteristics, 12, 13 MAP_SYNC flag, 385 kinds of, 158 MariaDB∗ storage engine leaked object, 213 leaks, 209 architecture, 264 Memory management unit (MMU), 49 creation Metaprogramming allocating, 116–118 database table, 266, 267 definition, 112 database table, closing, 268 persistent pointers, 112, 113 database table, snapshots, 115, 116 transactions, 113, 114 opening, 267, 268 mtx object, 282 data structure, 266 Multiple Device Driver (mdadm), 384 DELETE operation, 272–274 Mutexes handle commits and libpmemobj library, 283 main() function, 285 rollbacks, 265, 266 std::mutex, 282 INSERT operation, 268–270 synchronization primitives, 282–284 new handler instance, 265 SELECT operation, 275, 276 N UPDATE operation, 270, 271 storage layer, 264 ndctl and daxctl, installation memkind API functions, 159 Linux distribution package repository fixed-size heap creation, 160, 161 PMDK on Fedora 22, 393 kind creation, 160 PMDK on RHEL and kind detection, 162, 163 CentOS, 393–394 memory kind detection API, 163 PMDK on SLES 12 and variable size heap creation, 162 OpenSUSE, 394 memkind_config structure, 161 PMDK on Ubuntu 18.04, 394 memkind_create_pmem() searching, packages, 391–392 function, 160, 169 memkind_create_pmem_with_config() prerequisites, 389–390 function, 161, 164 ndctl utility, 376 memkind_destroy_kind() Network interface controller (NIC), 349 function, 164 Non-maskable interrupt (NMI), 18 memkind_detect_kind() function, 163 Nonuniform memory access memkind_free() function, 166 memkind library, 156, 157 (NUMA), 65, 156 memkind_realloc() functions, 165 automatic balancing, 382, 383 Memory 433

Index memory-mapped files on Linux, 36, 37 Nonuniform memory access with storage, 42 (NUMA) (cont.) on Windows, 38, 39, 41 BIOS options, 382 persistent memory, block storage, 33 CPU, 373 persistent memory, file systems, 34 defined, 373 Optimized flush instructions, 355 Intel MLC, 378, 379 Ordering ipmctl utility, 381 nodes, 20, 21 ndctl utility, 376, 378 steps, 20 numastat utility, 380 two-socket system, 373, 374 P Nonuniform Memory Architecture paintball_init() function, 97 (NUMA), 305 Patrol scrub, 337 Non-volatile memory express Performance difference, 6 Persistent libraries, 64 (NVMe), 156 Persistent memory, 173 numactl utility, 374 numastat utility, 380 advantages, 262 NVDIMM driver, 33 application, 176 NVDIMM Firmware Interface Table data structures, 188, 189 selective, 193 (NFIT), 343 snapshotting O performance, 190, 191 uses, 262, 263 Object identifier (OID), 314 Persistent Memory Development Kit open()method, 267 Operating systems (PMDK), 1, 13, 111, 207, 261, 307 persistent libraries, 63 memory direct access (DAX) benefits, 49 libpmem, 67, 68 I/O-accessed storage, 50 libpmemblk, 69 libpmem, 50 libpmemkv, 69 Linux, locating, 48 libpmemlog, 69 physical devices, regions, and libpmemobj, 68 namespaces, libpmemobj-cpp, 68 displaying, 44–46, 48 tools and command utilities pmem_map_file function, 52 pmemcheck utility, 70, 71 pmem_persist function, 53 pmempool utility, 70 programming example, 51 pmreorder utility, 71 system administrator, 43 Windows, locating, 49 434

volatile libraries Index libmemkind, 65 libvmem, 67 isolation, 58 libvmemcache, 66 func() function, 237 lambda functions, 235 Persistent memory programming multithreaded applications, 235 application’s responsibilities, 59 my_root data structure, creation, 229 AMD64 architectures, 247 object adding, 231 analyzing process, 257 persistence inspector, 230, 233 atomic persistent store, 56 pmemcheck code, 229, 232 CLFLUSH instruction, 217, 247 recovery operation, 235 code modification, 227 report, 239 data collection, 254 rollback mechanism, 234 data() function, 249 threads, 236 data structure, 252, 253, 256 Valgrind macros, 216, 217 dependency, flush, 244 valid flag approach, 219 flag/data variables, 224 Persistent queue implementation, 129 FlushFileBuffers on Windows, 59 Platform capabilities, 25, 26 GUI of Intel Inspector, 225, 226 Platform support, 13 hardware configuration, 60 PMDK, installation Intel inspector, 219, 220, 222, 223 Linux distribution package repository memory overwriting, 240–242 libraries, PMDK v1.6, 396 memory variables, 249 naming convention, 396 nonpersistent stores, 214 PMDK on Fedora 22, 399 out-of-order issue, 249–251 PMDK on Microsoft Windows, 402 output_file.log, 258, 259 PMDK on RHEL and persistent memory, 220, 221 pmemcheck code, 215, 217, 219 CentOS 7.5, 400 reader program, 245, 246 PMDK on SLES 12 and redundant flushing, 242, 243 reordering mechanism, 255 OpenSUSE, 400–401 resident data structures, 55 PMDK on Ubuntu 18.04, 401 SFENCE instruction, 247, 248 searching packages,396–398 storage-resident data structures, 56 prerequisites, 395 transactions pmem::allocator class template, 169–171 atomicity, 57 pmem_allocator.h header file, 170 consistency, 58 pmemcheck utility, 70, 208 database system, 233 .pmeminspdata directory, 222 durability, 58 Persistent memory (PMEM_KIND), 158 Pmemkv architecture characteristics, 144 creation, 143 435

Index SFENCE operation, 18 SNIA NVM programming model, 17 Pmemkv architecture (cont.) Profiling and performance optimization direct access, 144, 145 JavaScript bindings, 151, 152 techniques phonebook example, 147–151 computer bound vs memory pmemkv_config structure, 145 pmemkv_config.h, 146 bound, 295 programming languages, 144 default memory page size, 311 DRAM-only system, 302 pmemkv_config structure, 145 guided data placement, 307 pmem::kv::db class, 149, 150 Intel MLC, 304 pmemkv_errormsg() function, 145 I/O bound, 297 pmem_map_file function, 52 memory access optimization, 308 pmem_memcpy_persist() memory access pattern, 296 memory-capacity constraints, 308 function, 77 memory latency vs memory pmemobj pool, 197 pmemobj_close() function, 268 capacity, 296 pmem::obj:: condition_variable NUMA class, 283 addressable memory capacity, 306 pmemobj_create() function, 84, 266 bandwidth requirements, 307 pmemobj_direct() function, 87 BIOS options, 307 pmemobj_errormsg() function, 106 data allocation vs access, 309 pmem::obj::mutex class, 283 QPI/UPI, 305 pmemobj_open() function, 267 thread migration, 310 pmem::obj::persistent_ptr tuning hardware, 306 persistent memory technologies class, 113, 125 use cases, memory-capacity pmem::obj::pool_base, 130 pmemobj_root() function, 87 bound, 298 pmemobj_set_funcs() use cases, VTune function, 106 Profiler, 301, 302 pmem::obj::shared_mutex class, 283 use cases, WSS, 300, 301 pmem::obj::timed_mutex class, 283 read vs. write performance, 296 pmem::obj::v<T> class, 284 read–write ratio, 305 pmem_persist function, 53 working set size and memory pmempool utility, 70, 83 pmreorder utility, 71 footprint, 305 pop() method, 130 workload characteristics, 303 Power-fail protected domains, 16 pull requests, 142 push() method, 129 ADR and eADR, 18 put() method, 150 challenge, 18 436

Q Index Queue implementation, 126, 128 Reliability, availability, QuickPath Interconnect (QPI)/Ultra Path serviceability (RAS), 27 Interconnect (UPI), 305 Remote direct memory access (RDMA), 12, 347, 348 R software architecture, 357, 358 RDMA networking protocols remote_open routine, 368, 369 commands, 350 remove() method, 150 NIC, 349 Resource acquisition is initialization RDMA Read, 350 RDMA Send (and Receive), 350, 351 (RAII), 113 RDMA Write, 350 rpmem_drain(), 363 rpmem_flush(), 363 RDMA over Converged Ethernet (RoCE), 348 S Redis, 143 show() method, 131 Redo logging, 320 Single instruction, multiple data (SIMD) Reliability, Availability, and processing, 190 Serviceability (RAS) Single linked list (SLL), 266 device health Snapshotting optimization, 196 SNIA NVM programming ACPI NFIT, 343 ACPI specification, 342 model, 351 unsafe/dirty shutdown, 343 Solid-state disk (SSD), 1, 156 using ndctl to query, 340, 341 Stack/buffer overflow bug, 208 vendors, 342 Stackoverflow app, 211 ECC Standard Template Library inifinite loop, 334 MCE, 334 (STL), 168, 282, 287 using Linux, 335, 336 std::is_standard_layout, 122, 123 unconsumed uncorrectable error Storage and Networking Industry handling Association (SNIA), 33, 112 ARS, 338 symmetric multiprocessing (SMP) clearing errors, 339 memory root-device notification, 338 system, 373 petrol scrub, 337 runtime, 337 T unsafe/dirty shutdown, Thread migration, 310 DLC counter, 344, 345 Transactions and multithreading counter, 281 illustrative execution, 281 437

Index V Transactions and multithreading (cont.) Valgrind tools, 208, 209 incorrect synchronization, 279, 280 Vector of strings, 171 libpmemobj, 280 Versioning, 193 PMDK transactions, vmemcache_add() function, 185 278, 279, 281, 282 vmemcache_get() function, 185 Volatile libraries, 64 Translation lookaside buffer Volume managers (TLB), 187, 311, 325 advantages, 383 Tree insert operation, 204 Linux architecture, 384 Typed Object Identifiers mdadm, 384 NUMA systems, 383 (TOIDs), 92 W, X, Y, Z U Write pending queue (WPQ), 18 Undo logging, 321 write_row() method, 268 Uninterruptable power supply (UPS), 18 update_row() method, 270 438