Shared library
A shared library, also known as a dynamic-link library (DLL) on Windows or a shared object (.so) file on Unix-like systems, is a reusable file containing compiled code, data, and symbols that multiple executing programs can load and share in memory at runtime, enabling efficient code reuse without duplication in each executable.[1][2] Unlike static libraries, which are embedded directly into an application's binary during compilation, shared libraries are linked dynamically by the operating system's loader, resolving references to functions and variables only when the program starts or as needed.[1] This mechanism supports position-independent code (PIC), allowing the library to be loaded at arbitrary memory addresses without relocation overhead in most cases.[2] The concept of shared libraries traces its origins to the Multics operating system in the late 1960s, where dynamic linking resolved symbolic references to procedures and variables at runtime using segment tables and linkage pointers, facilitating modular code sharing across processes.[3] This approach influenced early Unix systems, which initially relied on static linking but adopted shared libraries in the 1980s with extensions like those in UNIX System V, evolving further with the Executable and Linkable Format (ELF) standard in the 1990s for Linux and other Unix variants to handle dependencies and relocations more flexibly.[3][2] On Windows, the equivalent DLL format was introduced with Windows 1.0 in 1985, promoting similar runtime loading and sharing.[1][4] Shared libraries offer key advantages, including reduced memory usage by loading a single instance into physical memory for sharing via virtual memory mappings across processes, which is particularly beneficial in resource-constrained environments.[2][1] They enhance modularity and maintenance by allowing libraries to be updated independently without recompiling dependent applications, while supporting features like symbol versioning to maintain backward compatibility through multiple application binary interfaces (ABIs).[2] However, they introduce complexities such as dependency resolution by dynamic linkers (e.g., ld.so on Linux) and potential issues with version conflicts or delayed loading overhead.[2] Today, shared libraries are fundamental to modern operating systems, powering everything from system calls in libc to plugins in applications like web browsers.[1]Fundamentals
Definition and Purpose
A shared library is a file containing executable code and data that can be loaded into memory and used by multiple programs or processes simultaneously, with the library's contents mapped into each process's address space without duplication.[1][5] This design allows the operating system to load a single instance of the library into physical memory, which is then shared across all dependent processes, promoting efficient resource utilization.[6] The primary purpose of shared libraries is to reduce memory consumption, disk storage requirements, and program startup times by eliminating the need to embed duplicate copies of common code within each executable.[1][5] They also facilitate modular software development, enabling developers to update or patch shared components centrally without recompiling or redistributing every application that uses them, which enhances maintainability and supports easier deployment of common functionalities like networking or graphics routines.[6][5] Shared libraries emerged in the 1980s as part of the evolution toward dynamic linking in Unix systems, initially introduced in UNIX System V around 1986 and further developed in SunOS implementations to overcome the inefficiencies of static linking, such as code bloat and redundant memory usage in multi-program environments.[5][7] In the basic workflow, programs reference shared libraries during compilation by including their symbols in the object files, but the actual resolution and loading of the library occur at runtime via a dynamic linker, which binds the necessary addresses when the program executes.[5][1]Types of Libraries
In programming, libraries are categorized primarily into static and dynamic (also known as shared) types, with static libraries serving as archives of object files and dynamic libraries enabling runtime loading. Static libraries are linked during the compilation phase, embedding their code directly into the resulting executable file, which makes the program self-contained and independent of external dependencies at runtime.[8] This approach offers advantages such as portability across systems without needing additional files, but it increases the executable's size since the library code is duplicated in every application that uses it, and updates to the library require recompiling and relinking all dependent programs.[9] Shared or dynamic libraries, in contrast, are loaded into memory at runtime and can be shared among multiple processes, promoting efficient resource use. These libraries remain as separate files outside the executable, allowing a single copy to serve multiple applications simultaneously, which reduces overall storage requirements and facilitates easier updates without altering the executables. Examples include .so files on Unix-like systems and .dll files on Microsoft Windows. However, they introduce runtime dependencies, potentially complicating deployment if the required libraries are missing or incompatible.[9][6] Beyond these core types, libraries may take other forms such as archive libraries, which are essentially static libraries stored in formats like .a files on Unix systems, containing collections of object files (.o) for linking.[10][11] Object files are individual compiled units (e.g., .o files on Unix-like systems) that can be directly linked into executables or archived into static libraries. Runtime libraries, such as variants of the C standard library (e.g., libc on Unix-like systems or the C runtime library on Windows), provide essential functions like input/output and memory management; these can be implemented as either static or dynamic variants to suit different linking needs.[12]| Aspect | Static Libraries | Shared/Dynamic Libraries |
|---|---|---|
| Linking Phase | Compile time; code embedded in executable | Runtime; separate files loaded as needed |
| Memory Impact | Higher usage; full copy per executable | Lower usage; shared across processes |
| Update Mechanism | Requires recompilation of dependents | Independent updates to library files |
| Executable Size | Larger due to included code | Smaller; defers code inclusion |
| Dependencies | None at runtime | Requires library presence at runtime |
Technical Foundations
File Formats
Shared libraries are encoded in platform-specific binary file formats that support dynamic loading and linking. The most common formats include the Executable and Linking Format (ELF) used on Unix-like systems, the Portable Executable (PE) format employed by Microsoft Windows for dynamic-link libraries (DLLs), and the Mach-O format utilized on macOS and iOS for dynamic shared libraries.[13][14][15] These formats share a common architectural foundation consisting of headers, sections, and supporting tables to organize code, data, and metadata. Headers typically begin with a magic number for identification—such as0x7F 'E' 'L' 'F' for ELF files—and include details like the file type (e.g., ET_DYN for shared objects in ELF), machine architecture, and an optional entry point address.[13] Sections delineate distinct content areas: executable code resides in read-only sections like .text (marked with execute permissions in ELF and PE), initialized data in .data, uninitialized data in .bss, and read-only constants in .rodata or equivalent.[13][14][15] Symbol tables, such as .symtab and .dynsym in ELF or export tables in PE's .edata section, catalog function and variable names for linking, while Mach-O uses __nl_symbol_ptr and __la_symbol_ptr sections for non-lazy and lazy symbol pointers.[13][14][15] Relocation tables, found in .rel or .rela sections for ELF, .reloc for PE, and pointer sections in Mach-O, store offset and type information to adjust addresses at load time, enabling dynamic resolution of external references without fixed positioning.[13][14][15]
A key requirement for shared libraries is the use of position-independent code (PIC), which allows the library to execute correctly when loaded at arbitrary memory addresses across multiple processes.[16] Unlike absolute addressing, where code embeds fixed memory locations that demand runtime modifications to the read-only text segment (reducing sharability and incurring overhead), PIC employs relocatable addressing through mechanisms like the Global Offset Table (GOT) and Procedure Linkage Table (PLT) in ELF, base relocations in PE (enabled by the IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE flag), and symbol stubs in Mach-O's __picsymbol_stub section.[13][14][15][16] This approach confines relocations to writable data segments, preserving the immutability of code sections and facilitating efficient memory sharing.[16]
To manage application binary interface (ABI) evolution, ELF incorporates symbol versioning, which tags symbols with version identifiers (e.g., via mapfiles assigning labels like GLIBCXX_3.4) to coexist multiple implementations within the same library file.[17][13] This mechanism supports forward compatibility by allowing new symbols and changes without invalidating existing binaries dependent on prior versions, as the dynamic linker resolves calls to the appropriate versioned symbol.[17] For instance, libraries can append new functionality while retaining stable ABIs through release versioning in filenames and DT_SONAME tags.[17]
Memory Sharing
Shared libraries are loaded into a process's memory by the operating system mapping the library file directly into the virtual address space, typically using themmap system call on Unix-like systems. This mapping treats sections of the library file as if they were part of the process's memory, with the kernel handling the allocation of physical pages on demand. Read-only segments, such as executable code and constant data, are mapped in a shared manner, allowing multiple processes to reference the same physical memory pages without duplication.[18][19]
Process isolation is maintained through separate virtual address spaces for each process, ensuring that one process's modifications do not directly affect others. However, for the shared read-only portions of libraries, the physical memory pages remain common across all processes that load the same library, promoting efficient reuse. Writable data segments, like initialized variables, use a copy-on-write (COW) approach: they are initially mapped as shared read-only, but any write attempt by a process triggers the kernel to create a private copy of the affected page for that process alone, leaving the original intact for others. This mechanism, facilitated by flags like MAP_PRIVATE in mmap, balances sharing with safety.[19][20]
The primary benefit of this memory sharing is a substantial reduction in overall system RAM consumption, as a single copy of the library's code and read-only data serves multiple applications. For example, the standard C library (libc), which provides essential functions like printf and malloc, is commonly shared among hundreds of processes on a typical system, preventing redundant loading that could otherwise multiply memory usage dramatically. However, limitations exist, particularly with thread-local storage (TLS), where per-thread variables in shared libraries face challenges such as static allocation limits, dynamic resizing issues during loading (e.g., via dlopen), and potential failures if the thread control block cannot accommodate additional TLS space across multiple libraries. This memory sharing becomes effective following the dynamic linking phase, where symbols are resolved to enable the mapping.[5][21]
Dynamic Linking
Dynamic linking in shared libraries involves a multi-phase process that defers the final resolution of external symbols until program execution, enabling flexible loading of libraries at runtime. During the compile-time phase, the compiler generates object code with calls to external functions treated as unresolved references, often producing position-independent code (PIC) using flags like-fPIC to facilitate later address adjustments. At link-time, the static linker (e.g., GNU ld) creates an executable file that includes stubs for these external calls, records dependencies on shared libraries via entries like DT_NEEDED in the ELF dynamic section, and sets up structures such as the Procedure Linkage Table (PLT) and Global Offset Table (GOT) for deferred resolution, without embedding the actual library code. This deferred approach contrasts with static linking by avoiding the inclusion of library routines in the executable, thus keeping it smaller and allowing library updates without recompilation.[22][23]
Symbol resolution occurs primarily at runtime by the dynamic linker (e.g., ld.so on Linux), which performs a breadth-first search across the dependency chain of shared objects listed in DT_NEEDED. For each undefined symbol in the executable or a loaded library, the linker consults the dynamic symbol tables (.dynsym sections) and associated hash tables in the libraries, looking up exports in a defined scope that prioritizes the main executable, followed by dependencies in load order. If a symbol remains unresolved after traversing the chain, the program typically fails to start with an error; however, visibility attributes like DF_SYMBOLIC can restrict searches to local scopes for faster resolution in certain libraries. This process ensures that symbols from multiple libraries are correctly mapped, handling interdependencies without manual intervention.[22][23]
Binding modes determine when symbol resolution and associated relocations are performed: lazy binding, the default, delays these actions until the first use of a function (e.g., via a PLT stub that jumps to the dynamic linker for resolution and updates the GOT entry), improving startup time by avoiding unnecessary work for unused symbols. In contrast, eager binding resolves all dynamic relocations immediately upon loading the libraries, triggered by environment variables like LD_BIND_NOW or linker flags such as -z now, which is useful for debugging or security but increases initial load overhead. ELF supports this through specific relocation tables like DT_JMPREL for lazy procedure linkages, separating them from immediate ones.[22][23]
Relocation adjusts code and data references to account for the actual memory addresses where libraries are loaded, which may vary due to address space layout randomization (ASLR) or other factors. Absolute relocations (e.g., R_X86_64_GLOB_DAT in ELF) require full symbol resolution to compute fixed addresses relative to the base load address, making them suitable for global data but costlier at runtime. Relative relocations (e.g., R_X86_64_RELATIVE), computed using offsets known at link-time, are position-independent and faster, as they avoid symbol lookups and remain valid regardless of load position; ELF formats support these via tables like DT_REL and DT_RELA for implicit or explicit addends. The dynamic linker processes these entries in sections such as .rel.dyn or .rela.dyn to patch the in-memory image correctly.[22][23]
Runtime Mechanisms
Locating Libraries at Runtime
When a program requires shared libraries at runtime, the operating system's dynamic linker must locate these files using a prioritized set of search strategies to ensure efficient and correct loading.[18] In Unix-like systems, which commonly employ the Executable and Linking Format (ELF), the dynamic linker such as ld.so follows a conditional search sequence based on the ELF dynamic section attributes: if the DT_RPATH tag is present (and no DT_RUNPATH), it first searches directories listed in DT_RPATH; otherwise, it begins with the user-defined environment variable LD_LIBRARY_PATH (if not in secure-execution mode). Next, if present, it searches DT_RUNPATH directories. The process then consults the ldconfig cache before default system directories.[13][18] These mechanisms are specific to Unix-like systems; other platforms, such as Microsoft Windows, use distinct search orders (see Platform Implementations section). Search paths form the core of library location in these systems. Default directories, such as /lib and /usr/lib (or their 64-bit variants like /lib64), are hardcoded into the dynamic linker and searched last.[18] The environment variable LD_LIBRARY_PATH allows users to specify a colon-separated list of directories searched early in the process (after DT_RPATH but before DT_RUNPATH and defaults), though it is ignored in secure-execution modes for safety.[18] Executables and libraries can embed runtime search paths via the DT_RPATH or DT_RUNPATH tags in their ELF dynamic section: DT_RPATH lists directories with highest precedence (searched before LD_LIBRARY_PATH and applying to the entire dependency tree), while DT_RUNPATH provides similar functionality but is searched after LD_LIBRARY_PATH (thus overridable by it) and applies only to direct dependencies.[13] To accelerate lookups, systems employ caching mechanisms that precompute library locations. On Linux, the ldconfig utility scans specified directories—defined in /etc/ld.so.conf or passed via command line—and builds a binary cache file at /etc/ld.so.cache, which the dynamic linker consults after environment variables and embedded paths (DT_RUNPATH or DT_RPATH) for rapid resolution of library names to full paths.[24] This cache includes symbolic links for versioned libraries (e.g., libfoo.so pointing to libfoo.so.1.2) and is updated periodically, typically by the system administrator, to reflect new installations without runtime overhead.[24] Dependency resolution involves parsing the ELF dynamic section to identify and load prerequisites recursively. The DT_NEEDED tag in an object's .dynamic array lists the names of required shared libraries as null-terminated strings, which the linker processes in order to build a dependency tree; for each, it applies the search paths to locate and load the file, then recurses on that library's own DT_NEEDED entries.[13] This traversal ensures all transitive dependencies are resolved before program execution, using the applicable search paths including embedded ones from the parent object where relevant.[18] Error handling occurs when resolution fails, typically resulting in immediate program termination. If a required library specified in DT_NEEDED cannot be found in any search path or cache, the dynamic linker issues an error such as "file not found" and exits with a non-zero status, preventing execution of potentially unstable programs.[18] Fallbacks are limited; for example, the linker may attempt versioned matches (e.g., seeking libc.so.6 if libc.so is requested) but does not substitute incompatible libraries, emphasizing the need for complete installations.[24]Dynamic Loading
Dynamic loading enables programs to explicitly load shared libraries at runtime under programmatic control, allowing flexibility in extending functionality without requiring recompilation or restart. This mechanism contrasts with implicit linking by providing handles to loaded modules, which can then be queried for symbols and managed independently. On Unix-like systems adhering to POSIX standards, the primary API for this is provided by the<dlfcn.h> header, which includes functions such as dlopen() to load a library and return a handle, and dlclose() to unload it. The dlopen() function accepts a pathname to the shared object file and optional flags like RTLD_LAZY for deferred symbol resolution, ensuring the library is mapped into the process's address space only when invoked.
On Microsoft Windows, the equivalent functionality is offered through the Windows API in libloaderapi.h, where LoadLibrary() or LoadLibraryEx() loads a dynamic-link library (DLL) and returns a module handle, while FreeLibrary() decrements the reference count to facilitate unloading.[25] These functions search for the library using standard paths, such as the system directory or application directory, as a prerequisite for loading.[25]
Once a library is loaded, programs retrieve addresses of functions or variables via symbol lookup APIs: dlsym() on POSIX systems, which takes the handle from dlopen() and a symbol name to return a pointer, and GetProcAddress() on Windows, which uses the module handle from LoadLibrary() to obtain the procedure address.[26] This allows direct invocation of library code, such as casting the returned pointer to the expected function type in C or C++ programs.
Common use cases for dynamic loading include implementing plugin architectures, where a host application scans a directory and loads extension modules on demand to add features like image filters in graphics software; just-in-time loading of optional components to reduce initial memory footprint, such as cryptographic libraries loaded only when encryption is needed; and hot-swapping libraries for updating functionality without restarting the process, as seen in database servers or web browsers.[27]
Unloading shared libraries employs reference counting to ensure safe memory management: both POSIX dlclose() and Windows FreeLibrary() decrement a per-module count maintained by the dynamic linker, and the library is only unmapped from memory when the count reaches zero, preventing premature deallocation if multiple components reference it. This mechanism avoids dangling pointers and resource leaks, though developers must track handles carefully to avoid over-unloading.[27]
Platform Implementations
Unix-like Systems
In Unix-like systems, shared libraries are commonly distributed as files with the.so extension, following the shared object naming convention.[28] These files include a special identifier known as the soname, which encodes versioning information to ensure binary compatibility across updates; for example, the C standard library is often named libc.so.6, where 6 represents the major interface version, allowing programs linked against it to continue functioning even if minor updates occur without breaking the ABI.[28][29] The soname is embedded during compilation and serves as a logical name for the dynamic linker to resolve dependencies at runtime, preventing mismatches between library versions and linked executables.[28]
Several tools facilitate the creation, inspection, and management of shared libraries. The ld linker, part of the GNU Binutils suite, combines object files into shared libraries by resolving symbols and generating the necessary dynamic sections, often invoked via compiler flags like -shared in GCC.[30] For examining dependencies, the ldd utility lists the shared libraries required by an executable or another library, querying the dynamic linker to display resolved paths and versions without executing the program.[31] Inspection of library internals, such as symbols, sections, and relocations, is performed using objdump, which disassembles and extracts metadata from ELF object files, aiding in debugging and verification.[32]
The dynamic loading process is handled by the runtime linker, typically ld.so on generic Unix-like systems or ld-linux.so variants on Linux architectures, which searches standard paths like /lib and /usr/lib to load and relocate shared libraries into the process address space.[18] Environment variables provide fine-grained control; for instance, LD_PRELOAD allows preloading specific libraries before others, overriding default symbols for debugging or testing, while LD_LIBRARY_PATH extends the search path for non-standard locations.[18] These mechanisms enable flexible runtime behavior without recompiling executables.
Advanced features enhance security and observability. The LD_AUDIT environment variable supports audit modules by loading specified shared objects that intercept linker events, such as library loading or symbol resolution, through callbacks defined in the GNU C Library's runtime linking interface; this is useful for tracing, profiling, or custom interventions but is disabled in secure-execution modes to prevent abuse.[33][18] For protection against exploits targeting relocations, RELRO (Relocation Read-Only) marks the global offset table (GOT) and other relocation sections as read-only after initial linking, mitigating attacks like GOT overwrites; partial RELRO applies lazily per relocation, while full RELRO processes all at startup for stronger hardening, enabled via compiler flags like -Wl,-z,relro,-z,now.[34]
Shared libraries in these systems are based on the Executable and Linkable Format (ELF), which structures the file with sections for code, data, and dynamic linking metadata.[30]