Apama
Apama is a complex event processing (CEP) and event stream processing (ESP) platform that enables real-time monitoring, analysis, and response to high-volume streams of events from sources such as financial markets, Internet of Things (IoT) devices, and operational data feeds.[1] Developed initially for algorithmic trading and risk management in capital markets, it processes streaming data at ultrafast speeds, detects complex patterns, and integrates with historical databases for context-aware decisions.[2][3] Founded in 1999 in Cambridge, United Kingdom, by Dr. John Bates, Dr. Giles Nelson, and Dr. Mohamad Afshar—PhD graduates from the University of Cambridge—Apama originated as a startup focused on innovative solutions for analyzing and acting on high-speed data streams in quantitative finance.[2][4] The company was acquired by Progress Software Corporation in 2005 for approximately $25.4 million, which expanded its reach into broader enterprise event processing applications.[2] In 2013, Progress sold Apama to Software AG for an undisclosed amount. In 2024, IBM acquired Software AG's StreamSets and webMethods businesses, including Apama, for €2.13 billion, integrating it into IBM's portfolio of real-time analytics and integration solutions. As of 2015, it had supported over 200 deployments in Fortune 500 companies across industries like logistics, manufacturing, and telecommunications.[5][6][7] Key features of Apama include its Event Processing Language (EPL), a declarative language for defining event-driven applications, and support for predictive analytics, machine learning integration, and hybrid cloud deployments.[8] It handles massive scalability, processing millions of events per second with sub-millisecond latency, and connects to diverse data sources via adapters for protocols like MQTT and Kafka.[6] Apama's architecture separates event processing from application logic, allowing developers to build scenarios that trigger automated actions, such as fraud detection in banking or predictive maintenance in industrial settings.[9] As of 2023, the platform's version 10.15 emphasized containerization via Docker for modern DevOps workflows, with maintenance ending in October 2025.[10][11]History
Founding and early years
Apama Ltd. was founded in late 1999 by Dr. John Bates and Dr. Giles Nelson, two PhD graduates from the University of Cambridge's Computer Laboratory who had collaborated on research there.[12][13] The pair established the company in Cambridge, UK, to commercialize advancements in real-time event processing, drawing from their academic work on handling complex data streams and patterns.[14] Their vision centered on enabling businesses, particularly in high-stakes environments, to detect and respond to events in live data flows with unprecedented speed and accuracy.[15] In its formative phase, Apama focused on developing a complex event processing (CEP) platform tailored for the financial sector, where rapid analysis of market data was critical for algorithmic and high-frequency trading.[16] The inaugural release of the Apama software occurred in 2001, introducing patented technologies for processing and acting on high-velocity event streams in real time.[15][16] This innovation addressed key challenges in capital markets, such as detecting patterns across disparate data sources to support automated decision-making and risk management.[14] During the early 2000s, Apama rapidly gained traction among financial institutions, establishing itself as a leader in CEP for trading applications.[16] The platform's event-driven architecture allowed for low-latency processing, which was essential in volatile markets, and it began incorporating features like graphical development tools to broaden accessibility beyond pure developers.[15] By 2005, the company's growth in this niche had positioned it for expansion, culminating in its acquisition by Progress Software Corporation for $25.4 million.[15]Acquisitions and evolution
Apama was founded in 1999 in Cambridge, United Kingdom, by John Bates and Giles Nelson, two PhD graduates from the University of Cambridge, with a focus on developing technology for analyzing and acting on live data streams in real time.[17] The company introduced its complex event processing (CEP) platform commercially in 2003, signing its first customer and establishing itself as a pioneer in high-speed event-driven applications, particularly for financial markets.[18] In April 2005, Progress Software Corporation acquired Apama for $25.4 million in cash, integrating it into its Real Time Division to enhance offerings in algorithmic trading and low-latency data processing.[15] Under Progress, Apama evolved as a key component for event stream processing, with Bates joining as vice president and later becoming chief technology officer, driving advancements in areas like capital markets trading and risk management.[19] The platform saw expanded use in proprietary trading algorithms, benefiting from Progress's resources to scale deployments in high-volume environments.[12] By 2013, Progress sought to refocus on cloud and mobile development, leading to the sale of Apama to Software AG on July 16, 2013, for an undisclosed amount estimated at around $44 million.[5][20] This acquisition bolstered Software AG's "big data in motion" strategy, combining Apama's low-latency CEP with the company's existing Terracotta in-memory technology and messaging solutions for enhanced real-time analytics.[21] Bates transitioned to lead Apama as a distinct business unit within Software AG, retaining the brand and expanding its teams across Cambridge (UK), Bedford (USA), and Hyderabad (India).[5] Post-acquisition, Apama evolved beyond its financial roots into a broader streaming analytics platform, integrating with Software AG's Digital Business Platform to support IoT and enterprise applications. In 2014, it began addressing non-financial use cases, such as fraud detection and customer experience optimization.[22] By November 2015, release 9.9 introduced predictive analytics capabilities, allowing users to forecast events using historical and real-time data for scenarios like predictive maintenance, alongside support for IoT protocols including MQTT and AMQP.[6] Subsequent releases, such as version 10.15 in October 2022 (with patch 10.15.3 in May 2023), added features for containerized deployments via Docker and enhanced event model processing. In April 2025, Apama released version 26.x, featuring upgrades to Java 17, Python 3.13, and Debian 12 base images for improved compatibility and performance.[23][24] These developments maintain active support through at least 2026 for prior versions. Following Silver Lake's 2023 acquisition of Software AG, Apama remained part of the retained application modernization portfolio, unaffected by the sale of other units like webMethods to IBM (completed July 2024).[25][26]Overview
Core purpose and capabilities
Apama is a real-time event processing platform designed for complex event processing (CEP) and streaming analytics, enabling organizations to monitor high-velocity event streams, detect patterns, and derive actionable insights instantaneously.[1] At its core, it facilitates event-driven architectures that allow digital businesses to identify changes in data flows, assess their implications, and respond swiftly to opportunities or risks across diverse domains such as finance and IoT.[27] This capability is powered by the Apama Correlator, a high-performance engine that processes large volumes of inbound events—typically delivered via messaging infrastructures—with sub-millisecond latency, supporting scenarios involving millions of concurrent data sources.[28] Key capabilities include advanced pattern detection through the Event Processing Language (EPL), a declarative language optimized for defining complex rules and correlations over streaming data, which outperforms traditional imperative languages like Java or C++ in tasks such as real-time option pricing benchmarks.[27] Apama supports graphical development tools accessible to business users, allowing the creation of analytics scenarios without deep coding expertise, alongside integration with Java for custom extensions.[28] The platform employs a HyperTree indexing structure for multi-dimensional event filtering, ensuring logarithmic scalability as event volumes grow, and can handle 1 million to 10 million concurrent users while maintaining stable response times.[27] Integration features enable seamless connectivity to external systems through the Integration Adapter Framework (IAF), plug-ins, and software development kits (SDKs) for C, C++, Java, and .NET, facilitating the ingestion of data from sources like IoT devices, financial feeds, or enterprise messaging protocols such as JMS.[27] Productivity tools, including real-time dashboards and event replay mechanisms, allow for monitoring, simulation, and optimization of applications, while resiliency options like snapshots ensure fault-tolerant operation in distributed environments.[29] Overall, these elements empower Apama to bridge streaming and static data for continuous analytics, automating decisions in latency-sensitive contexts without compromising on precision or throughput.[1]Technical specifications
Apama is an event-driven platform designed for real-time complex event processing, featuring a core correlator engine that processes high-velocity event streams with sub-millisecond latency for pattern detection and response.[28] The system supports scalability through distributed architectures, handling tens of thousands of concurrent monitoring scenarios across multiple correlators, with workload partitioning via configurable rules.[28] Key technical components include the Event Processing Language (EPL) for application development, in-memory data stores like MemoryStore for cross-monitor data sharing, and connectivity plug-ins for integration with external systems such as Kafka or MQTT.[28]System Requirements
Apama requires approximately 2.5 GB of disk space for a full installation, including the Software AG Designer IDE on Windows, with about 1.5 GB actively consumed; additional space is needed for deployed applications and logs.[30] The platform operates on 64-bit architectures, with the system clock required to advance monotonically without jumps or reversals, typically managed via NTP synchronization.[30] Without a license, usage is limited to 1024 MB of memory, 4 EPL processing threads, 20 contexts, and 5 persistent monitors.[28] Supported operating systems for Apama 10.15 include:| Operating System | Architecture | Notes |
|---|---|---|
| Microsoft Windows 11 | x86-64 | - |
| Microsoft Windows 10 | x86-64 | - |
| Microsoft Windows Server 2022 | x86-64 | - |
| Microsoft Windows Server 2019 | x86-64 | - |
| Raspberry Pi OS 2021-03-04+ | ARMv7HF (32-bit) | - |
| Red Hat Enterprise Linux 9+ | x86-64 | SELinux must be disabled |
| Red Hat Enterprise Linux 8+ | x86-64 | SELinux must be disabled |
| Ubuntu 20.04 LTS | x86-64 | SELinux must be disabled |
Event Processing Language (EPL)
EPL is Apama's native, domain-specific language for defining event-driven applications, using a syntax that supports monitors as the primary unit of execution for pattern detection and response.[32] It enables concurrent processing through contexts, with event expressions for temporal and logical correlations using operators likefollowed-by, and, and or.[32] Monitors are defined with lifecycle actions such as onload(), ondie(), and onunload(), and can be marked persistent for state retention across restarts.[32]
EPL supports primitive types (boolean, decimal with 16-digit precision, float per IEEE 754, 64-bit integer, UTF-8 string) and reference types including dictionary, sequence, event, and context, all with automatic garbage collection.[32] Events are defined with fields (up to 65,536 per type, with 32 indexes maximum) and actions, published via emit or send with millisecond-resolution timestamps.[32] The language includes built-in aggregates like avg() and sum(), stream queries for data flow processing, and concurrency via context creation, with methods such as getId() and isPublic() for management.[32] Files use .mon extension for EPL monitors and .evt for event definitions.[28]
Performance and Scalability
The correlator engine achieves sub-millisecond event detection latency, processing rapidly moving streams for applications in finance and IoT.[28] Scalability is provided through clustering with Software AG Terracotta Store 10.15 or BigMemory Max 4.4.0 for distributed in-memory storage, enabling horizontal scaling across nodes.[31] C++ plug-ins and connectivity extensions can be built with GCC 8.3/8.4 or Microsoft Visual Studio 2019, supporting custom integrations without restricting to specific compilers beyond C++11 compatibility.[31]Architecture
Core engine and components
The core engine of Apama is the correlator, a high-performance, real-time event processing engine that executes complex event processing (CEP) logic defined in the Event Processing Language (EPL).[28] It employs a patented, in-memory architecture optimized for sub-millisecond responsiveness, enabling the handling of millions of events per second across distributed environments.[28] The correlator processes inbound event streams, correlates patterns using temporal and spatial rules, and triggers actions such as alerts or automated responses.[28] Key internal components of the correlator include the HyperTree structure, which facilitates efficient multi-dimensional matching of events against predefined patterns; the Temporal/Stream Sequencer, responsible for managing time-based correlations and stream ordering; the EPL Virtual Machine, which interprets and executes EPL code; and dedicated event input/output queues that buffer and route data to ensure low-latency processing.[28] These elements work in tandem within an event-driven model, where events are injected via connectivity interfaces, analyzed in real time, and output to external systems or dashboards.[28] Surrounding the correlator are supporting components such as monitors, which encapsulate EPL logic including event listeners for pattern detection and actions for response execution; the MemoryStore, an in-memory, table-based repository accessible by all monitors for shared state management; and streams, which represent ordered sequences of events or data items processed via stream queries and listeners.[28] Connectivity plug-ins, implemented in Java or C++, integrate the correlator with external sources like messaging systems (e.g., Kafka, MQTT) or databases, ensuring reliable bidirectional event flow.[28] For visualization and control, Apama includes dashboard components built with the Dashboard Builder tool and viewed through the Dashboard Viewer, which connect to the correlator via a data server to display real-time analytics and KPIs.[28] This modular architecture supports scalability through clustering and partitioning, allowing multiple correlators to distribute workload while maintaining consistent event processing.[28]Event model and processing
Apama's event model represents real-world changes as discrete events, each consisting of a collection of attribute-value pairs that capture relevant data, such as a stock price update with fields for symbol, price, and volume. These events are formally defined in the Event Processing Language (EPL) using declarative syntax, for example:event StockTick { string [symbol](/page/Symbol); float [price](/page/Price); float [volume](/page/Volume); }, allowing for structured, type-safe handling of diverse data types including primitives, dictionaries, and sequences. This model supports multi-dimensional event types to facilitate complex queries and correlations across multiple attributes, enabling efficient representation of both simple notifications and composite patterns derived from raw inputs.[28][27]
The core of event processing in Apama is the correlator, a high-performance runtime engine that ingests, analyzes, and responds to event streams in real time. Incoming events from external sources—translated via connectivity plug-ins such as those for JMS, Kafka, or MQTT—are routed into the correlator, where they are matched against user-defined patterns specified in EPL monitors. The correlator employs specialized data structures for efficiency: the HyperTree algorithm performs multi-dimensional indexing and filtering on event attributes, achieving logarithmic-time pattern matching even at high volumes (millions of events per second), while the Temporal Sequencer manages time-ordered correlations to enforce constraints like event sequences within specific windows (e.g., 5 minutes). This architecture ensures sub-millisecond latency for detection and response, processing events asynchronously to handle bursts without blocking.[28][27]
Processing logic is encapsulated in monitors, which are modular EPL components that define event listeners to detect patterns and trigger actions upon matches. Listeners can target individual events, sequences, or streams; for instance, on all StockTick(symbol="[IBM](/page/IBM)", price >= 75.5, *) { log "Threshold reached"; } continuously monitors for qualifying trades and executes imperative actions like sending alerts or updating shared state via the MemoryStore. Complex event processing (CEP) extends this with temporal and spatial operators, such as every StockTick(a) within 60.0 followed by StockTick(b) where b.[price](/page/Price) > a.price * 1.05, to identify derived events like rapid price surges. Streams further enhance processing by treating event sequences as continuous flows, where queries apply aggregations (e.g., averages over sliding windows) to produce derived outputs, and dynamic windows (tumbling or hopping) limit scope for computational efficiency.[8][28]
For scalability, the correlator supports parallel execution across multiple contexts or instances, partitioning event streams to distribute load while maintaining global state synchronization through features like replicated MemoryStores. This model prioritizes deterministic, rule-based evaluation over procedural loops, inverting traditional polling approaches to reactively process only relevant events, which minimizes resource use in high-throughput environments like financial trading or IoT sensor networks. All processing remains stateless by default per monitor instance unless explicitly managed, ensuring fault tolerance and hot-deployability without downtime.[27][28]
Features
Streaming analytics and pattern detection
Apama's streaming analytics capabilities enable the real-time processing of high-volume event streams from diverse sources, such as financial markets, IoT devices, and sensor networks, to deliver actionable insights with minimal latency.[1] The platform employs complex event processing (CEP) techniques to filter, aggregate, enrich, and analyze asynchronous data flows, identifying correlations, anomalies, and trends as events occur.[33] This allows organizations to respond swiftly to dynamic conditions, such as adjusting IoT device operations or triggering trading alerts, by sifting through rapidly moving streams without storing all data in traditional databases.[33] At the core of Apama's pattern detection is its Event Processing Language (EPL), a declarative syntax used within monitors to define event listeners that observe and match patterns in incoming streams.[1] An event listener, specified via anon statement, continuously monitors events until a specified pattern is detected, at which point it executes a predefined action, such as logging an alert or invoking an external service.[34] Patterns can range from simple single-event matches—using templates with wildcards like StockTick(*,*)—to complex sequences incorporating temporal constraints, logical operators, and aggregations.[34]
For instance, a listener might detect a sequence where a NewsItem event is followed by a StockTick showing a 5% price increase within 300 seconds, enabling real-time market surveillance.[33] Operators like -> (followed-by), all (repeating matches), and/or/not (logical combinations), and within (time windows) facilitate sophisticated detection, such as identifying fraud through anomalous transaction clusters or predictive maintenance signals from sensor data patterns.[34] Monitors can spawn parallel instances to handle concurrent event processing, ensuring scalability for high-throughput environments while maintaining zero effective latency through self-optimizing pattern evaluation.[35]