Introduction As organisations grapple with cybersecurity landscapes that only appear to worsen, they are turning to large volumes of their own data for insights to protect themselves.  Federating enterprise data to cloud storage solves the problem of containing the data, but gaining value from the data depends on the approach taken to federation.  The previous article in this two-part series identified four vendor approaches to federating data, and their appealing elements.  This article identifies some drawbacks of each federation approach, and steps to manage associated federation challenges. An understanding of the challenges helps enterprises to avoid unfulfilled expectations as they choose data federation approaches to underpin data analytics and risk management. Close-Coupled Federation [Centralised normalised data, localised for access] In this type of federation, we discussed the fact that elastic cloud storage becomes a de-coupled extension of the finite storage local storage, or cache. However, the cloud storage introduces latency when the requirement is speed, so in this architecture, there are two mechanisms to keep data local even while federation is in use.  Firstly, keeping recent and more frequently searched data in a local cache, while leaving space in the cache to retrieve older and less frequently searched data from its federated location.  Secondly, moving out older and less frequently searched data from the cache to the federated location for retrieval if needed.  With management logic governing the mechanisms, this approach can keep the benefits of local storage while leveraging federation. Drawbacks In scenarios like threat hunting, if data that is older than a few months is frequently being searched, the size of the local data storage must be designed to be large from the start, to accommodate the large volumes that will be copied to local storage.  This causes a corresponding increase in both local storage and compute, which reduces the cost-effectiveness of the architecture. In scenarios such as forensics, if a large amount of old data suddenly needs to be queried and correlated (say, three months of data from multiple sources) the close-coupled federation architecture may have storage contention issues. This is because the cache is typically sized to retain recent data and frequently queried data, say data from the current month to six months past.  If, for instance, most data for months six to nine is being queried, a vicious cycle may arise where the cache allocated is constantly swapping data in and out of the federated storage.  This is because a proportion of more recent data, whose location should be stable due to frequent querying, will be moved out to accommodate the older data for forensic queries, but then the older data will be removed when more recent data is queried.  The cycle repeats, which can waste compute cycles and slow down analytics. Managing Close-Coupled Federation Considering the drawbacks, enterprises must: size the local storage well for their use-cases, use resource management tools in their federation to prevent queries whose time frame or expected result size will conflict with the allocated resource, and consider placing high-volume, low-signal data in other storage architectures. This advice has in fact been given by some close-coupled system vendors, along with product capabilities to do so. Data-Fabric Federation [Centralised matrixed data, accessed remotely] In this type of federation, we discussed how the architecture overcomes storage capacity limitations of close-coupled federations and provides domain and cross-domain normalisation. Drawbacks Latency – the architecture is unable to query received data in true real-time, due to the separation between the data lake and the compute resource. While this architecture is intended to reduce repeated full-scale replication, data like asset information must be duplicated between it and a SIEM. Where inputs overlap with those into a close-coupled system, findings may be duplicated. From a TCO perspective, some additional compute costs may be generated inside the data lake. Managing Data-Fabric Federation Considering the drawbacks, enterprises must: Identify data sources for which sub-1-minute delays in querying are non-negotiable and continue to use a capable fast architecture (often close-coupled) for those situations To reduce costs, complement close-coupled systems with supplemental detections in a data-fabric federation that analyses high-volume security data efficiently Manage detections as code in a common repository and ensure detections are deployed only to one platform to avoid alert duplication Understand data lake cost regimes to avoid unanticipated costs Inter-Platform Federation [Remote multi-siloed data, fractionally normalised] In this type of federation, we discussed how disparate enterprise repositories function as federated storage that is queried by a central layer, and where only the query results are normalised. Drawbacks Though the central layer presents a unified place for queries, the data in the repositories remains siloed.  This prevents any correlation before query results reach the central layer and often leads to atomic analytic results where each result never references data from another silo. This can cause a high proportion of false positives because at the time of query, contextual data from other repositories is not available to filter benign activity. The effectiveness of the central layer may be reduced because the central layer may inadvertently obstruct the visibility of issues in the data repositories. For example, security analysts get used to checking query results in the central layer and assume that a reduction in security detections means that there is a reduction in threat activity.  However, there may be issues with analytics or the receipt of data at one of the data repositories, which might be a close-coupled system. The siloed architecture may not be oriented for answering a question that is framed as a storyline, such as: ‘Tell me about particular types of activity involving a device during a specific time frame’.  Without raw data being centralised to tie seemingly disparate activities together, the activity may be missed by the siloed queries.  For example, a successful brute force attack might be missed if there is no visibility of login attempts and outcomes across siloed authentication systems. Latency and inability to manage it across silos can prevent event correlation.  For instance, if three repositories are queried for events that happened in the past hour, but one repository may receive data from sources three hours late (it happens), that incomplete information becomes the basis for decisions and responses. Managing Inter-Platform Federation Considering the drawbacks, enterprises must: Monitor the feed of data into their downstream repositories to be sure that any deviations in volume, latency or quality are addressed and do not reduce effectiveness at the central layer Ideally converge as many complementary data feeds into a single repository so that gaps are reduced when single-silo queries are made – for example, DNS, DHCP, Firewall, EDR and Proxy data are complimentary sources which could comprehensively answer questions about network activity even when these five data sources are alone in one repository Evaluate the suitability of this architecture for use-cases that routinely require wide-ranging correlation, such as insider threat monitoring Reductive Federation [Remote multi-siloed data, accessed in its default state] We discussed how this type of federation shares much in common with Inter-Platform Federation – but to prevent unwanted surprises involving data integrity and availability, there is much more to manage.  This is because data received by a destination can be destructively changed, with original security events losing their fidelity or dropped altogether. Such systems offer safety nets for this destruction, of course, being either of: offering the ability to replay the original events to the close-coupled system, like the previous destruction never happened, or copying original events to federated storage repositories, and then providing a central layer for querying such data while it remains in place at those federated locations.  This typically requires learning a unified query language. Drawbacks Arising from its similarity to Inter-Platform Federation, Reductive Federation shares the previous four drawbacks, in addition to a couple more: Because it is often adopted as a workaround for close-coupled systems, a scenario such as a forensic investigation may arise where a decision is made to send replay data to the close-coupled system.  It may however fail for either of two reasons: First, if the close-coupled system has a retention period (say 3 months) that is shorter than the age of replayed data (say 6 months), the data will be deleted upon receipt at the close-coupled system. Second, given the drive to reduce costs, the enterprise would almost certainly have reduced the local storage and/or ingestion capacity of its closed-coupled system.  This means a sudden dump of forensic data would exceed that reduced capacity and cause the cache contention scenario described earlier. The result would be an under-capacity system that struggles to meet BAU while crawling to answer urgent forensic questions. Data would remain un-normalised without deliberate implementation activity, so datasets will be both siloed and misaligned for correlation. Managing Reductive Federation In addition to the management factors of Inter-Platform Federation, enterprises must: Evaluate what contingency they have prepared if they reduce the capacity of their close-coupled system Consider mitigations for any foreseeable scenarios in which they will require coherent and consistent correlation capabilities for large volumes of data Conclusion For enterprises drowning in data but starved for insights, data federation is emerging as a part of many ostensible solutions. This article provided an analysis of four federation approaches seen across different vendors, highlighted the respective advantages, and noted areas where management is required.  Enterprises should tightly align their use-cases with solutions whose federation approach best meets their needs. The right choice of federated solution empowers enterprises to gain insights, defend themselves better, save money and minimise remedial management activity.

Federating cyber security data Large enterprises are in a difficult position when it comes to data.  They are contending with exploding data volumes. They operate in business with multiple incongruent cloud and SIEM applications.  The quest has been on for liberation through effective security tools that scale affordably. From as long ago as 2016, IDC was spotlighting software architectures that rely on highly decoupled distributed systems[1]. In the IDC report, Five Benefits of Decoupling Compute and Storage for Big Data Deployments it was further noted that: Decoupling compute and storage is proving to be useful in big data deployments. It provides increased resource utilization, increased flexibility, and lower costs. In the intervening years through to the time of writing (2024), this decoupling has been federated to the comparatively cheap, elastic compute and storage of the cloud.  Cloud-based federation has been implemented both by long-established players like Splunk, and new services such as Hunters, Panther, Cribl, Anvilogic and DataBee, which define themselves in part by this decoupled, federated architecture. A skim across each of the preceding products indicates that the idea of decoupling often has interpretations that vary by vendor.  The resulting different implementations each have challenges associated with the respective product.  In this article, the architectures are introduced, while a subsequent article will analyse the architectures and their associated challenges. [1] Directions 2016 - Digital Transformation at Scale: Innovation in a Changed World. Hilwa (2016). IDC Introducing the federation approaches The central point of comparison for each federation approach is the data residency – whether the data received from disparate sources is centralised or siloed.  Two other attributes distinguish the approaches, summarised in the table below: Federation Approach Data Residency Location of Queried Data Data Schema Outcome Close-Coupled Centralised Local Normalised Data-Fabric Centralised Remote Matrixed Inter-Platform Remote Remote Fractionally Normalised Reductive Remote Remote Default State The following drawing shows the federation approaches and their attributes, organised around their data residency: The diversity of the approaches is remarkable because vendors have incorporated the federation attributes quite differently, leading to clear dissimilarities in federation approaches and their associated solutions. It is worth noting too, that each vendor applying a particular form of federation believes that their approach works best for enterprises.  The federation approaches are now explored in turn. Close-Coupled federation [centralised normalised data, localised for access] In this type of federation, the software often relies on traditional close-coupled hardware (i.e. memory, compute and storage that are (or perform like), they are locally attached.  This aligns with critical design priorities, being the speed of querying, correlating and performing analytics on data from disparate sources.  The fact that the data being queried is centralised, normalised and local to the compute resource facilitates true real-time receipt, querying and analytics of data. Despite those benefits, decoupling becomes necessary because the storage for large volumes of security data, say from 1TB/day, becomes expensive when it is scaled proportionally with compute resources.  So rather than having more units that combine both compute and storage, a separate elastic cloud storage stores long term data and feeds it to local storage as needed. Data-Fabric federation [centralised matrixed data, accessed remotely] In this type of federation, data from various sources is centralised and normalised not only by IT domain (such as network and access management), but also between IT and business domains to create a matrix of highly correlated data for comprehensive query results.  The federation of this data occurs by storing it in a data lake, where storage will expand independently of compute.  Unlike in close-coupled federation, the software is often microservices-based, and the data lake is remote, eliminating the simultaneous increase of compute costs with storage. Here, the appeal is from correlation and re-usability of the centralised data by different consumers across the enterprise, including for continuous controls monitoring. This architecture overcomes storage capacity limitations of close-coupled federations. Inter-Platform federation [remote multi-siloed data, fractionally normalised] In this type of federation, a central layer is designed as the single place to query multiple data repositories such as data lakes, cloud storage and SIEMs. Quite importantly, the data remains in place at these locations, while the results of queries (typically alerts) are sent to the central layer.  When the results are received, they are normalised into a consistent format for further actions in the central layer. This approach has easy adoption appeal because it effectively leaves federation to customers by treating their customer’s existing data repositories as federated storage. Reductive federation [remote multi-siloed data, accessed in its default state] This type of federation shares much in common with Inter-Platform Federation.  Again, a central layer is designed as the single place to query multiple data repositories. In contrast, the design does not include normalising all results of a query.  As the name implies, this approach is often implemented within a data volume reduction strategy, where data from various sources is fed through a size-reducing processor while in transit to any number of repositories.  Oftentimes, that data is stripped, aggregated or sampled.  The data reduction is often to limit the volume received by a close-coupled system.  This approach also has appeal, for two reasons: the opportunity to reduce the cost of owning a close-coupled system by drastically reducing the amount of data stored in that more expensive architecture, sometimes saving a small fortune getting away with said cost reduction Conclusion Data federation is a part of many ostensible solutions for addressing the data volume explosion, but the various incarnations of the federation are imperfect. This article introduced four federation approaches seen across different vendors. A subsequent article will explore drawbacks to each architecture, and the corresponding consequences that need management.   Continue Reading: Minding the stepping stones across security data silos

The drawing below is a simplified version of the typical data morass that large organisations face, composed of sources spanning on-premises data, multi-cloud data and (I/P/S/C)-as-a-service. This is such a fractured state of affairs, that it prompts a few questions: Have information security departments given up on centralising cybersecurity data? Why did organisations ever try? How could any organisation succeed at centralising security data today? Why centralise security data? Let’s start by answering the easiest of three questions first – the why: many security standards specify controls where event logs from multiple security tools are centralised for analysis and correlation, for example in a SIEM.[1] While the SIEM acronym may be out of favour with some readers, the concept of “Security Information and Event Management” is indisputably essential for large, digitised organisations.  One should emphasise the management part of the concept, which has fallen off more recent acronyms.  For those who prefer the cybersecurity mesh concept, it too, has a centralised security data layer.[2] In any case, the successful execution of the SIEM concept is as important as ever, regardless of which acronyms and labels are thrown around.  We hear that the latest security data product named with three/four capital letters is “next-gen”, “intelligent”, “cloud native”, “AI-first”, et cetera, while the others are “legacy” technologies.  Regardless of the year that a SIEM-concept product was first marketed, the insidious legacy aspect that plagues many of these is the paradigm of siloing that applies to security data: intra-company silos – oftentimes, the SOC and SIEM are the only designated consumers of security data, while willingness and capability for sharing the data are lacking[3] vendor silos – data mobility[4]and interoperability are often costly or tedious to attain Now that we have discussed a couple of motivations and challenges related to centralising cybersecurity data, we can begin to answer all the earlier questions in the context of data residency and organisational capability. Are information security departments succeeding in centralising cybersecurity data? As long ago as 2013, a SIEM was considered table stakes (the minimum outlay) for organisations wanting to build an advanced cybersecurity capability. At the time, analysis and correlation were applied to detect threats across event streams from internal infrastructure such as network traffic, email, and server logs.[5] In that context, collecting events, perhaps for an on-premises SIEM, was a relatively finite and achievable task. However, through the present time, as organisations undergo digital transformation and migrate operations to the cloud, it has re-emerged as a challenge to integrate and centralise event streams from infrastructure, cloud, and software-as-a-service platforms.[6] The problem is not merely that data is siloed – it’s that organisational cybersecurity responsibility is centralised, as is the risk to the organisation’s balance sheet and stakeholders[7], whilst conversely, fragmented data prevents organisational knowledge of consolidated security risk exposure. This fragmented data can look like a sea of seemingly disconnected activities. However, to proactively reduce risk, organisations must try to identify relationships between activities before they have an impact.[8] Nonetheless, security offices are contending with high rates of data output, large varieties of data sources, and incongruent data structures. These all make it difficult to assemble an integrated understanding from the limited vantage points of each source’s dataset. It’s unsurprising then, that despite the potential blind spots, some vendors and security offices are leaving data in their respective silos. Examples of such siloed architectures are Inter-Platform Federation and Reductive Federation, which overlay remote, fragmented cyber data. Those architectures are however accompanied by additional daily operational difficulties such as latency and uncorrelatable data.[9] So, for security offices that are seeking centralised cybersecurity data, what is a solution? Historically, the answer would have been a SIEM, just by itself. Unfortunately, SIEMs from many vendors have architectural challenges that persist even when the SIEM federates to the cloud in a Close-Coupled architecture.  Despite the challenges, for many organisations, a quality SIEM still has certain speed and usability advantages that many purported replacements do not. For example, it is reasonable for security analysts to experience “real-time” receipt and querying of data when that is promised by a product. Another example is where the SIEM alternative requires navigating, converting, and joining SQL table elements, which SOC analysts can experience as time-sapping obstacles to answering urgent, ad hoc questions emerging from investigations. Many organisations seek workarounds for scaling and retention constraints that they may experience with their primary SIEM. One workaround is to reduce the SIEM workload and maintain its speed by prioritising certain data sources for analytics and limiting their retention. As illustrated earlier, this is supplemented by storing data the organisation would like to retain in one or more of: a secondary SIEM (where query speed and latency are lower priorities) an object store a data lake (or warehouse, or lake-house, etc.) The data offloads evidently do not centralise security data and they may perpetuate SOC-centric ownership of security data, when the organisation would be more resilient if, for instance, the compliance function could leverage aspects of the same data. Consider a governance risk and compliance (GRC) function enabled with the data to continuously monitor security controls like secure configuration and vulnerability management.  GRC can in turn work with IT and relevant leaders to reduce the likelihood of incidents for which a SOC is needed.[10] How could any organisation succeed at centralising security data today? Consider an organisation which centralises some cyber data that is a year old or newer in a SIEM - how could it centralise the remaining data cost-effectively without introducing another silo? The simple version of the answer is to augment the SIEM, but with what? A subordinate data layer that helps cost-effectively address the retention and scalability issues, while maintaining correlation and analytics capability. Fortunately, technology like this already exists, but in the following table, let’s summarise the solution requirements first. Requirement Effect Addressing retention and scalability issues Scaling elastically in the cloud to receive high throughput data and store long-retained data, even for many years.  Implemented through a decoupled architecture that facilitates storage growth without driving compute costs simultaneously.  Achieved at the cost of the latency associated with the decoupled storage layer. Maintaining correlation and analytics capability Rather than serving as a dumping ground for raw data in its default state, can ingest and transform received data to a suitable schema.  Analytical, machine-learning and other processes can be applied to the data for insights. Subordinate data layer Avoids creating another screen to pivot to for context. Instead, security findings from correlation and analytics are pushed to the SIEM with supporting raw data.  This enables the SIEM to present findings of the subordinate data layer as though the SIEM processed the data, and avoids creation of another silo. One could simply call this a SIEM augmentation layer.  An incarnation of this is a Data Fabric Federation architecture9. Here is what the SIEM augmentation and data relationships could look like. This architecture achieves a few goals: For data sources excluded from SIEM ingestion due to cost trade-offs, recovering their value by correlating the full scope of security-relevant activity. Centralising the data without producing another silo, which not only avoids disruption to existing security operations workflows, but enhances the intelligence available to them. The data fabric facilitates tailoring of security data products for AI, machine learning and security governance risk and compliance.  This is the “multi-consumer” vision of an enterprise data mesh, the organisational model used to manage data products.[11] Conclusion Organisations have made the business decision to adopt distributed architectures across networks and the cloud.  Those decisions renewed the challenge of centralising cybersecurity data, where in response, some organisations are looking to a data mesh approach to produce data products on top of data fabric architectures. This article discussed the practicality of this method to expand the data sources and retention available for enterprise cybersecurity operations, while simultaneously empowering GRC and other teams to act proactively in reducing cyber risk.   [1] See for example, NIST 800-53 rev 5 Controls AU-2, AU-6, PL-9; CIS Control 8; ISO 27002 Control 12.4 [2] Ramos-Cruz, B., Andreu-Perez, J., & Martínez, L. (2024). The cybersecurity mesh: A comprehensive survey of involved artificial intelligence methods, cryptographic protocols and challenges for future research. Neurocomputing, 581, 127427. Gartner. (2024). Cybersecurity Mesh. [3] Goasduff, L. (2021, May). Data Sharing Is a Business Necessity to Accelerate Digital Business. [4] Sommerfeld, B., Elbekry, A., & Ene, L. (2015, October). Inside: CIO Edition by Deloitte. [5] Deloitte. (2013). Tech Trends 2013 - Elements of postdigital. [6] McKinsey & Company. (2020). Cybersecurity in a Digital Era. [7] Maze, T., & Musselwhite, D. (2020, May 19). Primary vs. Secondary Loss in FAIRTM Analysis: What’s the Difference and Why It Matters. [8] Howard, R. & Olson, R. (2020). Implementing Intrusion Kill Chain Strategies. The Cyber Defense Review. 5(3), pp.59–76 [9] Asiedu, E. (2024). Four ways of federating exploding cyber data. [10] Abdillahi, Y. (2024). The Rise of Cybersecurity GRC. Corporate Compliance Insights. [11] Beyer, M., & Thanaraj, R. (2024). Data Mesh v/s Data Fabric? Identify the Benefits and Risks Before Making Your Decisive 2024 Investment. Gartner Data & Analytics Summit.