With growing cyber threats, the need to develop an experimental testbed for understanding rapidly evolving complex US Army’s tactical cyber domain is highly crucial to close the gap between threat and defense. Traditionally, cyber security related researches are conducted using either discrete event simulator or network emulator narrowly modeled for the given objective of research study. Such research approach limits reuse of experimental components. In this whitepaper Ad hoc Research Associate’s solution will propose an experimental workbench which can provide a framework that can incorporate several different types of experiments and conducts wide array of cyber security related research.

Experiment repeatability, reusability and accuracy are the important factors that should go behind designing an experimentation workbench that models Army C4ISR heterogeneous networks and systems for cyber security related research. Considering above factors, discrete event simulation based experimentation workbench is not a viable option for cyber research alliance members. In discrete event simulation environment, there is a tradeoff between model fidelity and number of events required to be executed. Experiments utilizing high fidelity model in discrete event simulators will run slower than real time. This would also limit researchers to plug and play live systems into their testbed. Network Emulation based experimentation testbed is the best viable option for cyber security researchers. Network Emulation Testbed also provides ability to quickly prototype, test and integrate a solution into deployed systems. Emerging emulation testbed technologies definitely meets requirements to cover following important factors.

Accuracy – Emulation Testbed accommodates real algorithms on general purpose hardware. It reduces model abstractions and offers better accuracy and realism.
Reusability – Models developed for emulation testbed framework could be effortlessly reused for various cyber research topics. It may only require few minor modifications.
Repeatability – Experiments in emulation testbed are executed in real-time. Hence, in given practical time, researchers can execute several iterations of runs to conduct empirical based studies or do quick what-if type of analysis.

The term experiment is often used to describe the experimental apparatus or environment that researchers have assembled from network nodes, tactical links, physical & virtual hosts, traffic generators, data collectors and analytical tools. In addition to the apparatus, an experiment also includes experiment objective, scenario building, procedures to execute test runs and analyzing results from an experiment. Along with experimental apparatus it is necessary to first layout experimentation workbench verification & validation (V&V) process. Most model based experimentation has some level of abstraction. Abstracted models induces uncertainty in results. In complex system of systems modeling environment, measurement of uncertainty is not feasible.

Following key capabilities will be needed for experimentation V&V: (1) Describe and measure both required and achieved experimentation fidelity, (2) determine proper ways to merge information from experimentation with different levels of resolution and accuracy, (3) compare experimentation results against real world data, and (4) capture and store all pertinent experimentation V&V information in repositories.