Journal of Green Engineering

Vol: 6    Issue: 3

Published In:   July 2016

An End-to-End Testing Ecosystem for 5G the TRIANGLE Testing House Test Bed

Article No: 3    Page: 285-316    doi: https://doi.org/10.13052/jge1904-4720.633    

Read other article:
1 2 3 4

An End-to-End Testing Ecosystem for 5G
the TRIANGLE Testing House Test Bed

Andrea F. Cattoni1, German Corrales Madueño1, Michael Dieudonne2, Pedro Merino3, Almudena Diaz Zayas3, Alberto Salmeron3, Frederik Carlier4, Bart Saint Germain4, Donal Morris5, Ricardo Figueiredo5, Jeanne Caffrey5, Janie Baños6, Carlos Cardenas6, Niall Roche7 and Alastair Moore7

  • 1Keysight Technologies, Denmark
  • 2Keysight Technologies, Belgium
  • 3Universidad de Málaga, Andalucía Tech, Spain
  • 4Quamotion, Belgium
  • 5RedZinc, Ireland
  • 6AT4 wireless, Spain
  • 7University College London, UK

E-mail: {andrea.cattoni; german.madueno}@keysight.com; {pedro; almudiaz; salmeron}@lcc.uma.es; {dmorris; ricardo}@redzinc.net

Received 8 September 2016; Accepted 9 December 2016;
Publication 24 December 2016

Abstract

Mobile applications will be a dominant element in the 5G domain. Ensuring the correct and efficient behaviour of the applications and devices on which they run becomes a critical factor to ensure the mobile communications market meets the expectations of final users. The EU project TRIANGLE is building a framework to help app developers and device manufacturers in the evolving 5G sector to test and benchmark new mobile applications, devices, and services utilizing existing and extended FIRE testbeds. This innovative framework will facilitate the evaluation of the Quality of Experience and enable certification for new mobile applications and devices.

Keywords

  • 5G systems
  • Testing
  • Quality of Service
  • Quality of Experience
  • Mobile Apps
  • Software Defined Networks
  • Business Network Applications

1 Introduction

Mobile communications is one of the biggest growth areas in the Information and Communication Technologies (ICT) market, and it is expected to reach 4.7 billion mobile devices connected to the Internet by 2020 [1], with an overall economic revenue of $373 billion in 2017 in the smartphone market alone [2]. Mobile apps will be a major software stakeholder in this huge ecosystem. At the same time 5G will become the de facto connecting technology standard for the majority mobile devices, including cellular-based Internet of Things (IoT) ones. Ensuring a correct and efficient behaviour of the applications and devices becomes a critical factor in meeting the expectations of end users. Extended battery life for any type of device, whether a smartphone or an IoT sensor, is one of those expectations. The TRIANGLE framework represents a great opportunity for European enterprises providing innovative testing tools based on testbeds with 5G capabilities. The purpose is to develop realistic testing methodologies that will complement other 5G research and innovation activities, helping to maintain the traditional European leadership in the mobile communications arena.

While radio related certification of mobile devices already has a strong standards based ecosystem thanks to the collaboration among the European Telecommunication Standards Institute (ETSI), 3rd Generation Partnership Program (3GPP), Global Certification Forum (GCF), vendors and test houses, there is still a lack of consensus on the benchmarking or testing methods at the apps level. The Key Performance Indicators (KPIs) and techniques to perform such kind of testing for apps could be rather different than the certification methods associated with 3GPP standards for mobile devices (user equipment or UEs), partially due to the proprietary nature of many widely used apps, lack of a testing methodology and lack of standardized interfaces. The actors involved in producing and exploiting mobile apps are different in number and categories compared with those involved in mobile device manufacturing and integration. Thousands of Small and Medium Enterprises (SMEs) build apps compared to the few big vendors that build smartphones. Therefore, existing solutions for testing apps represent a very fragmented market with many ad-hoc tools associated to vendors.

From a technical point of view, the key issue is to base the testing of apps and mobile devices in real but controllable conditions (testbeds) that simulate/emulate realistic 5G scenarios as defined by Next Generation Mobile Networks (NGMN) [3]. Testbeds that combine proper realistic hardware and software, are currently provided in Europe by the FIRE project, but do not cover the new 5G scenarios. It is nowadays clear that 5G will be a very complex system. Such complexity could negatively affect the expected duration of the device battery. While studies of power consumption correlated to device models and network conditions are present in the literature, they are unfortunately very rare and use ad-hoc methods that cannot be directly considered for the implementation of a certification process.

From a market point of view, traditional test houses, that are key actors in the success of mobile technologies, could be playing again a leading role in the definition and exploitation of a new generation of tools for the emergent market of mobile apps plus devices in the 5G ecosystem. Other players like research institutes, innovative SMEs, test equipment industry, device vendors will also be necessary to develop such solutions.

2 The TRIANGLE Project

The “5G Applications and Devices Benchmarking” (TRIANGLE) project [4] is a European Union (EU) funded project within the framework of the “Horizon 2020” (H2020) initiative under FIRE/FIRE+ objective devoted to Future Internet Research and Experimentation. The project started in January 2016 and has an expected duration of three years. Among its partners there are laboratories, SMEs, test equipment suppliers and academia. The primary objective of the TRIANGLE project is to promote the testing and benchmarking of mobile applications and devices in Europe as the industry moves towards 5G and to provide a pathway towards certification in order to support qualified apps and mobile developments using FIRE testbeds as testing framework.

TRIANGLE is developing a framework that facilitates the evaluation of the Quality of Experience (QoE) of new mobile applications, services and devices designed to operate in future 5G mobile broadband networks. The framework will exploit existing FIRE facilities adding new facilities when necessary. The project will identify reference deployment scenarios, will define new KPIs and QoE metrics, will develop new testing methodologies and tools, and will design a complete evaluation scheme. The project will develop a framework to ensure users QoE in new challenging situations, especially those due to heterogeneous networks and considering the role software will have in the new 5G ecosystem. The framework, will also provide the means to allow certification and obtain a quality mark for the applications, services and devices which are compliant with the requirements and test specifications developed in the project. This will allow vendor differentiation, especially among startups and SMEs, in the current globalized and competitive market. The framework, methods and tools developed during the project will focus on providing the mechanisms to incorporate new wireless technologies and topologies envisaged in 5G and contribute to the new ecosystem.

2.1 Why TRIANGLE?

During recent years the research activity growth has resulted in several wireless testbeds. But although the mobile traffic estimates grow year after year it has not reflected in the number of FIRE wireless testbed users. The main reason for this situation is the design of the FIRE testbeds themselves, which are network centric. FIRE testbeds are normally focused on network configuration and have very complex and sophisticated configuration mechanisms, while the experimenters are not familiarized with the complex setup of the network resources and most of the time end up using only the default configuration. Based on previous experience running the federation of PerformLTE testbed [5], it has been identified that testbed users spend most of their time gaining access to all the low level parameters that impact the user traffic transport performance as they lack enough know-how on setting these parameters to generate a consistent experimentation scenario. This can be frustrating for them because it prevents them from obtaining quick insights on the performance of their products. This is particularly critical for power consumption tests, because an erroneous network configuration can negatively impact the performance of the device [6].

The main idea underpinning the methodology to be used in the design and development of the TRIANGLE test framework is to ensure that the end user is not overwhelmed by the complexity of the overall testbed by not being exposed to all the details, which requires multi-disciplinary knowledge (e.g. protocols, radio propagation, software) to fully understand. To overcome the need for multi-disciplinary knowledge, TRIANGLE will ensure an abstraction of the underlying networking technologies by offering:

  • a high level configuration layer (personality) which calls on detailed scenario definition; and
  • a flexible framework architecture to incorporate new 5G networking topologies as they become available

The project will design a set of high level configurable scenarios that can be reproduced and whose final testing output is a behaviour score or TRIANGLE-MARK, which defines how well the product (application or device) behaves when used in a realistic network, including energy consumption and model-based runtime checking of the apps and devices. It will be possible to modify and setup very complex scenarios by means of a scenario editor, with an Application Protocol Interface (API) and a Graphical User Interface (GUI). This scenario editor will be able to load pre-defined scenarios and generate scenarios based on 5G use cases covering KPIs of interest for each one of them.

The project will provide a framework with different layers of abstraction using, when possible, commercial configuration interfaces as well as experimentation standards, including those provided by Fed4FIRE [7]. Where needed, advanced users will be exposed to deeper configuration details and flexibility.

2.2 Benchmarking-as-a-Service

Being ready for the 5G rollout in 2020 means in part understanding the technical behaviour of a new 5G product or application as well as the network capabilities. TRIANGLE can provide Benchmark-as-a-Service support to help new entrepreneurs and established suppliers ‘tool-up’ for 5G. Using the TRIANGLE framework a device or application developer can tune the parameters of its system to optimize performance under different conditions. Such conditions could include network load or other factors affecting radio power that could be emulated on the TRIANGLE testbed.

TRIANGLE can enable comparative studies of behaviour in different conditions. Profiles can be established and system behaviour can be measured, analysed and compared to reference applications on the TRIANGLE testbed. This gives a benchmark and an understanding of product position to the tester. Such comparative studies could help device or application development for the 5G market. New product entrants are often in a race to be early into the market to get an established position and early customer wins. TRIANGLE can help with this process and reduce the risk in bringing new products to market.

2.3 The TRIANGLE-MARK

TRIANGLE offers a TRIANGLE-MARK as a pre-standards pre-certification tool providing a relevant score based on a set of indicators. The TRIANGLE-MARK ensures that a mobile application or device operates in the future 5G mobile broadband networks with a guaranteed QoE by assessing and comparing an application or device capability in a 5G network emulation environment against a set of criteria or KPIs.

As an example, an application with the TRIANGLE-MARK, will have been tested in different hardware platforms (as defined by the TRIANGLE Testbed) to verify very different performance indicators such as stability, security, reliability or the performance, both in terms of time behaviour and in the use of resources. The application will get a final score that allows the users to compare its performance with that of similar applications in the market.

When a product gets the TRIANGLE-MARK, it will interwork and perform satisfactorily in the growing ecosystem of 5G networks, devices, services and applications.

2.4 The TRIANGLE Approach

TRIANGLE intends to address the needs of four test bed user profiles within a single test framework: an advanced Researcher, a Device Manufacturer, an App Developer, and a Mobile Network Operator (MNO). The project’s testing approach is portrayed in Figure 1.

images

Figure 1 The TRIANGLE approach for providing a unique certification grade.

2.4.1 Researcher profile

The most extended personality on FIRE wireless testbeds today is the Researcher personality, especially researchers who are working on protocols and network technologies. This personality is interesting but only represents a small group of possible testbed users. These users will be looking for the ability to fine tune low level parameters, including test adaptation within the network layers, create new use case scenarios; in general to verify the functionality of lower communication layers.

2.4.2 Device manufacturers profile

Device manufacturers will have a more general system performance level interest in the testbed to understand the capability of the overall device compared to other reference devices. Today, the number of device manufacturers is rather limited but, with the upcoming Internet of Things (IoT) and device variety, the numbers will grow rapidly. The skill and knowledge of these device makers is also changing from being a fully integrated in-house developer (with all components in house) towards a more system level integrator type that relies on modules and chipsets from external suppliers. This type of outsourcing, combined with the strict energy requirements of IoT, make TRIANGLE a powerful ally for understanding the real capabilities of the devices in terms of energy saving.

2.4.3 App developer profile

These pure software personalities are the largest target population for the TRIANGLE project. These are usually start-up companies who wish to develop an app based on a great idea and need platforms to validate its functionality and its performance under challenging radio conditions. The app developer, not being a telecommunication engineer, typically does not care about the network details and radio propagation challenges. The developer typically assumes the wireless connection is a reliable data pipe with characteristics similar to a wired one. TRIANGLE will provide a framework to test the impact of realistic data network capability on the app as well as the app impact on the network resource usage, making possible to provide insight on how the App uses the device resources and battery. Developers will be able to have clear insights on the energy performance of their own code.

2.4.4 Mobile network operator profile

While primary targets for testing and certification are the device manufacturers and App developers, the project also wants to provide MNOs with a fully controlled, realistic sandbox where to test advanced network components and configurations. This type of setup is not new to MNOs. As a matter of fact some major operators have similar testbeds in their labs. But although there is wide interest among operators to understand the impact of network configurations on the performance of mobile devices and the QoE of mobile apps, only few major actors can afford it.

2.5 Open to Experimenters

The test framework developed within the project will be accessible to the developer community, initially to the European community and internationally later on. During the project, access to the test framework will be organized via open calls, where TRIANGLE will seed 15 Open Call Experiments focused on applications and devices mostly from innovative SMEs and entrepreneurs. These Open Call Experiments will be used to fuel the market for benchmarking and certification towards 5G. The TRIANGLE-MARK will be given with the benchmark measurements or score obtained.

The Open Calls will be used also for improving the testbed capabilities with advanced technologies that are relevant towards the 5G evolution. The project will seed at least two to four technology providers that will include their products in the testbed, making them available to other experimenters.

3 TRIANGLE Test Bed

The main core of the TRIANGLE project is the testbed that will allow all the different users to run tests in a controlled environment that encompasses all the elements of the telco chain. An overview of the main elements that compose the test bed is shown in Figure 2.

images

Figure 2 The TRIANGLE test bed architecture.

3.1 Mobile Device Monitoring

In order to measure the KPIs perceived by the end device and the end user, several tools are needed on the smartphones to be tested. Such Monitoring tools are being internally developed by TRIANGLE partners and will be available to experimenters.

3.1.1 Performance tool

5G test scenarios will require high resolution for reporting target QoS KPIs. In order to extend the measurement capabilities of other tools like iperf, TRIANGLE will provide up to OSI layer 7 SDU packet resolution in the computation of data performance KPI thanks to the integration of the AT4 wireless Performance Tool. This tool is composed of two components, Controller and Agents (data endpoints), and uses proprietary mechanisms to synchronize the Agents and provide accurate one-way measurements. This tool includes a built-in traffic generator with the capability of generating constant rates, ramps, loops and statistical traffic patterns which is something of utmost importance for setting up the desired environment in terms of varying traffic loads (e.g., for measuring LTE-U impact on Wi-Fi networks). Additionally, this tool has the ability to automate some mobile Apps on Android devices and measure relevant QoE KPI such as YouTube buffering occurrences.

3.1.2 TestelDroid mobile monitoring app

TestelDroid [8] is a software tool that enables passive monitoring of radio parameters and data traffic in Android-based devices. Logging is implemented as an Android service that can be running in the background logging all the information while the application under test is being executed. This functionality enables monitoring of the traffic information generated by any application, which extends the testing to a very wide range of use cases. The parameters to be logged (network, neighbour cells, GPS, traffic) can be flexibly configured using the SCPI interface.

3.1.3 Power consumption analyser

In order to provide a complete overview of the performance of the App or the Device under test, it is nowadays unthinkable not to include an analysis of the energy consumed by the mobile platform. For this reason TRIANGLE plans to provide accurate measurement consumptions thanks to the Keysight N6705B Power Analyser. The instrument allows the provision of DC voltage to the device and accurately measures the drawn current to verify how much power is required by each app and network operation over time.

3.2 Network Emulation

The test bed also comprises a network emulation that spans from the Radio Access Network (RAN) to the core of the LTE system. All the available tools are orchestrated as well, to provide the highest degree of experimentation flexibility to the test bed user.

3.2.1 UXM RAN emulator

An essential component in the testbed is an instrument capable of emulating multiple cellular networks in a controlled manner. To that end, TRIANGLE envisions the usage of the UXM Wireless Test Platform device by Keysight Technologies, which supports multiple radio access technologies (multi-RAT), including GSM/GPRS, UMTS and LTE-Advanced networks (i.e., 2G, 3G and 4G). The UXM features include intra-RAT and inter-RAT handovers, protocol debugging, IP end-to-end delay and throughput measurements, and performing RF conformance tests. Finally, it should be noted that the UXM also features an advanced fading engine with the main channels models defined by 3GPP [9].

3.2.2 Over-the-air LTE small cells

UMA lab currently offers an indoor deployment of three LTE 2.6 GHz FDD cells, one pico-cell by Alcatel-Lucent and two small cells from Athena wireless. In the context of TRIANGLE this deployment will be updated with new small cells supporting more flexible configurations, including Carrier Aggregation. The use of attenuators and switches will allow controlled experimentation in this commercial based setup, including intra-LTE handovers and LTE-Wi-Fi offloading. Scenarios with several devices can be also reproduced if required to test some apps.

3.2.3 EPC emulator

The Enhanced Packet Core (EPC) is the core of LTE networks, and it will also play a central role in pre-5G deployments that include for instance LTE to Wi-Fi offloading. The EPC is a component with functions like controlling which devices are allowed to attach to the network, supporting mobility (handover) or routing traffic to/from the Internet. The UMX emulator already integrates a subset of this functionality. UMA lab brings Polaris Networks EPC to the project. Thus, the Project will support some scenarios with small cells, that provide real but non fully controllable radio propagation conditions, and others based on the UXM emulator, which provides fully controllable radio propagation conditions, both connected to Polaris EPC through standard 3GPP interfaces.

3.2.4 Virtual path slicing engine

Software Defined Networks (SDN), Network Function Virtualization (NFV) [10] and OpenFlow [11] are techniques that provide high flexibility and can cut costs in packet data networks. Virtual Path Slicing (VPS) is a method that has evolved from studies like [12] on how to implement end to end QoS in packet networks. RedZinc has implemented a VPS engine that can steer traffic on the edge of the network based on the request of the application and coordinate this traffic steering with other autonomous systems. In traditional 3GPP EPC the VPS engine can provide services to the application layer and invoke them using the Rx interface in the Policy Charging Rules Function (PCRF) of the EPC where differentiated QoS Class Identifier can be used for different service levels.

3.3 App Testing

In order to test the apps running on the mobile devices thoroughly, the user interaction needs to be taken into account. TRIANGLE takes advantage of several approaches in this respect, considering both real and simulated user interactions. These approaches are complementary and result in a greater coverage of tests and app-related KPIs. These tools are of particular importance to provide a realistic usage pattern of the apps whenever power consumption is measured.

3.3.1 App Lab live environment

TRIANGLE will use App Lab, a high level platform developed by UCL for distributing mobile apps to a large-scale testbed for pre-commercial testing and validation services. App Lab provides a private app store offering in-the-wild rapid user field-testing to UCL staff and students. App Lab is comprised of a server portal for application upload, management and distribution and native mobile client apps for both Android and iOS devices. App Lab uses analytics tools to track how users interact with apps in the real world. By examining the results of app usage, test cases which mimic actual users’ behaviour can be derived and executed in a variety of network conditions in the TRIANGLE testbed as shown in Figure 3.

images

Figure 3 Live testing interaction via App Lab.

3.3.2 Quamotion test automation tool

TRIANGLE will also use Quamotion test automation tool which provides the ability to write and execute automated tests for Android and iOS applications. Execution can be performed without any manual interaction. Not only gestures and keyboard interactions but also the full lifecycle of the application can be automated, i.e. installing, launching, terminating. This set of actions are exposed to the overall test orchestrator such that the end-user actions can be synchronized with various network events, e.g. an unstable network connection in a train. Quamotion will also add recording capability to increase the efficiency in the process of writing test cases. Each step performed on the device is recorded and translated automatically into an action as part of a test script.

3.3.3 Model based testing

Another alternative considered in the project is model-based testing. Instead of writing test scripts manually with sequences of user actions, e.g. pressing a button, entering some text, app developers can provide a high-level description of the intended workflow. Test cases could be automatically generated from this description, achieving a high coverage of realistic usage scenarios. These models may also be based on crowdsourced data, such as observed user interactions from live tests, including tests run using App Lab.

4 Test Bed Orchestration and Management

The orchestration and management architecture of the testbed will be based on OMF (Orbit Management Framework) [13]. OMF allows the definition of repeatable and automatable experiments using an OEDL definition language. OEDL is a domain-specific language defined for the description of an experiment’s execution. OEDL provides a set of experiment-oriented commands and statements which can be used to define the tests, the measurements and the graphical results. These OEDL scripts are interpreted by the Experiment Controller (EC), which orchestrates the resources present in the testbed during the execution of experiments. Each resource in the testbed is managed by a Resource Controller (RC), which will translate the actions defined in the experiment into the specific control commands understood by the resources. Measurement data during experiment execution will be collected via OML in a central server. OML provides a programming library for easy application instrumentation and a collection point, a server which stores measurements in an experiment database.

images

Figure 4 The TRIANGLE test bed orchestration system.

TRIANGLE testbed orchestration based on OMF is shown in Figure 4, starting with the definition of the app test cases by the app developer in the top left and the network scenarios on the top right. Once the test cases and the network scenarios, have been completely defined, they can be executed in the testbed. The RC associated with the device under test interacts directly with the Android and iOS devices over USB in order to install the apps and run the app test cases. In order to coordinate the execution of the different tools involved in the testbed an RC for each one of them will be deployed. These RCs allow the tools to be controlled as part of the test, and receive commands from the experiment controller to fetch and execute a particular test case. During the tests, the monitoring and performance tools will collect all the measurement results from all layers and measurement points present in the testbed. The results will then be passed to the Measurement inspection and KPI calculation module, which outputs the final test report.

5 Experiments Setup

The components provided by some partners have already been integrated into a first version of the TRIANGLE testbed. As an exercise the testbed has been used to evaluate the performance and the power consumption of a YouTube video playing on a mobile device, under different radio channel conditions in a single user scenario. An agent that drives the YouTube API was installed on a Samsung Galaxy S4 connected to the UXM LTE network emulator via a coaxial cable and from there to the YouTube server via internet. The performance tool was controlled remotely to run the agent and record KPIs using the algorithms proposed in [14]. The video used has a length of 5 minutes and resolutions ranging from 1080p to tiny. Power consumption is measured using the DC power analyzer which is directly connected to the power terminals of the mobile. The beginning of the YouTube session is synchronized with the power consumption capture. Once the power consumption measurement is initiated it lasts 120 seconds, independently of the duration of the playback of the video. To determine the total consumption the power measurements have to be extrapolated considering the total playback time.

The scheduler used during the experiments only transmits LTE PDUs to the mobile device (in the downlink) when there is IP data to carry. The UE transmission power is controlled automatically to be kept at 0 dBm in all the test scenarios for better analysis of the impact of the DL test configurations.

In the frequency domain, the full channel bandwidth has been assigned for downlink transmission. This translates into 100 Physical Resource Blocks (PRBs) in a 20 MHz cell.

Different levels of Additive White Gaussian Noise (AWGN) have been also added when required to model interference from neighbor cells transmitting at the same frequency. The Signal to Noise Ratio (SNR) is determined by the relation between the power density of the transmitting cell and the interfering noise.

The downlink transmissions follow an open-loop transmit diversity (TM3) MIMO scheme, and the modulation and coding schemes are dynamically adapted to the channel quality and rank reported periodically by the UE.

Channel models emulate the effects of the environment on the radio signal: fading, delay spread and Doppler Effect.

Two different scenarios have been considered: A first scenario which emulates the behavior of a channel while the mobile user is walking in a city (EPA5, 20 dB SNR) and a second scenario which represents a mobile user travelling by bus in a city (EVA5, 10 dB SNR).

The configuration of the experiment includes 4G LTE cells with two transmitting antennas, resulting in a 2 × 2 MIMO matrix as all LTE devices have minimum two receiving antennas. The user device reports the observed radio channel quality (CQI) and the rank (RI), the later representing the number of independent data flows that can be transmitted on the channel.

6 Results

The YouTube traffic has a bursty nature. Figure 5 depicts the downlink (DL) traffic captured by TestelDroid during one of the test executions.

images

Figure 5 DL IP throughput during a YouTube session.

At the beginning of the session, a larger throughput is observed as the application seeks to gather enough data before starting the playback.

Figure 6 displays the CQI and RI histogram recorded during the experiments under the pedestrian scenario. CQI is the mechanism used in LTE by the UE to report what modulation and coding scheme the eNodeB should use to achieve a nominal block error rate in the order of 10% at the link level for initial transmissions as stated in 3GPP TS36213. In terms of RI, both rank 1 (CW0) and rank 2 (CW1) were reported by the UE although it can be observed that rank 1 was the most frequently reported value. The transmission will most of the time just use one of the flows, which implies lower capacity but less interference on the second flow.

images

Figure 6 Channel quality and rank reported information in a pedestrian scenario.

Table 1 for the pedestrian scenario and Table 2 for the vehicular scenario display the number of transmissions successfully acknowledged (ACK) in the first transmission (1st) and, when they occurred, in the retransmissions (2nd, 3rd, 4th). Table 1 differentiates the acknowledged messages for rank 1 (CW0) and rank 2 (CW1).

Table 1 DL retransmissions in a pedestrian environment

% ACK CW0 % NACK CW0 % ACK CW1 % NACK CW1
1st 87.02 12.97 91.64 8.36
2nd 96.00 1.77 78.72 6.91
3rd 13.95 32.56 10.00 35.00
4th 5.41 43.24 5.56 44.44

Table 2 DL retransmissions in a vehicular environment

% ACK CW0 % NACK CW1
1st 90.83 9.17
2nd 100 0
3rd 0 0
4th 0 0

In the pedestrian scenario the first transmission has a high success (87.76%), the second transmission (i.e., the first retransmission), has also a high success rate (96%), but in the third and fourth transmissions only a small fraction of the data manages to get through. This is a symptom of bad channel conditions, where the upper layers (RLC and TCP) shall be in charge of ensuring packet delivery without loss of information by increasing transmission at their corresponding levels.

In the vehicular scenario, the selected signal to noise ratio is 10 dB lower than in the pedestrian. This has been chosen to represent the lower strength to be expected in the vehicle. It also has a higher delay spread which is aggressive in terms of multipath. Both factors result in worse propagation conditions being reported by the mobile device under test. As shown in Figure 6, all the recorded rank indications include requests for single layer (non-MIMO) transmissions (rank 1). Additionally, the reported quality (CQI) is lower than in the pedestrian case.

The different radio channel conditions impact the final QoE of the YouTube sessions. Figure 7 provides a summary of KPIs collected by the Performance Tool and a QoE score according to the method described in [14]. With only small differences in the initial buffering time, low number of re-bufferings (0) and zero re-buffering time (0) the resulting MOS score is high (>3.5). The video quality, not considered in the algorithm used to evaluate the MOS, also remains constant over time (see Figure 8).

images

Figure 7 Channel quality and rank reported information in a vehicular scenario.

Table 3 YouTube KPIs in single user scenarios

EPA EVA
Initial buffering 3.551 2.908
Playback Size (MB) 46.067 45.979
Playback Time (s) 160.638 160.982
MOS 3.89 3.89
% time in 1080p 100 100

images

Figure 8 Playback time and video quality versus time.

In terms of power consumption, only moderate differences in power consumption have been found. Figure 9 displays the averaged power consumption over time, with a sampling period of 100 ms. Only during the 10 initial seconds, averaging during 5 different sessions per test condition, the power consumption has been observed to be 11% higher in the vehicular scenario than in the pedestrian scenario (as shown in Figure 10). After the initial buffering and processing peak, the power is observed to be similar in both scenarios.

images

Figure 9 Power consumption over time.

images

Figure 10 Power consumption during initial buffering in single user scenario.

To provide more realistic insight, the experiments have been repeated allocating only 8 PRBs. This resembles a dense environment where the resources of the cell are shared with other users during the session. Under these more constrained conditions, remarkable differences between both scenarios can now be observed.

In the pedestrian scenario, the available capacity does not allow to keep the full 1080p resolution, and YouTube automatically changes the video quality to 720p (Figure 11). In the vehicular scenario, where the radio channel conditions are tougher, YouTube service reacts dropping the video quality down into the 480p format early in the playback (Figure 12).

images

Figure 11 Playback time and video quality versus time in a dense pedestrian scenario.

images

Figure 12 Playblack time and video quality versus time in a dense vehicular scenario.

Figure 13 shows the power consumption over time. An increment in the power in the pedestrian scenario can be observed. This is caused by the data being fragmented in a larger number of pieces when transmitted over the air because of the smaller amount of allocated PRBs. It must be noted that the mobile device has to process and transmit feedback in fewer occasions in the vehicular scenario because it has a lower resolution. Normalizing the power per Mbit of video the impact becomes clearer. The 480p requires 17 MB to be transmitted, whereas the 720p video generates only 29 MB of data. Thus, in the pedestrian scenario, the video can be reproduced with the double amount of data at the expense of similar power consumption.

images

Figure 13 Power consumption in dense scenarios.

Figure 13 shows the relevant KPIs and how the MOS is slightly reduced. Because the MOS score algorithm in [13] does not consider the video quality the project has identified an area of research to establish relevant KPIs in dynamic video quality adjustment conditions.

Table 4 YouTube KPIs in dense scenarios

EPA5 EVA5
Initial buffering (s) 3.325 5.416
Playback Size (MB) 29.425 17.172
Playback time (s) 157.639 159.733
MOS 3.89 3.81
% time in
1080p 5,23 5.84
720p 94.77 0
480p 0 94.16

6.1 Importance of KPIs Alignment

The synchronization of the different KPIs is very important to enable further analysis and to infer dependencies between different factors. Having synchronized the power monitoring with the application sessions has allowed TRIANGLE to easily characterize the average behaviour of multiple YouTube sessions. Even finer details can be observed within individual sessions, as the instantaneous power consumption is highly correlated with data traffic burstiness.

7 Conclusion

TRIANGLE is a research project that brings together SMEs, labs, test equipment vendors and academia, with the goal of providing an end-to-end testing ecosystem for Apps, devices, and services targeting the future 5G system. The test framework will be based on existing FIRE infrastructure and new elements for completing the ecosystem will be added to expand the testing possibilities offered by the test bed. In order to facilitate the exploitation of the test bed not only by researchers but also by App developers, device vendors, and other players in the telco market, the project will implement abstraction layers fitting the interest and knowledge of each profile. To show the potential of the test framework, some preliminary results of the impact of radio propagation conditions on a well-known service installed in one mobile have been presented.

Acknowledgment

The TRIANGLE project is funded by the European Union’s Horizon 2020 research and innovation programme, grant agreement No. 688712.

References

[1] The Rise of the Enterprise in the Smartphone Wars. New York City, NY: Goldman Sachs, 2013.

[2] Forecast Analysis: Mobile Phone Production and Semiconductors, Worldwide, 3Q13 Update. Stamford, CT: Gartner, 2013.

[3] NGMN (2015). NGMN 5G White Paper. Frankfurt: NGMN.

[4] TRIANGLE Project Official Website (2015). TRIANGLE Project Official Website. Available at: www.triangle-project.eu

[5] Díaz-Zayas, A. García-Pérez, C. A. Recio-Pérez, Á. M. and Merino, P. (2015). “PerformLTE: a testbed for LTE testing in the Future Internet,” Eds. Aguayo-Torres, Mari Carmen, Gómez, Gerardo, Poncela, and Javier, in Proceedings of the 13th International Conference on Wired and Wireless Internet Communications, Lecture Notes in Computer Science (LNCS) 9071, (Berlin: Springer), 313–326.

[6] Koc, A. T. Jha, S. C. Vannithamby R. and Torlak, M. (2014). “Device power saving and latency optimization in LTE-A networks through DRX configuration,” in Proceedings of the IEEE Transactions on Wireless Communications, Vol. 13, New York, NY, 2614–2625.

[7] Wauters, T., Vermeulen, B., Vandenberghe, W., Demeester, P., Taylor, S., Baron, L. et al. (2014). “Federation of internet experimentation facilities: architecture and implementation,” in Proceedings of the European Conference on Networks and Communications, Bologna.

[8] Álvarez, A., Díaz-Zayas, P., Merino, F., Tocado, J. R. (2012). “Field measurements of mobile services with Android Smartphones,” in Proceedings of the Consumer Communications and Networking Conference (CCNC) (Rome: IEEE), 105–109.

[9] 3GPP TS 36.101. (2015). “User Equipment (UE) Radio Transmission and Reception,” 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA). Available at: http://www.3gpp.org

[10] ETSI (2014). ETSI Network Functions Virtualisations White Paper #3.

[11] McKeown, N., Anderson. T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., et al. (2008). “OpenFlow: enabling innovation in campus networks,” Newsletter SIGCOMM Computer Communication Review 38.2 (New York, NY: ACM), 69–74.

[12] Braun, T., Diaz, M., Gabeiras, J. E., and Staub, T. (2008). End-to-End Quality of Service Over Heterogeneous Networks. (Berlin: Springer Science & Business Media).

[13] Rakotoarivelo, T., Ott, M., Jourjon, G., and Seskar, I. (2010). “OMF: a control and management framework for networking testbeds,” Newsletter ACM SIGOPS Operating Systems Review, Vol. 43 (New York, NY: ACM), 54–59.

[14] Lozano, F., Gómez, G., Aguayo-Torres, M.-C., Cárdenas, C., Plaza, A., Garrido, A., et al. (2015). Network performance testing system integrating models for automatic QoE evaluation of popular services: YouTube and Facebook. Wirel. Pers. Commun. 81, 1377–1397 doi: 10.1007/s11277-015-2479-y

Biographies

images

A. F. Cattoni received his M.Sc. and Ph.D., both from the University of Genoa, Italy, in 2004 and 2008, respectively. From 2008 to 2015 he was first a postdoctoral researcher and then an associate professor at Aalborg University, Denmark, where he investigated wireless technologies and networks ranging from cognitive radio networks, LTE-A, Wi-Fi advances, and lately to 5G. During this period he served as research consultant for Nokia Networks in 4G and 5G related research activities. In January 2016 he joined Keysight Technologies as a research engineer. His research interests are 5G systems and networks, software defined networks, and network virtualization. Andrea F. Cattoni has published more than 60 publications on international conferences, journal, and book contributions. He has been product owner of the ASGARD open source Cognitive Radio platform. He served as Danish national representative and work package leader within the COST Action IC0902 on Cognitive radio.

images

G. C. Madueño received the M.Sc. degree in mobile communications and Ph.D. degree in reliable radio access for massive machine-to- machine (M2M) communications from Aalborg University, Aalborg, Denmark, were he was responsible for the M2M laboratory. He is currently a Researcher Engineer at Keysight Technologies Denmark. His research interests include the Internet of Things and ultra-reliable communications. Dr. Madueño has served as a Technical Program Committee Member for the IEEE Journal on Selected Areas in Communications. He also served as a Reviewer for IEEE Wireless Communication Letters, IEEE Communications Magazine, and various IEEE conferences.

images

M. Dieudonné received his M.Sc. degree from De Nayer in 1999 and his MBA from Vlerick in 2005. He is managing a research department in Keysight Labs covering technologies such as Wireless communication, non-linear modelling and nanotechnologies. He coordinated the EC FP7 SAMURAI project. Over the years, he did co-author multiple papers in different domains and filed 4 granted patents.

images

P. Merino is an full professor at Universidad de Málaga (UMA) and belongs to the Mobile Networks and Software Reliability Team (MORSE). His research interests are foundations, tools, and applications of formal methods and protocol engineering for critical systems, particularly communications software. He is the main researcher at UMA in the European projects SAID, Fed4FIRE, FLEX, TRIANGLE and Q4HEALTH. He was the chair of the ERCIM Working Group on Formal Methods for Industrial Critical Systems, and member of the Executive Committee of ERCIM, one of the larger research consortium in ICT in Europe. He represents UMA in NetWorld2020 ETP and 5GPPP association.

images

A. D. Zayas is a post-doctoral researcher with the Department of Languages and Computer Science, Malaga University, Malaga, Spain. Díaz has a Ph.D. in telecommunication engineering from the same university. Her research interests include mobile communication protocols and experimental facilities. She participates in European research projects related with mobile network and experimental facilities such as Fed4FIRE and TRIANGLE. She has coauthored papers in relevant journals in the field of mobile networking and computing, like IEEE Pervasive Computing, Mobile Networks & Applications and Computer Communications, IEEE Internet Computing and IEEE Vehicular Magazine and in relevant conferences like IEEE AINA, IEEE GLOBECOM, IEEE CCNC, IEEE NOMS and IEEE IM.

images

A. Salmerón has been a researcher at the University of Málaga since 2006. His current research interests include the application of formal methods to the analysis and optimization of heterogeneous systems. Alberto earned his Ph.D. in Computer Science from the University of Málaga in 2014. He has participated in national and international projects related with systems reliability and performance, such as Rescue, SAID, and TRIANGLE.

images

F. Carlier, with a master degree in Physics and a strong background in information technology, Frederik joined Microsoft in 2008. As a consultant and architect, he oversaw major projects for both the European Institutions and NATO. Recognizing his talent for matching business needs to elegant technical solutions, Microsoft enrolled Frederik in the Master of Enterprise Architecture, further developing his business acumen. In 2013, Frederik joined Jamo Solutions, a mobile test automation company, where he worked as a technology strategist. He developed relations with the company’s partners, including clients, and service organizations and upstream partners such as HP and Microsoft. He built the company’s cloud capability through partnerships, lead initiatives to improve the ease of use of the software the company provided and briefed analyst firms on the company. In July 2014, Jamo Solutions was awarded the Frost Global Product Line strategy award as a recognition of this work. At the same time, he advised the company’s CEO with restructuring and reorganizing the company and preparing the company for capital investment. In November 2014, recognizing further potential in the market, he set out to create “Quamotion”.

images

B. S. Germain obtained a master degree in Computer Sciences at KU Leuven. Fascinated by industrial processes and planning challenges, Bart started as research fellow at the KU Leuven performing international research on a new generation of “Manufacturing Execution Systems”. This research resulted in a Ph.D. thesis elaborating a sustainable, maintainable and flexible software architecture for “Manufacturing Execution Systems”. As post-doc researcher, Bart’s main focus was on valorizing his Ph.D. thesis in an industrial context (at Materialise, a provider of additive manufacturing software solutions and 3D printing services). Meanwhile, he transferred the knowledge and architecture towards other domains: Ground processing, resulting in submitted patent: US 20130046525 A1; traffic control (see http://modum-project.eu/); Robotics. After this academic period, Bart joined Jamo Solutions as senior developer where he transitioned into a role as development manager. He guided and aligned the development, support and quality assurance team. At the same time, Bart developed a hands on experience in Android and Web automation. In November 2014, he co-founded “Quamotion” with the ambition to bring an easy-to-use, frictionless test automation solution for mobile devices to the market.

images

D. Morris is CEO of RedZinc, a telecommunications, research and consulting firm specialising in QoS business models for 5G and connected eHealth applications. Donal has played a leadership role in telecommunications research for twenty three years in the area of QoS. Technical Leader of the LiveCity Smart Cities project from 2011 to 2015 and technical leader of City Flow from 2013 to 2014. Currently playing a leader ship role In 2004 Donal established EuQoS, leading a consortium of twenty four organisations researching end to end QoS and which includes, France Telecom, Telefonica, Telecom Portugal, Ericsson and Juniper. Donal is currently chairman of the EuQoS Consortium. Prior to that Donal led the development of a DSL service execution product. In 1999, Donal established the Bandwidth2000 Consortium which developed broadband service ordering and accounting systems. In 1998 Donal established and lead the SUSIE Consortium which built a transatlantic network trial to test premium IP service charging. Prior to that Donal established and provided the leadership for the CANCAN Consortium to research broadband services charging and which included Telia, Cable & Wireless, Telefonica, Philips, Nortel and Cap Gemini. Donal was chairman of the EU “Chain” which included a number of research consortium studying accounting for new services. Prior to working in telecommunications Donal worked in the mechatronics and medical systems sectors. Donal graduated in Electronic Engineering from Dublin City University in 1986.

images

R. Figueiredo is principal engineer with RedZinc and responsible for the Design and Development of RedZinc’s Multi-Domain Virtual Path Slice Orchestrator, VELOX. Involved from IPSphere standardization trials to the current 5G work done in European Projects such as 5GEx, Q4Health and Triangle, with development done in the SAFE, LiveCity and CityFlow Projects in regards to SDN networks and LTE slicing bringing VELOX to the forefront of network bandwidth orchestration. Ricardo also helped design and oversee development of key systems of the RedZinc’s eHealth Emergency solution, Blue-Eye.

images

J. Caffrey, B.E., M.Sc., is principal engineer for RedZinc, a telecommunications, research and consulting firm specialising in QoS business models for 5G and connected eHealth applications. Jeanne is currently working in the projects TRIANGLE and Q4Health in the capacity of marketing and applications. Prior to this Jeanne worked as CEO of QuartzSpark, a telecommunications services, user and applications marketing company specialising in innovation and technology companies where she was involved in the user aspects of the schools and hospitals in the LiveCity project as well as with medical solutions company Lincor and software analytics company Jaspersoft in a marketing manager capacity. She has worked for Openet telecommunications billing software company in partner marketing. In 1999, Jeanne worked on the techno-economic aspects of the Bandwidth2000 project which developed broadband service ordering and accounting systems. In 1998 Jeanne defined and worked on a marketing program for the SUSIE Consortium which built a transatlantic network trial to test premium IP service charging. Subsequently, Jeanne worked in business development for Sherkin Communications (a consulting and software development company) in the area of IP charging. Prior to Sherkin, Jeanne worked in GEC Alsthom in the UK, in ESB International in Dublin. Jeanne has delivered projects for technology companies such as BT and Intel. Jeanne graduated with a Bachelor of (Electronic) Engineering from University College Dublin in 1991 and holds an MSc from the University of Manchester Institute of Science and Technology.

images

J. Baños received her M.Sc. (1981) and Ph.D. (1987) degrees in Telecommunications Engineering from the Polytechnic University of Madrid (Spain). She has more than 30 years of experience in mobile communication systems, project management testing for several technologies and various certification programs. She has worked for AEG-Telefunken, Alcatel, CETECOM and AT4 wireless where she is currently the head of the Test Engineering and R&D department. She has also lectured at the Polytechnic University of Madrid and the University of Málaga. Has represented AT4 wireless, CETECOM and Alcatel in several working groups and industry alliances and is a member of IEEE. She has participated in several European R&D projects and has led various national R&D projects. She has published 7 papers in international journals, 41 international technical publications and 23 national publications and is co-author of a chapter in a book.

images

C. Cardenas has more than 10 years of experience in the telecommunication industry. He has experience in aspects related to performance testing for wireless technologies, test tools specification, development and management; participation in international committees, wireless technology development and management.

images

N. Roche is the tech lead at UCL’s Centre for Entrepreneurship, working with startups at IDEA London and across Tech City. Working in tech startups for over 15 years, he teaches mobile development for the W3C and technology entrepreneurship at UCL. Niall has worked as a Software Engineer and Project Manager, specializing in mobile and telco technologies mainly in startup companies at various stages of growth. He has over 15 years’ experience in developing mobile web and cross platform native applications.

images

A. Moore is a UCL Ph.D. computer scientist with 10 years’ experience in AI/ML, UX design, marketplaces, mobile and early stage tech innovation. He was on the founding team of UCL spin-out Satalia.com and venture backed Wearepopup.com. His activities at UCL include helping create The Mobile Academy and the set up IDEA London innovation centre at the heart of TechCity with Cisco and DC Thomson. He has been a member of the Special Group for Entrepreneurship for the British Computer Society and have mentored at the Wayra Academy, Springboard Mobile (now Techstars London) & Startupbootcamp IoT. In recent years Alastair has mainly been involved in helping design and administer innovation/startup support programmes, most recently at UCL Centre for entrepreneurship. Current activities include projects on innovation in business models for 5G networked environments (SME funding available), scale-up support programme for SME’s requiring Big Data/Analytics expertise (free support for SMEs available) and commercialisation of smart cities platform using Sematic Web technology.

Abstract

Keywords

1 Introduction

2 The TRIANGLE Project

2.1 Why TRIANGLE?

2.2 Benchmarking-as-a-Service

2.3 The TRIANGLE-MARK

2.4 The TRIANGLE Approach

images

2.5 Open to Experimenters

3 TRIANGLE Test Bed

images

3.1 Mobile Device Monitoring

3.2 Network Emulation

3.3 App Testing

images

4 Test Bed Orchestration and Management

images

5 Experiments Setup

6 Results

images

images

images

images

images

images

images

images

images

6.1 Importance of KPIs Alignment

7 Conclusion

Acknowledgment

References

Biographies