RFC 8867 | Test Scenarios for RMCAT | January 2021 |
Sarker, et al. | Informational | [Page] |
The Real-time Transport Protocol (RTP) is used to transmit media in multimedia telephony applications. These applications are typically required to implement congestion control. This document describes the test cases to be used in the performance evaluation of such congestion control algorithms in a controlled environment.¶
This document is not an Internet Standards Track specification; it is published for informational purposes.¶
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.¶
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8867.¶
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
This memo describes a set of test cases for evaluating congestion control algorithm proposals in controlled environments for real-time interactive media. It is based on the guidelines enumerated in [RFC8868] and the requirements discussed in [RFC8836]. The test cases cover basic usage scenarios and are described using a common structure, which allows for additional test cases to be added to those described herein to accommodate other topologies and/or the modeling of different path characteristics. The described test cases in this memo should be used to evaluate any proposed congestion control algorithm for real-time interactive media.¶
The terminology defined in RTP [RFC3550], RTP Profile for Audio and Video Conferences with Minimal Control [RFC3551], RTCP Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback (RTP/AVPF) [RFC4585], and Support for Reduced-Size RTCP [RFC5506] applies.¶
All the test cases in this document follow a basic structure allowing implementers to describe a new test scenario without repeatedly explaining common attributes. The structure includes a general description section that describes the test case and its motivation. Additionally the test case defines a set of attributes that characterize the testbed, for example, the network path between communicating peers and the diverse traffic sources.¶
Every test case needs to define an evaluation testbed topology. Figure 1 shows such an evaluation topology. In this evaluation topology, S1..Sn are traffic sources. These sources generate media traffic and use the congestion control algorithm(s) under investigation. R1..Rn are the corresponding receivers. A test case can have one or more such traffic sources (S) and their corresponding receivers (R). The path from the source to destination is denoted as "forward", and the path from a destination to a source is denoted as "backward". The following basic structure of the test case has been described from the perspective of media-generating endpoints attached on the left-hand side of Figure 1. In this setup, the media flows are transported in the forward direction, and the corresponding feedback/control messages are transported in the backward direction. However, it is also possible to set up the test with media in both forward and backward directions. In that case, unless otherwise specified by the test case, it is expected that the backward path does not introduce any congestion-related impairments and has enough capacity to accommodate both media and feedback/control messages. It should be noted that, depending on the test cases, it is possible to have different path characteristics in either of the directions.¶
In a testbed environment with real equipment, there may exist a significant amount of unwanted traffic on the portions of the network path between the endpoints. Some of this traffic may be generated by other processes on the endpoints themselves (e.g., discovery protocols) or by other endpoints not presently under test. Such unwanted traffic should be removed or avoided to the greatest extent possible.¶
defines the end-to-end transport level path characteristics of the testbed for a particular test case. Two sets of attributes describe the path characteristics, one for the forward path and the other for the backward path. The path characteristics for a particular path direction are applicable to all the sources "S" sending traffic on that path. If only one attribute is specified, it is used for both path directions; however, unless specified the reverse path has no capacity restrictions and no path loss.¶
defines the traffic source behavior for implementing the test case:¶
defines the characteristics of the media sources. When using more than one media source, the different attributes are enumerated separately for each different media source.¶
describes the media encoder behavior. It defines the main parameters that affect the adaptation behavior. This may include but is not limited to the following:¶
More detailed discussions on expected media source behavior, including those from synthetic video traffic sources, can be found in [RFC8593].¶
describes the characteristics of the competing traffic source, the different types of competing flows are enumerated in [RFC8868].¶
Any attribute can have a set of values (enclosed within "[]"). Each member value of such a set must be treated as different value for the same attribute. It is desired to run separate tests for each such attribute value.¶
The test cases described in this document follow the above structure.¶
This section describes recommended test case settings and could be overwritten by the respective test cases.¶
To evaluate the performance of the candidate algorithms, the implementers must log enough information to visualize the following metrics at a fine enough time granularity:¶
Flow level:¶
Transport level:¶
Each path between a sender and receiver as described in Figure 1 has the following characteristics unless otherwise specified in the test case:¶
Examples of additional network parameters are discussed in [RFC8868].¶
For test cases involving time-varying bottleneck capacity, all capacity values are specified as a ratio with respect to a reference capacity value, so as to allow flexible scaling of capacity values along with media source rate range. There exist two different mechanisms for inducing path capacity variation: a) by explicitly modifying the value of physical link capacity, or b) by introducing background non-adaptive UDP traffic with time-varying traffic rate. Implementers are encouraged to run the experiments with both mechanisms for test cases specified in Section 5.1, Section 5.2, and Section 5.3.¶
Unless otherwise specified, each test case will include one or more media sources as described below:¶
Video¶
Audio¶
In this test case, the minimum bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [RFC8836], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:¶
It should be noted that the exact variation in available capacity due to any of the above depends on the underlying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviors in a certain network environment.¶
This test uses bottleneck path capacity variation as listed in Table 1.¶
When using background non-adaptive UDP traffic to induce a time-varying bottleneck, the physical path capacity remains at 4 Mbps, and the UDP traffic source rate changes over time as (4 - (Y x r)), where r is the Reference bottleneck capacity in Mbps, and Y is the path capacity ratio specified in Table 1.¶
Variation pattern index | Path direction | Start time | Path capacity ratio |
---|---|---|---|
One | Forward | 0 s | 1.0 |
Two | Forward | 40 s | 2.5 |
Three | Forward | 60 s | 0.6 |
Four | Forward | 80 s | 1.0 |
This test case is similar to Section 5.1. However, this test will also consider persistent network load due to competing traffic.¶
Variation pattern index | Path direction | Start time | Path capacity ratio |
---|---|---|---|
One | Forward | 0 s | 2.0 |
Two | Forward | 25 s | 1.0 |
Three | Forward | 50 s | 1.75 |
Four | Forward | 75 s | 0.5 |
Five | Forward | 100 s | 1.0 |
Real-time interactive media uses RTP; hence it is assumed that RTCP, RTP header extension, or such would be used by the congestion control algorithm in the back channel. Due to the asymmetric nature of the link between communicating peers, it is possible for a participating peer to not receive such feedback information due to an impaired or congested back channel (even when the forward channel might not be impaired). This test case is designed to observe the candidate congestion control behavior in such an event.¶
It is expected that the candidate algorithms are able to cope with the lack of feedback information and to adapt to minimize the performance degradation of media flows in the forward channel.¶
It should be noted that for this test case, logs are compared with the reference case, i.e., when the backward channel has no impairments.¶
Variation pattern index | Path direction | Start time | Path capacity ratio |
---|---|---|---|
One | Forward | 0 s | 2.0 |
Two | Forward | 20 s | 1.0 |
Three | Forward | 40 s | 0.5 |
Four | Forward | 60 s | 2.0 |
Variation pattern index | Path direction | Start time | Path capacity ratio |
---|---|---|---|
One | Backward | 0 s | 2.0 |
Two | Backward | 35 s | 0.8 |
Three | Backward | 70 s | 2.0 |
In this test case, more than one media flow share the bottleneck link, and each of them uses the same congestion control algorithm. This is a typical scenario where a real-time interactive application sends more than one media flow to the same destination, and these flows are multiplexed over the same port. In such a scenario, it is likely that the flows will be routed via the same path and need to share the available bandwidth amongst themselves. For the sake of simplicity, it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows individually. While this appears to be a variant of the test case defined in Section 5.2, it focuses on the capacity-sharing aspect of the candidate algorithm. The previous test case, on the other hand, measures adaptability, stability, and responsiveness of the candidate algorithm.¶
Flow ID | Media type | Start time | End time |
---|---|---|---|
1 | Video | 0 s | 119 s |
2 | Video | 20 s | 119 s |
3 | Video | 40 s | 119 s |
4 | Audio | 0 s | 119 s |
5 | Audio | 20 s | 119 s |
6 | Audio | 40 s | 119 s |
In this test case, multiple media flows share the bottleneck link, but the one-way propagation delay for each flow is different. For the sake of simplicity, it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows. While this appears to be a variant of test case 5.2 (Section 5.2), it focuses on the capacity-sharing aspect of the candidate algorithm under different RTTs.¶
Flow ID | Media type | Start time | End time |
---|---|---|---|
1 | Video | 0 s | 299 s |
2 | Video | 10 s | 299 s |
3 | Video | 20 s | 299 s |
4 | Video | 30 s | 299 s |
5 | Video | 40 s | 299 s |
6 | Audio | 0 s | 299 s |
7 | Audio | 10 s | 299 s |
8 | Audio | 20 s | 299 s |
9 | Audio | 30 s | 299 s |
10 | Audio | 40 s | 299 s |
In this test case, one or more media flows share the bottleneck link with at least one long-lived TCP flow. Long-lived TCP flows download data throughout the session and are expected to have infinite amount of data to send and receive. This is a scenario where a multimedia application coexists with a large file download. The test case measures the adaptivity of the candidate algorithm to competing traffic. It addresses requirement 3 in Section 2 of [RFC8836].¶
Includes the following metrics in addition to those described in Section 4.1:¶
Additionally, implementers are encouraged to run the experiment with multiple media sources.¶
In this test case, one or more congestion-controlled media flows share the bottleneck link with multiple short-lived TCP flows. Short-lived TCP flows resemble the on/off pattern observed in web traffic, wherein clients (for example, browsers) connect to a server and download a resource (typically a web page, few images, text files, etc.) using several TCP connections. This scenario shows the performance of a multimedia application when several browser windows are active. The test case measures the adaptivity of the candidate algorithm to competing web traffic, and it addresses requirement 1.E in Section 2 of [RFC8836].¶
Depending on the number of short TCP flows, the cross traffic either appears as a short burst flow or resembles a long-lived TCP flow. The intention of this test is to observe the impact of a short-term burst on the behavior of the candidate algorithm.¶
Includes the following metrics in addition to those described in Section 4.1:¶
In this test case, more than one real-time interactive media flow share the link bandwidth, and all flows reach to a steady state by utilizing the link capacity in an optimum way. At this stage, one of the media flows is paused for a moment. This event will result in more available bandwidth for the rest of the flows as they are on a shared link. When the paused media flow resumes, it no longer has the same bandwidth share on the link. It has to make its way through the other existing flows in the link to achieve a fair share of the link capacity. This test case is important specially for real-time interactive media, which consists of more than one media flows and can pause/resume media flows at any point of time during the session. This test case directly addresses requirement 5 in Section 2 of [RFC8836]. One can think of it as a variation of the test case defined in Section 5.4. However, it is different as the candidate algorithms can use different strategies to increase efficiency, for example, in terms of fairness, convergence time, oscillation reduction, etc., by capitalizing on the fact that they have previous information of the link.¶
Includes the following metrics in addition to those described in Section 4.1:¶
The general description of the testbed parameters are the same as Section 5.4 with changes in the test-specific setup as below:¶
Other test-specific setup:¶
It has been noticed that there are other interesting test cases besides the basic test cases listed above. In many aspects, these additional test cases can help further evaluation of the candidate algorithm. They are listed below.¶
In this test case, media flows will have different priority levels. This is an extension of Section 5.4 where the same test is run with different priority levels imposed on each of the media flows. For example, the first flow (S1) is assigned a priority of 2, whereas the remaining two flows (S2 and S3) are assigned a priority of 1. The candidate algorithm must reflect the relative priorities assigned to each media flow. In this case, the first flow (S1) must arrive at a steady-state rate approximately twice that of the other two flows (S2 and S3).¶
The candidate algorithm can use a coupled congestion control mechanism [RFC8699] or use a weighted priority scheduler for the bandwidth distribution according to the respective media flow priority or use.¶
This test case requires running all the basic test cases with the availability of Explicit Congestion Notification (ECN) [RFC6679] feature enabled. The goal of this test is to exhibit that the candidate algorithms do not fail when ECN signals are available. With ECN signals enabled, the algorithms are expected to perform better than their delay-based variants.¶
In this test case, one congestion-controlled media flow, S1->R1, traverses a path with multiple bottlenecks. As illustrated in Figure 7, the first flow (S1->R1) competes with the second congestion-controlled media flow (S2->R2) over the link between A and B, which is close to the sender side. Again, that flow (S1->R1) competes with the third congestion-controlled media flow (S3->R3) over the link between C and D, which is close to the receiver side. The goal of this test is to ensure that the candidate algorithms work properly in the presence of multiple bottleneck links on the end-to-end path.¶
Additional wireless network (both cellular network and Wi-Fi network) specific test cases are defined in [RFC8869].¶
The security considerations in Section 6 of [RFC8868] and the relevant congestion control algorithms apply. The principles for congestion control are described in [RFC2914], and in particular any new method must implement safeguards to avoid congestion collapse of the Internet.¶
The evaluation of the test cases are intended to be run in a controlled lab environment. Hence, the applications, simulators, and network nodes ought to be well-behaved and should not impact the desired results. Moreover, proper measures must be taken to avoid leaking nonresponsive traffic from unproven congestion avoidance techniques onto the open Internet.¶
This document has no IANA actions.¶
Much of this document is derived from previous work on congestion control at the IETF.¶
The content and concepts within this document are a product of the discussion carried out within the Design Team.¶