../06-rebuttal

总体来看攻击novelty以及evaluation的最多. 没有什么说我们的问题不重要设计不够好.

回应限制2000字.

  1. A看起来主要是因为我们没有比较DAMON的range方法觉得novelty不够.

    1. (security)
    2. (performance) I do not find it entirely clear why guest-managed tiered memory would inherently be more performant and scalable.
    3. (cloud) cloud providers and VM guests sacrificing CPU cycles for memory savings
    4. (motivation) why the guest should be able to make decisions based on observations from the guest’s virtual address space
    5. (novelty) What are the fundamental differences between DAMON?
    6. (evaluation) No hypervisor-based tiering.
    7. (evaluation) GUPS variation, no skewed workload
    8. (evaluation) TPP is best in Graph500

  2. B看起来比较nice. 主要也是找的evaluation的问题.

    1. (figure) results hard to parse; figures unreadable
    2. (evaluation) performance is too good to be true
    3. (evaluation) why SOTA so bad?
    4. (evaluation) Are we seeing a pathologically bad workload?
    5. (figure) 4/9/10 should be bigger

  3. C看起来是利益相关方. 认为我们guest/host指责分配是HeteroOS的功劳. 得详细比较一下.

  4. D看起来是security background. 我们需要用他的思路通过threat model讲一下. 另外D看起来是个好人, 不仅肯定了我们的novelty, 也提出了改进意见.

    1. (security) The threat model is not clear enough.
    2. (security) Security concerns with malicious VMs
    3. (security) Is the hypervisor trusted in your threat model?
    4. (security) How to handle potential manipulations of memory access information by malicious guest VMs?

  5. E看起来不太懂, 只关注evaluation以及细枝末节的东西.

感谢reviewer对我们以下贡献的肯定:

balbalbalba

我们将address以下问题:

Guest modifications

The guest part of our design is fully contained in the kernel modules. Cloud vendors aleady provide customized linux distributions such as Amazon Linux with custom kernel tunning, we believe bundle additional modules into such distribution requires no guest modifications.

GUPS

Our GUPS implementation is a work-stealing variant and is entirely moticated by reproducibility. The original GUPS assigns each thread with a fixed amout of read-modify-write operations at startup, early terminating thread will stay idle and cobbor the execution time and overall throughput measurements.

Review #522A 2 | 3

HyperTier is a tiered memory management scheme designed with virtualization in mind. It delegates the memory management operations to the guest, recognizing that guests can safely utilize hardware performance counters. By incorporating range-based hotness detection, leveraging PEBS counters, and implementing a more efficient page migration mechanism within the guest, HyperTier enables effective hot-cold page placement decisions.

Strengths and weaknesses

Strengths:

Weaknesses:

Detailed Comments for authors

However, I believe the ==contributions of this work are somewhat incremental==. The paper seems to ==lack some key related work that it builds upon or reimplements==, and the ==evaluation does not provide sufficient evidence to strongly support the proposed benefits==.

Motivation

Motivation for shifting the tiering responsibilities to the guest is not entirely clear in the paper:

Design & Novelty

Evaluation

Review #522B 3 | 3

HyperTier is a system for dynamic data placement in a multi-tiered memory system, targeted at virtual machine hosting, and dividing the task between an in-VM (guest) component and a hypervisor (host) component. The guest component is responsible for tracking the access frequency of pages and migrating them between fast (local) memory and slow (remote) memory, exposed to it as two NUMA nodes. The host component is responsible for balancing the allocation of fast and slow memory between VMs, using a memory ballooning driver within the guest OS.

Hotness tracking in the guest component is implemented by sampling hardware performance counter events (with PEBS) to estimate the access frequency of memory segments. Segments are dynamically split and merged to minimise bookkeeping overhead on the assumption that hot regions are relatively sparse and clustered. ==The host component supports overprovisioning, but no details are provided on the algorithms used at this level.==

Evaluation is performed on VMs with 16G of guest memory on a system with 128GB local DRAM as fast memory, and two types of slow memory: 512GB of locally-attached Optane DIMMs, and 128GB of NUMA-remote DRAM. Benchmarks seem to suggest a factor of 2 or better throughput improvement on memory-intensive workloads compared to to the next-best alternative included in the comparison.

Summary of strengths and weaknesses

Pro

Con

Questions for authors’ response

Detailed Comments for authors

The figures are compressed beyond the point of legibility - they need to be significantly bigger. Figures 4, 9, and 10 are particularly bad, and are essentially unreadable.

Review #522C 1 | 4 (Sudarsun Kannan, Ada Gavrilovska)?

The paper describes a solution for managing tiered memory in virtualized clouds. The authors claim as contributions principles regarding memory management in virtualized tier memory systems, system components at the hypervisor and guest OS level based on those principles, and their the evaluation and open sourced implementation.

Summary of strengths and weaknesses

Strengths:

Weaknesses:

Detailed Comments for authors

I appreciate the work presented in this paper. Open-sourcing access to the functionality presented in the paper is timely given interest in using disaggregated memory in virtualized datacenters.

Particularly valuable are the implementation of the efficient hotness tracking mechanism and guest-hypervisor coordination, which are based on effective use of hardware features for more efficient implementations on hotness tracking and guest-hypervisor interactions.

The use of range filtering to focus hotness tracking to most likely/most relevant memory regions (as opposed to relying on host hints) is an important mechanism.

However, the authors’ ==claim to other significant novel contributions to virtualizing tiered memory systems, are overstated.== The principle of ==separating the data placement policy from the low level provisioning of different memory tiers is the main approach advertised by the HeteroOS work cited in this paper.== The main difference in that work is that, due to lack of present-day hardware features the design relied on hypervisor support for tracking hotness. The HeteroOS paper discusses leaving placement decisions to the guest, adjusting the local/fast and remote/slow memory capacities across tenants, and other concepts mentioned in this paper. So, while it is still very valuable to adapt these ideas to the features of present-day technologies, the novelty in this paper is overclaimed.

说明我们用词问题. 我们实际上想说... 另外我们bridge gap between HeteroOS‘ host hotness info with guest data placement. 我们更往前一步.

Hardware

Hardware support for hotness tracking can be found early as 2008 when Intel introduced PEBS into Nehalem microarchitecture. And PML was also integrated into Linux in 2015. Where HeteroOS is published 2017, we believe

PEBS: Nehalem

https://lore.kernel.org/all/20150205145248.GA14367@potion.redhat.com/T/

The evaluation of the work shows the effectiveness of the PEBS-supported hotness tracking combined with the range-based filtering. It seems that ==the end-to-end benefits shown in the paper are really a function of having more or better hotness information, and/or more lightweight way to track hotness.== ==It’s not clear that there are better placement decisions, or better/more elastic sizing decision.== Efficient and effective hotness tracking is an important problem, and my suggestion would be to focus the revision of the paper on this aspect of your contributions.

我们就是认为在guest中能做更好的tracking来enable更好的placement. tracking本身就是一个non trivial问题. 我们就是希望kernel-based TM来benefit 虚拟化环境. 我们就是在任意一个given sizing情况下做到最好的placement decisions.

Review #522D 2 | 3

This paper presents HyperTier, a novel system for managing tiered memory in virtualized cloud environments. HyperTier addresses the limitations of existing hypervisor-based tiered memory management systems by enabling guest operating systems to manage their own memory tiers, while the hypervisor focuses on providing isolated and elastic tiered memory provisioning. The key components of HyperTier include HyperFlex, which provides elastic tiered memory provisioning to guest VMs, and HyperPlace, which optimizes tiered memory management within the guest OS. Evaluation results show that HyperTier can improve performance and efficiency compared to existing systems.

Strengths

Weaknesses

Questions for authors’ response

  1. Is the hypervisor trusted in your threat model?
  2. How to handle potential manipulations of memory access information by malicious guest VMs?

Detailed Comments for authors

This paper presents a novel and promising approach to managing tiered memory in virtualized cloud environments. The proposed HyperTier system addresses several limitations of existing systems, such as the lack of isolation and elasticity, and demonstrates significant performance improvements. However, there are a few concerns and suggestions that the authors may want to consider.

==The motivation for isolation in this paper is unclear==, as the threat model is ambiguous. It is mentioned that "memory access patterns, remains confidential, not just from other tenants but also from the hypervisor itself". Does it mean the hypervisor is untrusted? Actually, this ==design itself cannot protect the VM from attacks by a malicious hypervisor==, while previous works have more practical advantages, such as being more transparent and requiring no or fewer modifications to the guest OS. Without Confidential VM or TEE technologies, a malicious host OS/hypervisor can easily access VM data. The paper should clarify the threat model and design goals in Chapter 3, perhaps ==shifting the focus of isolation to "safety and reliability" instead of "security" or "confidentiality."==

==There are some security concerns with malicious VMs.== While this paper claims that HyperTier enhances security and isolation by delegating data placement to guest VMs, it does not fully address the issue of malicious guest VMs which can manipulate memory access information. ==A malicious VM could potentially modify the HyperPlace component or directly change page tables to mislead the hypervisor and gain unfair access to resources.== This could lead to performance degradation or even denial-of-service attacks for other VMs on the same host.

我们希望malicious VM不会在placement中gain more advantages. 同样我们的架构天然适合implement defense techniques 比如...

Specifically, ==a malicious guest could deliberately block the HyperFlex virtio front-ends to prevent the hypervisor from reclaiming fast memory (FMEM) through the sub-balloon inflation mechanism.== This could lead to an unfair allocation of FMEM, depriving other VMs on the same host of crucial resources and potentially impacting their performance. 1

Some other questions:

1

hypervisor有更高级的控制权, 如果virtio被block可以直接让VM停机. 可以在hypervisor中implement一些defense. 2: 我们是job-stealing gups, 增加了任务分配的公平性. 3: [23, 24]中十年内从1.5TiB/30thread到4TiB/144thread. 4: 我们的设计build atop virtio来实现异步, 额外的异步支持还有...

Review #522E 2 | 3

This paper proposes a framework that uses ballooning techniques and a novel replacement/migration algorithm to handle tiered memory in virtualized environments.

Strengths

Weaknesses

Detailed Comments for authors

Thank you for submitting this paper to OSDI. I enjoyed the key idea of using the performance counters and the replacement/migration algorithms proposed. In general, the paper solves an important solution and proposes a reasonable system design to tackle the problem. However, there are quite a few problems with the evaluation section.

Somehow, most of the graphs in this paper are missing the units, and some graphs do not even have axis legends. Furthermore, in several cases, the captions also do not report the units. This makes all the results hard to interpret without a lot of guessing, which is very problematic for any paper. Furthermore, I found the evaluation section hard to follow for other reasons too. For example, there is a lot of emphasis on presenting end-to-end performance results but very little on analyzing and discussing the low-level metrics that justify the good results. ==There is also a lack of analysis on the configuration space and the impact on application latency (including tail latency) for interactive applications.==

We believe large memory capacity is mainly needed by analytical applications, for which we mainly presents throughput results. Despite this, we have included a responsiveness evaluation in figure 5, where our design is able to most agilly place hot data shown as the slope of curve. Our sampling design naturally avoid the batching tradeoff. We actively drain the sample buffer during every context switch, the frequency of which is so high that application should never be delayed due to a buffer full interrupt. What's more, our evaluated applications contains an interactive/transactional database, silo, we are happy to include the average and tail latency results of such application.

The key to our great end-to-end improvement is the low cost hotness tracking and range-based classification, for which we present an overhead breakdown study in figure 7 and a configuration space exploration in figure 8. Our design is able to classify the most amount of hot data in the most agile manner, shown as the peak value and slope in figure 5, while inhibits the lowest overhead, both overall and in the sampling stage, as shown in figure 7. In figure 8, we evaluate the configuration space of both the tracking and classification stages. For the sampling stage, we present a performance matrix across what samples to capture (load latency threshold) and how fast should we sample (sample peroid). And for the classification stage, we presents a similar matrix across how sensitive to hotness difference across ranges (split threshold) and how frequent should we react to collected samples (split period).

In the meantime, system administrators are able to select the most suitable parameters according to such performance matrix across their representative applicaitons. Our tracking parameters depends on PMU hardware. Cloud vendors usually have a large amount of physical machines with similar CPU microarchitectures which share similar PMU hardware. Parameters selected on one platform should be easily expanded to other machines.

The analysis of application latency is particularly important for such a system because many improvements can be achieved by batching actions (e.g., reducing sampling rates) at the expense of sacrificing application responsiveness (or accuracy), which can be as important or more than throughput.

The range-based algorithm proposed seems mostly orthogonal to the use of performance counters and seems generic enough to be usable for general replacement policy in other contexts. Is this the case, or ==is there a reason why this algorithm only works well in the context of virtualized tiered memory?==

是为了降低开销设计的算法. 没有imply不能推广.

==Some claims/arguments about the security of the existing and the proposed system are debatable,== to say the least. In particular, the claim that designs that require introspection of the VM to read the guest page tables are insecure because they require the hypervisor to handle sensitive data is problematic. ==In general, security claims or concerns should be made under a concrete threat model, which this paper does not provide.== But there is no objective reason to say that there is a difference in the security of introspection approaches and the proposed approach, given that they both allow the hypervisor to read the guest's memory. Whether or not the hypervisor is required to analyze it does not affect security. Furthermore, it is typical for the hypervisor to have to read the guest data when migrating pages.

==Who clears the memory when it gets returned from the VM?== 6 Is it the responsibility of the guest, the host, or both? It seems hard to imagine that in typical cloud settings either the guest or the host would likely fully trust the other party to do it. So, I would imagine that in a realistic setting, both the guest and host would clear a returned page, which is inefficient.

==There are many configuration options in this paper, and in the end, it is not very clear how dependent these options are on the specific hardware configuration== and, if so, how would an administrator configure the system. This applies to the sampling frequency (for which there is no unit in 4.2) and the "hotness management" algorithms.

fig 8中explore了最重要的几个param

The concern with the cycles reported in 4s.2 is not clear. Why is there a need to consume the entire CPU budget? If a certain low sampling frequency is enough to produce accurate results, would it not be a plus that the system consumes less CPU resources than the budget allows?

每个vm都做自己的tracking, 希望整机开销最低.

5

GUPS 本身就是unit 6: Balloon driver takes care of the cleaning.

Junk

A core insight bethind our work is that existing hypervisor-based hotness tracking all comes with great performance penalties. HeteroVisor/HeteroOS (introduced in section 3.1) and MMU-notifier based solution (proposed by reviewer A) capture A-bit hotness information by trapping every guest pagetable modification. That implies an expensive trap on every first access to a clean page. vTMM use PML hardware to track pagetable modifications, it only reduces the trap amount by a constant factor, but can not eliminate them. Making thing worse, for all these designs, further guest page table walks are also needed for translation to host-understandable addresses.

HeteroOS, published in 2017, did not solve the crux of hotness tracking, instead they chose the most expensive software-based method, despite the availablility of hardware features including PEBS and PML, which are introduced in 2008 with Nehalem, and 2015 with Boardwell respectively. Their major contribution lies in delegating only page migraiton into the guest.

Out of such insight we propose the core idea (thanks reviewer D for pointing out) of fully disaggregating data placement responsibilities—not only data migrations but most importantly hotness tracking—into guests from tiered memory provisioning in the host. We leverage PEBS that produce samples with ready-to-use guest virtual address that are directly written to buffer in guest memory. With routine buffer draining during context switches, we eliminate trapping or VMexits. Our identification algorithm operates in virtual address space, further eliminting the need for pagetable walks. The choice of virtual address also avoids the fragmentation issue often found in physical space caused by linux page allocator and map-on-first-touch policy which clobbers spatial locality.

Another insight is that apart from low-cost tracking, accurate identification is another essence for making tiered mory performant, for which we demonstrate our range-based hotness identification, while migration only carries out the decision made during identificaiton. We are indeed inspired by DAMON[Middleware'19][HPDC'22], we thank their contribution and appolgize for unintentionally leaving out due to space constraints. However, DAMON is a region based memory profiling tools that are both inaccurate and requires user interaction. DAMON relies on a randomly selected page's A-bit access information to estimate the whole region's hotness from which that page is selected. Although DAMON also performs region split, such action is user configured, DAMON will only produce a user instructed number of ranges. Such design might be helpful for manual performance tunning, but is unable to serve as a full fledged automatic hotness identification solution and facilate data placement of a wide range of applications.

HeteroOS

We are very grateful to receive recognition from reviewers A, B, D, and E for our contribution as the first work to fully disaggregate data placement responsibilities—not only data migrations but also hotness tracking—into guests from tiered memory provisioning in the host. While HeteroOS exploits guest access information, hotness tracking is still performed in the host. As introduced in Section 3.1, this design choice, combined with their para-virtualization architecture, necessitates triggering a page fault on every guest page table modification to capture hotness information, severely hindering performance and eliminating the possibility of real-world adoption. As reviewer C aptly noted, emerging modern hardware motivates us to consider a novel responsibility assignment that leverages the newly available capabilities provided by PEBS to realize such disaggregation.

I am also deeply sorry for the negligence of limited evaluation and crowded figures due to the desire to elaborate our complete design in great detail and present our results across a wide range of applications.