FaaSnap

目前看其是自己用go+py实现了个简单的测试框架, 用来准备环境/启动VM/跑测试/抓trace. 其中有些code似乎是从vHive中复制的.

其中trace相关有用到zipkin和bpftrace. 前者后面可以调查下.

Fn Workloads

源码分析

先从已有知识出发: 我们想测试的是snapshot restoration time, 所以核心的东西应该包括a) VM启动; b) 打snapshot; c) snapshot恢复; d) 停止测试; e) 关键时间片的统计;

测试框架

FaaSnap的测试框架整体还是C-S结构, 中间由swagger API定义连接. 从API定义出发:

可以看到我们关心的主要包括:

• VM的
  1. 创建POST /vms;
  2. 查询GET /vms/{vmId};
  3. 删除DELETE /vms/{vmId};
• Snapshot的
  1. 创建POST /snapshots;
  2. 复制PUT /snapshots;
  3. 修改PATCH /snapshots/{ssId}
  4. 另外FaaSnap的mincore相关设计:
     1. 查询GET /snapshots/{ssId}/mincore
     2. 复制PUT /snapshots/{ssId}/mincore
     3. 修改PATCH /snapshots/{ssId}/mincore
     4. layering POST /snapshots/{ssId}/mincore
  5. 以及REAP的相关设计:
     1. 查询GET /snapshots/{ssId}/reap
     2. 修改PATCH /snapshots/{ssId}/reap
     3. 删除DELETE /snapshots/{ssId}/reap
• 函数
  1. 注册POST /funcitons
  2. 调用POST /invocations

测试代码test.py中的API调用:

# 大体流程
run -> (run_snap | run_warm)
	run_snap -> (prepareVanilla | prepareMincore | prepareReap | prepareEmuMincore) + invoke
		prepareVanilla
		prepareMincore
		prepareReap
		prepareEmuMincore
    run_warm -> invoke_warm

(run_snap)           PUT /net-ifaces/{namespace}
                     POST /functions
    (prepareVanilla) POST /vms
                     POST /invocations
                     POST /snapshots
                     DELETE /vms/{vmId}                         
                     PUT /snapshots                               
                     PATCH /snapshots/{ssId}         # drop cache
    (invoke)         POST /invocations
                     DELETE /vms/{vmId}

(run_warm)           PUT /net-ifaces/{namespace}
                     POST /functions
                     POST /vms
                     POST /invocations
    (invoke_warm)    POST /invocations
                     DELETE /vms/{vmId}

# prepareVanilla
POST /vms
POST /invocations
POST /snapshots
DELETE /vms/{vmId}                         
PUT /snapshots                               
PATCH /snapshots/{ssId}         # drop cache

# prepareMincore
POST /vms
POST /snapshots
DELETE /vms/{vmId}
PATCH /snapshots/{ssID}         # drop cache
POST /invocations
POST /snapshots
DELETE /vms/{vmId}
PUT /snapshots/{ssId}/mincore   # carry over mincore to new snapshot
PATCH /snapshots/{ssId}/mincore
PUT /snapshots/{ssId}
PATCH /snapshots/{ssID}         # drop cache
PATCH /snapshots/{ssID}         # drop cache
PATCH /snapshots/{ssId}/mincore

# prepareReap
POST /vms
POST /invocations
POST /snapshots
DELETE /vms/{vmId}
PATCH /snapshots/{ssID}         # drop cache
POST /invocations
DELETE /vms/{vmId}
PATCH /snapshots/{ssID}         # drop cache
PATCH /snapshots/{ssID}/reap    # drop reap cache

# prepareEmuMincore
POST /vms
POST /invocations
POST /snapshots
DELETE /vms/{vmId}
PATCH /snapshots/{ssID}         # drop cache
POST /invocations
DELETE /vms/{vmId}
PATCH /snapshots/{ssID}/reap    # drop reap cache
PATCH /snapshots/{ssID}/mincore 
PATCH /snapshots/{ssID}         # drop cache

目前主要关心的点是FaaSnap如何通过mincore扫描来获取loading set的?

从上面分析来看, 其在prepareMincore中两次通过POST /snapshots创建快照, 而其他方法均只创建了一次. 且其第二次创建快照之前触发了函数调用, 而第一次快照前并没有. 所以应该是通过两次快照之间的差距来计算出loading set. 注意到触发函数调用的过程中(InvokeFunction函数)调用了ScanMincore且整个代码库仅有一次调用. 可以推测这里就是loading set的创建过程. 阅读源码看到, ScanMincore前有判断当前snapshot的mincoreLayers是否空, 如果为空则触发ScanMincore. 这与刚才发现的两次快照以及触发函数后快照的行为非常吻合. 另外由test.py传入的参数mincore=-1 (scanInterval=-1)可以看到实际上是调用了ScanFileMincoreBySize的持续扫描. 每当RSS大小增长mincore_size个页(1024)时扫描一次.

总结一下FaaSnap:

• 在调用函数时触发mincore扫描
• 每当RSS增长阈值大小后调用mincore扫描

接下来来看看REAP的实现. 阅读InvokeFunction代码发现其在加载快照之前通过reap.Activate激活了REAP. 此时REAP通过FetchState中的fetchState读取了内存文件, 将其预取到pagecache中. 关于REAP的working set, 其同样是在reap.Activate中启动的. 启动函数为mmanger.Activate(), 其中使用了epoll来异步处理uffd ¹的缺页. 缺页处理函数s.servePageFault则在某(页)地址第一次缺页时将其记录为working set. 其记录模式类似于logging, 在每次关闭VM的时候由ProcessRecord函数在writeWorkingSetPagesToFile中回写到snapshot文件. ¹: 这里注意uffd其实是由修改过的firecracker注册并启动的.

总结一下REAP:

• 启动VM时
  • 通过读取到pagecache预先加载上一次记录的working set
  • 启动uffd的处理函数来以log形式记录触碰到的内存页地址
• 关闭VM时
  • 将所有记录到属于working set的地址对应内容写入working set文件