feat: add manual disk discovery and weekly capacity cache
This commit is contained in:
@@ -3,3 +3,8 @@ __pycache__/
|
|||||||
*.pyc
|
*.pyc
|
||||||
*.pyo
|
*.pyo
|
||||||
*.log
|
*.log
|
||||||
|
.data/
|
||||||
|
dist/
|
||||||
|
/storage-release.json
|
||||||
|
/ops/deploy-storage.py
|
||||||
|
/ops/release-storage.py
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
# PIGWay Raspberry Pi Agent v3.7.1
|
# PIGWay Raspberry Pi Agent v3.11.2
|
||||||
|
|
||||||
[简体中文](README.zh-CN.md) | English
|
[简体中文](README.zh-CN.md) | English
|
||||||
|
|
||||||
|
Disk discovery is manual: Device Console → System configuration → select the device → Storage settings → Scan disks. The inventory persists across restarts; known filesystem capacity updates weekly. Names and visibility belong to the Agent. See [deployment and validation](STORAGE-DISCOVERY.md).
|
||||||
|
|
||||||
|
|
||||||
PIGWay Raspberry Pi Agent is a lightweight local monitoring service for Raspberry Pi. It monitors CPU, memory, temperature, disk, networking, services, processes, system health alerts, and journal events. Ethernet monitoring reads the negotiated link speed and can alert when it remains below the configured minimum. It has no embedded Web UI and does not require a hardware plugin or Device Console.
|
PIGWay Raspberry Pi Agent is a lightweight local monitoring service for Raspberry Pi. It monitors CPU, memory, temperature, disk, networking, services, processes, system health alerts, and journal events. Ethernet monitoring reads the negotiated link speed and can alert when it remains below the configured minimum. It has no embedded Web UI and does not require a hardware plugin or Device Console.
|
||||||
|
|
||||||
@@ -69,3 +71,7 @@ API 状态新增 system.observed_at、network.observed_at 和 collection.interva
|
|||||||
## v3.10.0 独立采集周期
|
## v3.10.0 独立采集周期
|
||||||
|
|
||||||
实时状态提供分组采样时间与下一次更新计划。容量每天读取一次,RAM 缓存跨服务重启恢复;读取失败延后一小时重试,避免反复查询存储。网络周期可独立配置,生产计划调整为 1 秒。
|
实时状态提供分组采样时间与下一次更新计划。容量每天读取一次,RAM 缓存跨服务重启恢复;读取失败延后一小时重试,避免反复查询存储。网络周期可独立配置,生产计划调整为 1 秒。
|
||||||
|
|
||||||
|
## v3.11.2 容量周期
|
||||||
|
|
||||||
|
当前已知磁盘容量及根盘兼容指标每周更新;新增磁盘只手动发现。容量缓存保留在/run并支持服务重启复用,整机重启后首次初始化容量缓存属于启动检查;不自动调用lsblk发现新设备。当前策略覆盖历史版本的30秒/每日容量刷新描述。
|
||||||
|
|||||||
+3
-1
@@ -1,7 +1,9 @@
|
|||||||
# PIGWay Raspberry Pi Agent v3.7.1
|
# PIGWay Raspberry Pi Agent v3.11.2
|
||||||
|
|
||||||
简体中文 | [English](README.md)
|
简体中文 | [English](README.md)
|
||||||
|
|
||||||
|
新增磁盘由 Console → 系统配置 → 选择设备 → 磁盘设置 → 扫描磁盘手动发现;已保存清单跨重启保留,已知磁盘容量每周刷新。支持在 Agent 保存名称和隐藏设置。见 [部署与验证](STORAGE-DISCOVERY.md)。
|
||||||
|
|
||||||
|
|
||||||
极轻的树莓派本地监控服务:CPU、内存、温度、磁盘、网络、服务/进程、系统健康告警与 journal 日志。有线网络会读取真实协商速率,并可在持续低于配置阈值时告警。无 Web 界面;无需硬件插件,也无需管理中心。
|
极轻的树莓派本地监控服务:CPU、内存、温度、磁盘、网络、服务/进程、系统健康告警与 journal 日志。有线网络会读取真实协商速率,并可在持续低于配置阈值时告警。无 Web 界面;无需硬件插件,也无需管理中心。
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# Raspberry Pi Agent v3.11.2
|
||||||
|
|
||||||
|
新增磁盘只由用户手动扫描登记,已知磁盘容量及根分区兼容指标每周更新;服务重启复用运行缓存。保留现有CPU、内存、网络、硬件插件及监控协议。
|
||||||
|
|
||||||
|
磁盘清单持久化,支持名称/显示设置;整机重启后容量缓存首次初始化不自动发现新盘。没有调整风扇、RGB或OLED控制协议。
|
||||||
|
|
||||||
|
验证:10项后端测试通过;pigway-gateway生产升级后原两个服务正常,容量周期为604800秒,配置和令牌校验和保留。未执行整机重启或七天自然周期测试。
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# 磁盘手动扫描(2026-10-15)
|
||||||
|
|
||||||
|
后续显示层已升级为 Console v2.7.0 / Ubuntu 插件 v1.5.0 / Raspberry Pi 插件 v1.6.0 / Synology 插件 v1.9.0:默认原样使用设备返回顺序,取消 NAS 特殊倒序和优先级。显示名称、同组排序、隐藏统一保存到 Console,首页与配置共用,支持恢复默认顺序。设备已有名称/隐藏定义保留为默认上报值;本篇后文“名称保存在 Agent”等描述为当时的设备定义行为,不代表最新 Console 显示偏好的存储位置。最新验收见相邻 pigway-web-manager/docs/STORAGE-PRESENTATION-v2.7.0-2026-10-15.md;所有实际扫描行为保持手动。
|
||||||
|
|
||||||
|
## 已上线状态
|
||||||
|
|
||||||
|
Ubuntu Agent v1.3.1 / Console 插件 v1.4.1,Raspberry Pi Agent v3.11.1 / 插件 v1.5.1,Synology 插件 v1.8.1 已上线。两台 Agent 各识别两个实际分区,监控/API 服务 active,循环错误 0。
|
||||||
|
|
||||||
|
三种设备在线;中英文及 1440/820/390 三种屏宽的生产界面通过。设备名称、顺序和隐藏设置保留。Console 重启、页面加载及浏览器验收前后,NAS storage_observed_at 未改变,本次没有额外扫描 NAS 硬盘。尚未插入新物理磁盘,也未进行休眠/唤醒实测。
|
||||||
|
|
||||||
|
用户最终确认“三种设备都手动发现新增磁盘”。此前自动发现与 NAS 每 5 分钟扫描的候选方案已替换。
|
||||||
|
|
||||||
|
## 使用
|
||||||
|
|
||||||
|
Ubuntu、树莓派:Console → 系统配置 → 选择设备 → 磁盘设置 → 扫描磁盘。扫描后重新打开该页查看新磁盘,概览随正常轮询更新。名称和隐藏设置保存到 Agent,隐藏不停止历史采集。
|
||||||
|
|
||||||
|
NAS:与其他设备一样在配置 → 磁盘设置中点击“扫描磁盘”,首页只展示数据。磁盘清单、容量、温度、健康状态仅在手动扫描时更新;页面保留采集时间。CPU、内存和网络等快速采集继续工作。已有卷别名仍由 Console 保存。
|
||||||
|
|
||||||
|
v1.4.1 / v1.5.1 / v1.8.1 统一配置页扫描入口,保存按钮上方保留 20px 间距,扫描与保存反馈分别显示。按用户最终要求,NAS 每个机箱内硬盘采用 DSM 返回顺序反转;存储空间同样反转,主空间和 SSD 空间优先。生产三种屏宽、中英文已验证入口、间距、倒序及优先级;NAS 存储采集时间未改变。前两次发布因验收脚本固定计入网络卡片位置而自动回退,修正为直接验证存储卡片后通过。
|
||||||
|
|
||||||
|
扫描有 60 秒冷却,NAS 在途请求合并。页面加载、普通刷新、定时轮询和重启不会重新枚举磁盘。部署时执行过一次明确的初始扫描以保存现有 Agent 磁盘清单。
|
||||||
|
|
||||||
|
## 实现与边界
|
||||||
|
|
||||||
|
认证 PUT /api/v1/storage/scan 调用 lsblk,读取宿主机挂载点,将块设备清单保存到对应配置旁的 .storage.json(0600)。监控服务只读取清单;已知磁盘挂载和容量每 30 秒更新,新增磁盘必须手动扫描。根分区原有指标和告警保留兼容。
|
||||||
|
|
||||||
|
UUID 优先作为标识,其次为设备序列号和分区编号,再退回路径并标记 stable_identity=false。展示可用叶设备,去除重复挂载及 loop/swap 等伪设备。未挂载设备不伪造使用率。配置过的条目在后续扫描未发现时保留为断开;扫描失败保留原清单。
|
||||||
|
|
||||||
|
不执行挂载、格式化、分区或 SMART 指令。本次没有增加每磁盘断开/容量告警。
|
||||||
|
|
||||||
|
Agent 手工配置使用 status.system.storage_inventory[].id 的 32 位十六进制 key:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[storage_names]
|
||||||
|
0123456789abcdef0123456789abcdef = 媒体盘
|
||||||
|
[storage_visible]
|
||||||
|
0123456789abcdef0123456789abcdef = true
|
||||||
|
```
|
||||||
|
|
||||||
|
NAS 普通 collect 不再调度存储任务,只有 device.refresh(group=storage) 触发慢查询。重启复用上次摘要;过旧健康数据标为未确认,不确认旧告警恢复。只展示 DSM hdd_info / vol_info 上报的硬盘、扩展柜和卷,未验证任意 USB 外接盘是否均由接口提供。
|
||||||
|
|
||||||
|
## 验证和备份
|
||||||
|
|
||||||
|
两种 Agent 覆盖挂载点转义、bind 去重、伪设备、未挂载状态、UUID 稳定性、百分号名称、隐藏设置、手动发现和重启后禁止自动枚举。两种插件的中英文/三屏宽测试覆盖 0% 使用率、名称转义、隐藏恢复和显式扫描;打开设置不会扫描。NAS 15 项 Python 回归及六种浏览器布局通过。
|
||||||
|
|
||||||
|
生产备份:
|
||||||
|
|
||||||
|
- Ubuntu:/var/backups/pigway-ubuntu-agent-storage-1792043168
|
||||||
|
- Raspberry Pi:/var/backups/pigway-pi-control-storage-1792043209
|
||||||
|
- Console 容器:/data/storage-release-backups/1792043974(旧插件与 SQLite 备份)
|
||||||
|
|
||||||
|
ops/deploy-storage.py 检查 SHA-256 清单、备份三个程序文件与配置,仅重启两个项目服务;验证失败恢复原文件。Console 更新验证三个插件版本、全部设备在线、显示设置保留、NAS 未额外扫描,并执行生产浏览器检查。
|
||||||
@@ -1,12 +1,12 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
import time,socket,subprocess,configparser,signal,re,argparse,json,os,platform
|
import time,socket,subprocess,configparser,signal,re,argparse,json,os,platform
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
APP_VERSION="3.10.0"
|
APP_VERSION="3.11.2"
|
||||||
import sys
|
import sys
|
||||||
sys.path.append("/usr/local/share/pigway-pi-control")
|
sys.path.append("/usr/local/share/pigway-pi-control")
|
||||||
from plugin_api import Publisher,registered,discover
|
from plugin_api import Publisher,registered,discover
|
||||||
from network_rates import NetworkRates
|
from network_rates import NetworkRates
|
||||||
from storage_cache import StorageCache
|
from storage_cache import StorageCache,StorageInventory
|
||||||
storage_cache=StorageCache("/run/pigway-pi-control/storage-cache.json")
|
storage_cache=StorageCache("/run/pigway-pi-control/storage-cache.json")
|
||||||
|
|
||||||
CFG="/etc/pigway-pi-control.conf"
|
CFG="/etc/pigway-pi-control.conf"
|
||||||
@@ -199,6 +199,7 @@ def load_cfg():
|
|||||||
if target and not target.startswith("#"): watches.append(("process",label.upper(),target))
|
if target and not target.startswith("#"): watches.append(("process",label.upper(),target))
|
||||||
return c,watches
|
return c,watches
|
||||||
cfg,watches=load_cfg()
|
cfg,watches=load_cfg()
|
||||||
|
storage_inventory=StorageInventory(cache_path=Path(CFG).with_suffix(".storage.json"),capacity_path="/run/pigway-pi-control/storage-inventory-usage.json")
|
||||||
|
|
||||||
def cf(section,key,default,cast=float):
|
def cf(section,key,default,cast=float):
|
||||||
try: return cast(cfg.get(section,key))
|
try: return cast(cfg.get(section,key))
|
||||||
@@ -427,15 +428,17 @@ def write_status(cpu,temp,mem,disk):
|
|||||||
display_state=alert_manager.status_snapshot()
|
display_state=alert_manager.status_snapshot()
|
||||||
memory_total,memory_used,memory_available,memory_calculated=memory_usage()
|
memory_total,memory_used,memory_available,memory_calculated=memory_usage()
|
||||||
storage=storage_cache.sample()
|
storage=storage_cache.sample()
|
||||||
|
disks=storage_inventory.sample(cfg)
|
||||||
disk_total,disk_used,disk_free=(storage[k] for k in ("total","used","free")) if storage else (None,None,None)
|
disk_total,disk_used,disk_free=(storage[k] for k in ("total","used","free")) if storage else (None,None,None)
|
||||||
signal=wifi_percent(net_metric) if net_kind=="WIF" else None
|
signal=wifi_percent(net_metric) if net_kind=="WIF" else None
|
||||||
state={
|
state={
|
||||||
"version":APP_VERSION,"timestamp":int(time.time()),
|
"version":APP_VERSION,"timestamp":int(time.time()),
|
||||||
"collection":{"interval_seconds":min(1.0,SENSOR_INTERVAL,NETWORK_INTERVAL,SERVICE_INTERVAL,HEALTH_INTERVAL),
|
"collection":{"interval_seconds":min(1.0,SENSOR_INTERVAL,NETWORK_INTERVAL,SERVICE_INTERVAL,HEALTH_INTERVAL),
|
||||||
"refresh":{
|
"refresh":{
|
||||||
|
"storage_inventory":{"interval_seconds":604800,"observed_at":storage_inventory.observed_at,"next_update_at":storage_inventory.next_update_at,"error":storage_inventory.error},
|
||||||
"system":{"interval_seconds":SENSOR_INTERVAL,"observed_at":sensor_observed_at,"next_update_at":sensor_observed_at+SENSOR_INTERVAL if sensor_observed_at else None},
|
"system":{"interval_seconds":SENSOR_INTERVAL,"observed_at":sensor_observed_at,"next_update_at":sensor_observed_at+SENSOR_INTERVAL if sensor_observed_at else None},
|
||||||
"network":{"interval_seconds":NETWORK_INTERVAL,"observed_at":network_observed_at,"next_update_at":network_observed_at+NETWORK_INTERVAL if network_observed_at else None},
|
"network":{"interval_seconds":NETWORK_INTERVAL,"observed_at":network_observed_at,"next_update_at":network_observed_at+NETWORK_INTERVAL if network_observed_at else None},
|
||||||
"storage":{"interval_seconds":86400,"observed_at":storage_cache.observed_at,"next_update_at":storage_cache.next_update_at,"error":storage_cache.error},
|
"storage":{"interval_seconds":604800,"observed_at":storage_cache.observed_at,"next_update_at":storage_cache.next_update_at,"error":storage_cache.error},
|
||||||
"display":{"interval_seconds":1,"next_update_at":time.time()+1},
|
"display":{"interval_seconds":1,"next_update_at":time.time()+1},
|
||||||
"alerts":{"interval_seconds":1,"next_update_at":time.time()+1}}},"uptime_seconds":uptime_seconds(),
|
"alerts":{"interval_seconds":1,"next_update_at":time.time()+1}}},"uptime_seconds":uptime_seconds(),
|
||||||
"machine":{"id":DEVICE_ID,"name":DEVICE_NAME,"hostname":socket.gethostname()},
|
"machine":{"id":DEVICE_ID,"name":DEVICE_NAME,"hostname":socket.gethostname()},
|
||||||
@@ -446,6 +449,9 @@ def write_status(cpu,temp,mem,disk):
|
|||||||
"memory_total_bytes":memory_total,"memory_used_bytes":memory_used,
|
"memory_total_bytes":memory_total,"memory_used_bytes":memory_used,
|
||||||
"memory_available_bytes":memory_available,
|
"memory_available_bytes":memory_available,
|
||||||
"storage_observed_at":storage_cache.observed_at,
|
"storage_observed_at":storage_cache.observed_at,
|
||||||
|
"storage_inventory":disks,"storage_inventory_observed_at":storage_inventory.observed_at,
|
||||||
|
"storage_inventory_error":storage_inventory.error,
|
||||||
|
"storage_scanned_at":storage_inventory.scanned_at,"storage_discovery_mode":"manual",
|
||||||
"disk_percent":round(disk,1) if disk is not None else None,"disk_total_bytes":disk_total,
|
"disk_percent":round(disk,1) if disk is not None else None,"disk_total_bytes":disk_total,
|
||||||
"disk_used_bytes":disk_used,"disk_free_bytes":disk_free,
|
"disk_used_bytes":disk_used,"disk_free_bytes":disk_free,
|
||||||
"root_readonly":bool(fs_readonly)},
|
"root_readonly":bool(fs_readonly)},
|
||||||
|
|||||||
@@ -7,14 +7,15 @@ from concurrent.futures import ThreadPoolExecutor,as_completed
|
|||||||
from http import HTTPStatus
|
from http import HTTPStatus
|
||||||
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
|
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
STORAGE_SCAN_LOCK=threading.Lock()
|
||||||
from urllib.parse import parse_qs,urlencode,urlparse
|
from urllib.parse import parse_qs,urlencode,urlparse
|
||||||
from urllib.request import Request,urlopen
|
from urllib.request import Request,urlopen
|
||||||
|
|
||||||
APP_VERSION="3.7.2"
|
APP_VERSION="3.11.2"
|
||||||
CFG=Path("/etc/pigway-pi-control.conf")
|
CFG=Path("/etc/pigway-pi-control.conf")
|
||||||
STATUS=Path("/run/pigway-pi-control/status.json")
|
STATUS=Path("/run/pigway-pi-control/status.json")
|
||||||
TOKEN=Path("/etc/pigway-pi-control-api.token")
|
TOKEN=Path("/etc/pigway-pi-control-api.token")
|
||||||
ALLOWED_SECTIONS={"device","services","processes","alerts","timing","dark_mode","api"}
|
ALLOWED_SECTIONS={"device","services","processes","alerts","timing","dark_mode","api","storage_names","storage_visible"}
|
||||||
NAME_RE=re.compile(r"^[A-Za-z0-9_.-]{1,64}$")
|
NAME_RE=re.compile(r"^[A-Za-z0-9_.-]{1,64}$")
|
||||||
UNIT_RE=re.compile(r"^[A-Za-z0-9_.@:-]{1,128}\.service$")
|
UNIT_RE=re.compile(r"^[A-Za-z0-9_.@:-]{1,128}\.service$")
|
||||||
RGB_KEY_RE=re.compile(r"^(cpu|power|memory|storage|network|service|normal)_[rgb]$")
|
RGB_KEY_RE=re.compile(r"^(cpu|power|memory|storage|network|service|normal)_[rgb]$")
|
||||||
@@ -74,9 +75,16 @@ def api_setting(key,default,cast=str):
|
|||||||
|
|
||||||
def config_json():
|
def config_json():
|
||||||
cfg=load_cfg()
|
cfg=load_cfg()
|
||||||
return {section:{k:v for k,v in cfg.items(section) if not (section=="timing" and k in {"oled_refresh_interval","page_interval"})} for section in cfg.sections() if section in ALLOWED_SECTIONS}
|
return {section:{k:v for k,v in cfg.items(section,raw=True) if not (section=="timing" and k in {"oled_refresh_interval","page_interval"})} for section in cfg.sections() if section in ALLOWED_SECTIONS}
|
||||||
|
|
||||||
def validate_value(section,key,value,existing):
|
def validate_value(section,key,value,existing):
|
||||||
|
if section in {"storage_names","storage_visible"}:
|
||||||
|
if not re.fullmatch(r"[a-f0-9]{32}",key):raise ValueError("invalid storage identifier")
|
||||||
|
if section=="storage_visible":
|
||||||
|
if value.lower() not in {"true","false"}:raise ValueError("visibility must be true or false")
|
||||||
|
elif len(value)>64 or any(ord(c)<32 for c in value):
|
||||||
|
raise ValueError("storage name must be at most 64 characters without control characters")
|
||||||
|
return
|
||||||
if section=="api" and key=="enabled":
|
if section=="api" and key=="enabled":
|
||||||
if value.lower() not in {"true","false"}:raise ValueError("enabled must be true or false")
|
if value.lower() not in {"true","false"}:raise ValueError("enabled must be true or false")
|
||||||
return
|
return
|
||||||
@@ -388,6 +396,16 @@ class Handler(BaseHTTPRequestHandler):
|
|||||||
|
|
||||||
def do_PUT(self):
|
def do_PUT(self):
|
||||||
path=urlparse(self.path).path
|
path=urlparse(self.path).path
|
||||||
|
if path=="/api/v1/storage/scan":
|
||||||
|
if not self.authorized():self.send_json(401,{"error":"bearer token required"});return
|
||||||
|
try:
|
||||||
|
from storage_cache import StorageInventory
|
||||||
|
with STORAGE_SCAN_LOCK:
|
||||||
|
result=StorageInventory(cache_path=CFG.with_suffix(".storage.json")).scan()
|
||||||
|
self.send_json(200,result)
|
||||||
|
except Exception:
|
||||||
|
self.send_json(503,{"error":"storage scan failed; previous inventory retained"})
|
||||||
|
return
|
||||||
if path not in {"/api/v1/plugins","/api/v1/plugin/config","/api/v1/config","/api/v1/services","/api/v1/services/control","/api/v1/processes"}: self.send_json(404,{"error":"not found"}); return
|
if path not in {"/api/v1/plugins","/api/v1/plugin/config","/api/v1/config","/api/v1/services","/api/v1/services/control","/api/v1/processes"}: self.send_json(404,{"error":"not found"}); return
|
||||||
if not self.authorized(): self.send_json(401,{"error":"bearer token required"}); return
|
if not self.authorized(): self.send_json(401,{"error":"bearer token required"}); return
|
||||||
try:
|
try:
|
||||||
|
|||||||
+155
-2
@@ -4,6 +4,159 @@ import time
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from shutil import disk_usage
|
from shutil import disk_usage
|
||||||
|
|
||||||
|
class StorageInventory:
|
||||||
|
"""Read host block devices without mounting, probing SMART, or writing disks."""
|
||||||
|
|
||||||
|
def __init__(self, clock=time.time, runner=None, mounts_path="/proc/1/mountinfo", usage=None, cache_path=None, capacity_path=None):
|
||||||
|
import subprocess
|
||||||
|
self.clock = clock
|
||||||
|
self.runner = runner or (lambda: subprocess.check_output(
|
||||||
|
["lsblk", "--json", "--bytes", "--paths", "--output",
|
||||||
|
"NAME,KNAME,TYPE,SIZE,FSTYPE,UUID,LABEL,MODEL,SERIAL,MAJ:MIN"],
|
||||||
|
text=True, timeout=5, stderr=subprocess.DEVNULL))
|
||||||
|
self.mounts_path = Path(mounts_path)
|
||||||
|
self.usage = usage or (lambda mount: disk_usage("/proc/1/root" + mount))
|
||||||
|
self.next_update_at = 0
|
||||||
|
self.observed_at = None
|
||||||
|
self.error = False
|
||||||
|
self.devices = []
|
||||||
|
self.cache_path = Path(cache_path) if cache_path else None
|
||||||
|
self.scanned_at = None
|
||||||
|
self.capacity_path = Path(capacity_path) if capacity_path else None
|
||||||
|
if self.capacity_path:
|
||||||
|
try:
|
||||||
|
saved=json.loads(self.capacity_path.read_text())
|
||||||
|
at=saved['observed_at']
|
||||||
|
if isinstance(at,(int,float)) and 0 <= self.clock()-at < 604800 and isinstance(saved['devices'],list):
|
||||||
|
self.devices=saved['devices'];self.observed_at=at
|
||||||
|
self.scanned_at=saved['scanned_at'];self.next_update_at=at+604800
|
||||||
|
except (OSError,ValueError,KeyError,TypeError):pass
|
||||||
|
|
||||||
|
def scan(self):
|
||||||
|
if self.cache_path is None:
|
||||||
|
raise ValueError("storage inventory cache path required")
|
||||||
|
try:
|
||||||
|
previous = json.loads(self.cache_path.read_text())
|
||||||
|
if 0 <= self.clock() - previous["scanned_at"] < 60:
|
||||||
|
return {"accepted": True, "scanned_at": previous["scanned_at"], "cooldown": True}
|
||||||
|
except (OSError, ValueError, KeyError, TypeError):
|
||||||
|
pass
|
||||||
|
data = json.loads(self.runner())
|
||||||
|
self.discover(data)
|
||||||
|
document = {"scanned_at": self.clock(), "inventory": data}
|
||||||
|
temporary = self.cache_path.with_suffix(".tmp")
|
||||||
|
with open(temporary,"w",opener=lambda path, flags: __import__("os").open(path, flags, 0o600)) as file:
|
||||||
|
json.dump(document, file)
|
||||||
|
temporary.replace(self.cache_path)
|
||||||
|
return {"accepted": True, "scanned_at": document["scanned_at"], "cooldown": False}
|
||||||
|
|
||||||
|
def sample(self, config=None):
|
||||||
|
data = None
|
||||||
|
changed = False
|
||||||
|
if self.cache_path is not None:
|
||||||
|
try:
|
||||||
|
document = json.loads(self.cache_path.read_text())
|
||||||
|
changed = document["scanned_at"] != self.scanned_at
|
||||||
|
data = document["inventory"]
|
||||||
|
self.scanned_at = document["scanned_at"]
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
except (OSError, ValueError, KeyError, TypeError):
|
||||||
|
self.error = True
|
||||||
|
if data is not None and (changed or self.clock() >= self.next_update_at):
|
||||||
|
try:
|
||||||
|
self.devices = self.discover(data)
|
||||||
|
self.observed_at = self.clock()
|
||||||
|
self.error = False
|
||||||
|
except (OSError, ValueError, TypeError, KeyError, __import__("subprocess").SubprocessError):
|
||||||
|
self.error = True
|
||||||
|
self.next_update_at = self.clock() + (3600 if self.error else 604800)
|
||||||
|
if self.capacity_path and not self.error:
|
||||||
|
try:
|
||||||
|
self.capacity_path.write_text(json.dumps(dict(devices=self.devices,observed_at=self.observed_at,scanned_at=self.scanned_at)))
|
||||||
|
except OSError:pass
|
||||||
|
names = dict(config.items("storage_names", raw=True)) if config is not None and config.has_section("storage_names") else {}
|
||||||
|
visible = dict(config.items("storage_visible", raw=True)) if config is not None and config.has_section("storage_visible") else {}
|
||||||
|
result = []
|
||||||
|
for device in self.devices:
|
||||||
|
item = dict(device)
|
||||||
|
item["name"] = names.get(item["id"], "auto")
|
||||||
|
if item["name"].lower() == "auto":
|
||||||
|
item["name"] = item["label"] or item["mountpoint"] or item["device"]
|
||||||
|
item["visible"] = visible.get(item["id"], "true").lower() != "false"
|
||||||
|
if self.error:
|
||||||
|
item["state"] = "unknown"
|
||||||
|
result.append(item)
|
||||||
|
known = {d["id"] for d in result}
|
||||||
|
for identifier in sorted((names.keys() | visible.keys()) - known):
|
||||||
|
result.append(dict(id=identifier, name=names.get(identifier, identifier),
|
||||||
|
visible=visible.get(identifier, "true").lower() != "false",
|
||||||
|
state="unknown" if self.error else "missing", device="", mountpoint="",
|
||||||
|
label="", capacity_bytes=None, used_bytes=None, free_bytes=None,
|
||||||
|
used_percent=None))
|
||||||
|
return result
|
||||||
|
|
||||||
|
def discover(self, data):
|
||||||
|
import hashlib
|
||||||
|
import re
|
||||||
|
mounts, sources = {}, {}
|
||||||
|
for line in self.mounts_path.read_text().splitlines():
|
||||||
|
left, right = line.split(" - ", 1)
|
||||||
|
fields = left.split()
|
||||||
|
# Bind mounts and Btrfs subvolumes must not duplicate filesystem capacity.
|
||||||
|
mount = re.sub(r"\\([0-7]{3})", lambda m: chr(int(m[1], 8)), fields[4])
|
||||||
|
row = (fields[3] != "/", len(mount), mount)
|
||||||
|
mounts.setdefault(fields[2], []).append(row)
|
||||||
|
source = right.split()[1]
|
||||||
|
if source.startswith("/dev/"):
|
||||||
|
sources.setdefault(str(Path(source).resolve()), []).append(row)
|
||||||
|
devices = data["blockdevices"]
|
||||||
|
if not isinstance(devices, list):
|
||||||
|
raise ValueError("invalid block device inventory")
|
||||||
|
result, seen = [], set()
|
||||||
|
|
||||||
|
def walk(device, serial="", parent=""):
|
||||||
|
kind = device.get("type")
|
||||||
|
if kind in {"loop", "rom", "zram"}:
|
||||||
|
return
|
||||||
|
serial = str(device.get("serial") or serial).strip()
|
||||||
|
children = device.get("children") or []
|
||||||
|
uuid = str(device.get("uuid") or "").strip()
|
||||||
|
fs = str(device.get("fstype") or "")
|
||||||
|
# A parent disk with partitions/RAID/LVM is represented by its usable leaves.
|
||||||
|
if not children and kind not in {"loop", "rom"} and fs not in {"swap", "linux_raid_member", "LVM2_member", "crypto_LUKS"}:
|
||||||
|
partition = re.search(r"(\d+)$", str(device.get("kname") or device.get("name") or ""))
|
||||||
|
suffix = ":" + partition[1] if kind == "part" and partition else ""
|
||||||
|
source = "uuid:" + uuid if uuid else "serial:" + serial + suffix if serial else "device:" + str(device.get("name"))
|
||||||
|
identifier = hashlib.sha256(source.encode()).hexdigest()[:32]
|
||||||
|
if identifier not in seen:
|
||||||
|
seen.add(identifier)
|
||||||
|
candidates = mounts.get(device.get("maj:min"), []) or sources.get(str(Path(device["name"]).resolve()), [])
|
||||||
|
mount = sorted(candidates)[0][2] if candidates else ""
|
||||||
|
item = dict(id=identifier, device=str(device.get("name") or ""),
|
||||||
|
label=str(device.get("label") or ""), uuid=uuid,
|
||||||
|
stable_identity=bool(uuid or serial), model=str(device.get("model") or "").strip(),
|
||||||
|
mountpoint=mount, filesystem=fs, state="mounted" if mount else "unmounted",
|
||||||
|
capacity_bytes=int(device.get("size") or 0), used_bytes=None,
|
||||||
|
free_bytes=None, used_percent=None)
|
||||||
|
if mount:
|
||||||
|
try:
|
||||||
|
usage = self.usage(mount)
|
||||||
|
if usage.total <= 0:
|
||||||
|
raise ValueError("invalid filesystem capacity")
|
||||||
|
item.update(capacity_bytes=usage.total, used_bytes=usage.used,
|
||||||
|
free_bytes=usage.free, used_percent=round(100 * usage.used / usage.total, 1))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
item["state"] = "unknown"
|
||||||
|
result.append(item)
|
||||||
|
for child in children:
|
||||||
|
walk(child, serial, str(device.get("name") or parent))
|
||||||
|
|
||||||
|
for device in devices:
|
||||||
|
walk(device)
|
||||||
|
return sorted(result, key=lambda d: (d["mountpoint"] != "/", d["mountpoint"], d["device"]))
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class StorageCache:
|
class StorageCache:
|
||||||
def __init__(self, path, reader=disk_usage, clock=time.time):
|
def __init__(self, path, reader=disk_usage, clock=time.time):
|
||||||
@@ -23,7 +176,7 @@ class StorageCache:
|
|||||||
at = data['observed_at']
|
at = data['observed_at']
|
||||||
if isinstance(at, (int, float)) and 0 <= self.clock() - at < 90000:
|
if isinstance(at, (int, float)) and 0 <= self.clock() - at < 90000:
|
||||||
self.values, self.observed_at = values, at
|
self.values, self.observed_at = values, at
|
||||||
if not self.error:self.next_update_at = at + 86400
|
if not self.error:self.next_update_at = at + 604800
|
||||||
except (OSError, ValueError, KeyError, TypeError):
|
except (OSError, ValueError, KeyError, TypeError):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@@ -42,7 +195,7 @@ class StorageCache:
|
|||||||
return self.values
|
return self.values
|
||||||
self.values = dict(total=usage.total, used=usage.used, free=usage.free)
|
self.values = dict(total=usage.total, used=usage.used, free=usage.free)
|
||||||
self.observed_at = now
|
self.observed_at = now
|
||||||
self.next_update_at = now + 86400
|
self.next_update_at = now + 604800
|
||||||
self.error = False
|
self.error = False
|
||||||
self.persist()
|
self.persist()
|
||||||
return self.values
|
return self.values
|
||||||
|
|||||||
@@ -183,3 +183,9 @@ SSH = ssh.service
|
|||||||
#
|
#
|
||||||
# 示例默认全部注释,不会监控任何普通进程。
|
# 示例默认全部注释,不会监控任何普通进程。
|
||||||
# MYAPP = /opt/myapp/server
|
# MYAPP = /opt/myapp/server
|
||||||
|
|
||||||
|
# Use status.system.storage_inventory IDs, or edit through Console.
|
||||||
|
# auto uses the filesystem label, mountpoint, or device path.
|
||||||
|
[storage_names]
|
||||||
|
[storage_visible]
|
||||||
|
# <32-character ID> = false hides a disk while retaining collection.
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ done
|
|||||||
[ "$(id -u)" -eq 0 ] || { echo 'ERROR: run installer as root' >&2; exit 1; }
|
[ "$(id -u)" -eq 0 ] || { echo 'ERROR: run installer as root' >&2; exit 1; }
|
||||||
# The monitor has no I2C, GPIO, Pillow, smbus2 or cooling-board requirement.
|
# The monitor has no I2C, GPIO, Pillow, smbus2 or cooling-board requirement.
|
||||||
command -v python3 >/dev/null || { apt-get update; apt-get install -y python3; }
|
command -v python3 >/dev/null || { apt-get update; apt-get install -y python3; }
|
||||||
|
command -v lsblk >/dev/null || { apt-get update; apt-get install -y util-linux; }
|
||||||
mkdir -p /usr/local/share/pigway-pi-control
|
mkdir -p /usr/local/share/pigway-pi-control
|
||||||
install -m 0644 app/storage_cache.py /usr/local/share/pigway-pi-control/storage_cache.py
|
install -m 0644 app/storage_cache.py /usr/local/share/pigway-pi-control/storage_cache.py
|
||||||
install -m 0644 app/network_rates.py /usr/local/share/pigway-pi-control/network_rates.py
|
install -m 0644 app/network_rates.py /usr/local/share/pigway-pi-control/network_rates.py
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ from storage_cache import StorageCache
|
|||||||
|
|
||||||
|
|
||||||
class StorageTests(unittest.TestCase):
|
class StorageTests(unittest.TestCase):
|
||||||
def test_daily_reads_and_restart_cache(self):
|
def test_weekly_reads_and_restart_cache(self):
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
now=[1000];reads=[]
|
now=[1000];reads=[]
|
||||||
def reader(path):
|
def reader(path):
|
||||||
@@ -18,7 +18,7 @@ class StorageTests(unittest.TestCase):
|
|||||||
self.assertEqual(len(reads),1)
|
self.assertEqual(len(reads),1)
|
||||||
restored=StorageCache(p,reader,lambda:now[0]);restored.sample()
|
restored=StorageCache(p,reader,lambda:now[0]);restored.sample()
|
||||||
self.assertEqual(len(reads),1)
|
self.assertEqual(len(reads),1)
|
||||||
now[0]=87400;restored.sample()
|
now[0]=605800;restored.sample()
|
||||||
self.assertEqual(len(reads),2)
|
self.assertEqual(len(reads),2)
|
||||||
|
|
||||||
def test_failure_does_not_fabricate_zero_or_retry_each_second(self):
|
def test_failure_does_not_fabricate_zero_or_retry_each_second(self):
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
import importlib.util
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT / "app"))
|
||||||
|
stem = "pigway_ubuntu_agent_api" if (ROOT / "app/pigway_ubuntu_agent_api.py").exists() else "pigway_pi_control_api"
|
||||||
|
spec = importlib.util.spec_from_file_location("storage_api_under_test", ROOT / "app" / (stem + ".py"))
|
||||||
|
api = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(api)
|
||||||
|
|
||||||
|
|
||||||
|
class StorageConfigTests(unittest.TestCase):
|
||||||
|
def test_config_round_trip_and_invalid_edits_are_atomic(self):
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
config = Path(directory) / "agent.conf"
|
||||||
|
config.write_text("[device]\nname = auto\n")
|
||||||
|
identifier = "a" * 32
|
||||||
|
with patch.object(api, "CFG", config):
|
||||||
|
api.update_ini({"storage_names": {identifier: "媒体盘 100%"}, "storage_visible": {identifier: "false"}})
|
||||||
|
self.assertEqual(api.config_json()["storage_names"][identifier], "媒体盘 100%")
|
||||||
|
self.assertEqual(api.config_json()["storage_visible"][identifier], "false")
|
||||||
|
original = config.read_text()
|
||||||
|
for updates in [
|
||||||
|
{"storage_visible": {identifier: "yes"}},
|
||||||
|
{"storage_names": {"invalid-id": "Bad"}},
|
||||||
|
{"storage_names": {identifier: "x" * 65}},
|
||||||
|
{"storage_names": {identifier: "bad\tname"}},
|
||||||
|
]:
|
||||||
|
with self.assertRaises(ValueError):
|
||||||
|
api.update_ini(updates)
|
||||||
|
self.assertEqual(config.read_text(), original)
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
import configparser
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from collections import namedtuple
|
||||||
|
from pathlib import Path
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "app"))
|
||||||
|
from storage_cache import StorageInventory
|
||||||
|
|
||||||
|
Usage = namedtuple("Usage", "total used free")
|
||||||
|
|
||||||
|
|
||||||
|
class StorageInventoryTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.tmp.cleanup)
|
||||||
|
self.mounts = Path(self.tmp.name) / "mountinfo"
|
||||||
|
self.mounts.write_text("1 0 8:1 / / rw - ext4 /dev/sda1 rw\n2 0 8:2 / /media/My\\040Disk rw - ext4 /dev/sda2 rw\n3 0 8:2 /folder /bind rw - ext4 /dev/sda2 rw\n")
|
||||||
|
self.data = {"blockdevices": [
|
||||||
|
{"name": "/dev/sda", "type": "disk", "serial": "serial", "children": [
|
||||||
|
{"name": "/dev/sda1", "type": "part", "uuid": "root", "maj:min": "8:1", "fstype": "ext4", "size": 1000},
|
||||||
|
{"name": "/dev/sda2", "type": "part", "uuid": "data", "maj:min": "8:2", "fstype": "ext4", "size": 2000},
|
||||||
|
]},
|
||||||
|
{"name": "/dev/sdb", "type": "disk", "size": 3000},
|
||||||
|
{"name": "/dev/loop0", "type": "loop", "size": 4000},
|
||||||
|
{"name": "/dev/zram0", "type": "disk", "fstype": "swap"},
|
||||||
|
]}
|
||||||
|
self.now = 100
|
||||||
|
self.reader = StorageInventory(clock=lambda: self.now, runner=lambda: json.dumps(self.data),
|
||||||
|
mounts_path=self.mounts, usage=lambda p: Usage(1000, 0, 1000),
|
||||||
|
cache_path=Path(self.tmp.name)/"inventory.json")
|
||||||
|
self.reader.scan()
|
||||||
|
|
||||||
|
def test_weekly_usage_cache_survives_service_restart(self):
|
||||||
|
capacity=Path(self.tmp.name)/'usage.json'
|
||||||
|
self.reader.capacity_path=capacity
|
||||||
|
self.reader.sample()
|
||||||
|
def forbidden(*args):raise AssertionError('Not due; should not read capacity or discover disks')
|
||||||
|
restarted=StorageInventory(clock=lambda:self.now+100,runner=forbidden,mounts_path=self.mounts,
|
||||||
|
usage=forbidden,cache_path=self.reader.cache_path,capacity_path=capacity)
|
||||||
|
self.assertEqual(len(restarted.sample()),3)
|
||||||
|
self.assertEqual(restarted.observed_at,self.now)
|
||||||
|
self.assertEqual(restarted.next_update_at,self.now+604800)
|
||||||
|
|
||||||
|
def test_mounts_raw_disk_and_pseudo_filters(self):
|
||||||
|
disks = self.reader.sample()
|
||||||
|
self.assertEqual(len(disks), 3)
|
||||||
|
self.assertEqual(disks[0]["mountpoint"], "/")
|
||||||
|
self.assertEqual(next(d for d in disks if d.get("uuid") == "data")["mountpoint"], "/media/My Disk")
|
||||||
|
self.assertEqual(disks[0]["used_percent"], 0)
|
||||||
|
raw = next(d for d in disks if d["device"] == "/dev/sdb")
|
||||||
|
self.assertEqual(raw["state"], "unmounted")
|
||||||
|
self.assertIsNone(raw["used_bytes"])
|
||||||
|
|
||||||
|
def test_uuid_survives_renumber_and_configuration(self):
|
||||||
|
disk = self.reader.sample()[0]
|
||||||
|
self.data["blockdevices"][0]["children"][0]["name"] = "/dev/sdz1"
|
||||||
|
self.now += 61
|
||||||
|
self.reader.scan()
|
||||||
|
config = configparser.ConfigParser()
|
||||||
|
config.read_string(f'[storage_names]\n{disk["id"]} = 数据盘 100%\n[storage_visible]\n{disk["id"]} = false\n')
|
||||||
|
result = next(d for d in self.reader.sample(config) if d["id"] == disk["id"])
|
||||||
|
self.assertEqual(result["name"], "数据盘 100%")
|
||||||
|
self.assertFalse(result["visible"])
|
||||||
|
|
||||||
|
def test_removal_and_discovery_error_are_distinct(self):
|
||||||
|
disk = self.reader.sample()[0]
|
||||||
|
config = configparser.ConfigParser()
|
||||||
|
config.read_dict({"storage_names": {disk["id"]: "System"}})
|
||||||
|
self.reader.runner = lambda: "invalid json"
|
||||||
|
self.now += 61
|
||||||
|
with self.assertRaises(ValueError):self.reader.scan()
|
||||||
|
self.assertEqual(self.reader.sample(config)[0]["state"], "mounted")
|
||||||
|
self.reader.runner = lambda: '{"blockdevices":[]}'
|
||||||
|
self.now += 61
|
||||||
|
self.reader.scan()
|
||||||
|
self.assertEqual(self.reader.sample(config)[0]["state"], "missing")
|
||||||
|
self.assertFalse(self.reader.error)
|
||||||
|
|
||||||
|
def test_hotplug_requires_manual_scan_and_inventory_survives_restart(self):
|
||||||
|
self.assertEqual(len(self.reader.sample()), 3)
|
||||||
|
self.data["blockdevices"].append({"name": "/dev/sdc", "type": "disk", "uuid": "new", "size": 9000})
|
||||||
|
self.now += 29
|
||||||
|
self.assertEqual(len(self.reader.sample()), 3)
|
||||||
|
self.now += 100000
|
||||||
|
self.assertEqual(len(self.reader.sample()), 3)
|
||||||
|
self.reader.scan()
|
||||||
|
self.assertEqual(len(self.reader.sample()), 4)
|
||||||
|
def forbidden():raise AssertionError("automatic enumeration is forbidden")
|
||||||
|
restored=StorageInventory(clock=lambda:self.now,runner=forbidden,mounts_path=self.mounts,
|
||||||
|
usage=lambda p:Usage(1000,0,1000),cache_path=self.reader.cache_path)
|
||||||
|
self.assertEqual(len(restored.sample()),4)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user