way/pigway-frame-flow

Fork 0

Files

T

Pigway f7e65de0a2 Make CI pip indexes explicit with safe diagnostics and sequential fallback

2026-10-07 20:51:05 +08:00

19 KiB

Raw Blame History

Pigway Frame Flow

Pigway Frame Flow is a lightweight self-hosted platform for programmable smart-frame automation.

独立运行的智能画框服务：中英文 Web UI、深浅色模式、设备驱动抽象、KoKonna 全部已公开接口、图片处理、模板、SQLite 调度与播放列表、临时覆盖恢复和 REST API。

KoKonna is a supported device integration and is not affiliated with this project unless otherwise stated.

Quick Start / 快速开始

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.lock
.venv/bin/python scripts/run_local.py

Open http://127.0.0.1:18787. The local demo creates a private .env with a random FRAME_FLOW_TOKEN. Read that token locally in an editor and enter it on the login screen. Do not paste a KoKonna API Key into Web UI. Demo always forces mock mode and disables real writes. It leaves data persistent in ./data and ./config.

网页使用账号密码登录。通过 python scripts/create-account.py --username YOUR_NAME 创建账号，再启动服务；密码不会写入代码或明文配置。勾选记住登录可保持 30 天，设置页面可修改密码或退出。

Architecture / 架构

providers → templates / renderers → generated image
                                  ↓
Web UI / external REST → DisplayService → FrameDeviceDriver → KokonnaDriver
Scheduler / Playlist ─────────────┘
                   SQLite: operations / history / jobs / overrides / assets

Package: pigway_frame_flow. Device composition is in core/registry.py; vendor-specific code is in drivers/kokonna. One process, one worker, no Redis, message broker, OpenClaw, Home Assistant or external application dependency.

Configuration / 配置

Copy .env.example to .env, set a strong FRAME_FLOW_TOKEN, chmod 600 .env.

KOKONNA_MOCK=true: independent persistent demo, no cloud calls.
KOKONNA_WRITE_ENABLED=false: live write protection, mandatory before verification.
KOKONNA_API_KEY: device key, never stored in DB or returned to Web UI.
*_FILE: optional Docker Secret equivalent.
WEB_BIND/WEB_PORT: Compose host bind and port. Internal app binds container network.
CORS_ALLOWED_ORIGINS: exact comma-separated origins; default empty.
IMAGE_URL_ALLOWED_HOSTS: exact public HTTPS hosts; default disabled. Redirects denied.
TZ=Asia/Shanghai; DATA_DIR=/data; CONFIG_DIR=/config.

For multiple devices copy deploy/devices.yml.example into config/devices.yml, reference a separate environment key per device. Jobs bind a device_id. Layout currently targets 1200 × 1600; incompatible geometry is refused before writes.

Docker

cp .env.example .env
# Configure a token; keep API key empty, mock true, writes false.
mkdir -p data/images data/generated data/cache config
sudo chown -R 10333:10333 data config
./scripts/build-verify.sh
docker compose up -d
python3 scripts/smoke-test.py

Image pigway-frame-flow:latest, service/container pigway-frame-flow, AMD64, Python 3.14.5, pinned runtime dependencies, CJK fonts, non-root UID/GID 10333, read-only root filesystem, writable /data and /config, no capabilities, no host network, no Docker socket. Restart unless-stopped. JSON-file logs capped 10m × 3. Healthcheck uses service /health, not cloud availability. No second log database.

Synology NAS

Target /volume2/docker/pigway-frame-flow/. See NAS deployment. Set WEB_BIND to the NAS LAN IP to restrict the published interface, or explicitly 0.0.0.0 when desired. Port 18787 must be free. No router/firewall/reverse proxy rules are modified. Source and persistent volumes stay in this project directory. No artificial logs directory is required. Compose creates its own bridge network.

KoKonna Setup / 真实设备

First deploy Mock and verify persistence. Then set KOKONNA_MOCK=false and fill KOKONNA_API_KEY through private environment/secret. Keep writes false. /ready checks configuration/DB/worker/token without a cloud call. Query device and images metadata; compare model, geometry, timezone, heartbeat, battery, imageId, coverId, sync. If different, stop and report. Do not enable writes or perform the single test until the owner explicitly authorizes it. Test design: REAL_DEVICE_TEST.md.

Security

Protected /api/v1 capabilities accept an administrator account session, or external Frame Flow Bearer auth with scope checks. SQLite stores token hashes. Device keys live only in env/secret and runtime memory. Browser login uses SQLite accounts with salted scrypt password hashes and revocable, HttpOnly/SameSite=Strict cookies (30 days when remembered, 12 hours otherwise). Mutations require a session CSRF header. Logout and password changes revoke sessions. Passwords and browser sessions are never stored in localStorage. Bootstrap hashes in private config/accounts.json seed new accounts only; restarting does not undo a password change. Five failed logins per source IP cause a five-minute lockout. The browser needs no API Token. Bearer authentication remains compatible for external clients; the KoKonna key is never used for account login. The LAN deployment uses HTTP; use an explicitly configured HTTPS endpoint if transport encryption is needed. No reverse proxy or firewall is changed automatically.

API

OpenAPI 3.x at /openapi.json; Swagger at /docs; stable /api/v1. Display/render/upload return 202 + operation_id. Poll /api/v1/operations/{id}; completion requires matching image ID and cloud synced flag. Idempotency-Key retained seven days. X-Request-ID traces history. UNKNOWN writes block later mutations; read-only reconciliation and explicit admin acknowledgement release the block, never replay a write. The older /api/display route is retained and deprecated (same v1 payload). See external integration contract.

Scheduler / Playback

Cron (5 fields), daily, weekdays, weekends, date and temporary tasks; Shanghai time. Use source {"type":"restore"} for explicit restore jobs. Unique job/slot and persistent idempotency prevent duplicate dispatch. Overdue tasks >5 minutes are skipped rather than unexpectedly overwriting artwork after downtime. State persists across restart. Overrides use priority then recency; lower priorities wait without uploading. If another app changes the frame, expired overrides do not overwrite its artwork. Frame Flow playlists support ordered/random cloud image IDs and >=5-minute intervals; only one enabled playlist per device. They pause for overrides/unknown writes. Native cloud slideshow mode/interval are readable; membership/editing are unavailable in the official API. Avoid having native and Frame Flow playback compete.

Templates / 图片

Six JSON layout templates: daily-dashboard, weather, calendar, quote, photo-text, system-alert. color and grayscale profiles. Photo data accepts photo_id = local asset ID. Mock weather is clearly labeled; no unreliable lunar date is fabricated. Contain preserves composition with margins; cover crops centrally; smart is an edge-saliency heuristic. Always inspect preview; face-aware guarantees are not claimed. Cloud uploads activate immediately, so gallery-only means local staging.

Development / Testing

.venv/bin/python -m pip install -e '.[test]'
.venv/bin/python -m pytest -q
.venv/bin/python -m ruff check pigway_frame_flow tests scripts
.venv/bin/python -m ruff format --check pigway_frame_flow tests scripts
.venv/bin/python -m mypy pigway_frame_flow
.venv/bin/python -m pip_audit -r requirements.lock --no-deps --disable-pip

Tests mock every cloud request; no real-device writes. scripts/ui-smoke.cjs runs an isolated headless browser and checks settings.mock before acting. CI builds AMD64, validates Compose, starts a container, checks health/ready and restarts for persistence.

Known Limitations / 已知边界

Cloud acknowledgements do not prove physical display pixels. Model fields may be omitted and are explicitly marked configured. No native slideshow membership API, no cloud upload-only API, no automatic retries after uncertain writes. Cache has no automatic quota/GC in 0.1.0; monitor data volume. Rendering assets are local, limited to 25M pixels/20 MiB. This release uses a single worker and is not multi-replica. Gallery reads have no pagination because upstream documents none. Web UI hides secrets and lazy-loads thumbnails; generating a first thumbnail fetches one original. Multi-client provisioning UI/webhook delivery are not implemented. Docker build/NAS status is recorded honestly in docs/ACCEPTANCE.md.

Roadmap

Additional drivers; scoped token administration; signed webhooks; cache retention; face-aware cropping; public weather provider; Google Calendar, Home Assistant, MQTT, NAS monitoring and AI integrations through the same API. External integration is optional.

Template editing

Templates provide bilingual form fields, event lists, local photo selection, advanced JSON, color/grayscale previews, persistent defaults and reset to built-in defaults. Changing content invalidates the previous preview. Weather uses the labeled Mock provider until a real provider is configured. Live display controls stay disabled until real writes are separately approved.

Browser smoke tests use a disposable Mock account via FRAME_FLOW_TEST_USERNAME and FRAME_FLOW_TEST_PASSWORD; never run the mutation smoke suite against live devices.

Current content and completed page workflows

Dashboard displays the actual current image and recorded content source. Frame Flow stores immutable template/data/profile snapshots when it renders an asset and maps successful upload results to the device/image ID. The current-content endpoint matches the live device image ID; vendor images with no recorded source remain explicitly unknown. Switching externally never makes the previous template appear as the current one. Restore also restores the matching content provenance.

Templates can load the current recorded template or continue editing a generated local image. They support direct photo import and protected image downloads. Gallery provides search, sorting, dimensions and downloads. Display confirmation supports duration, priority and restoration. History includes operation details. Automation offers content pickers, template profiles, previews and manual execution; playlists offer gallery image checkboxes. Real writes remain blocked by the server regardless of the UI controls.

Settings persists weather coordinates/provider and image URL host allowlists in SQLite. Open-Meteo provides current model weather and a three-day forecast, cached for ten minutes on demand. Network failures show unavailable/stale data and use a two-minute backoff, without replacing it with fictitious live data. Default remains Mock until configured in Settings. Open-Meteo attribution is included in the UI and weather/dashboard renders. Image URL import is HTTPS-only with explicit hostnames, public-address validation and pinned DNS; redirects and private addresses are blocked.

Additional v1 endpoints: /devices/{id}/current-content, /assets/{id}/source, /providers/weather, /settings/weather, /settings/image-url, /jobs/{id}, /jobs/{id}/preview, /jobs/{id}/run (all under /api/v1). Existing contracts remain compatible; new endpoints are included in generated OpenAPI and contract tests.

Daily Poetry / 每日诗词

The daily-poetry template includes 20 public-domain classical Chinese poems, selected deterministically by Shanghai date, or manually. Create a daily job from its editor; new poetry jobs are paused by default until you enable them. Custom title, author and verses are editable. This is an independent Frame Flow content provider, not the KoKonna mini-program subscription. Official public API does not expose mini-program subscription/template settings.

每日诗词支持按上海日期选诗、手选 20 首古典诗词、自定义文本以及每日任务。模板显示操作使用统一上传/显示服务；真实写入须开启 KOKONNA_WRITE_ENABLED，结果未知不会自动重试。官方小程序原有每日诗词订阅不能通过当前公开 API 查询或修改。

官方账号连接 / Official account connection

在 Templates → 官方小组件中，由管理员输入 KoKonna 官方账号和密码，查询账号下的设备后手动关联当前 Frame Flow 设备。此流程不绑定或解绑云端设备，也不改变图片或订阅。现有 API Key 接入继续负责设备、图库和显示能力。

密码不保存；会话与关联信息保存在 /config/official-{device_id}.secret，权限 0600。容器重启可恢复连接。会话过期需要重新登录；断开连接会清除本地会话，不撤销官方服务器上的其他登录。局域网访问沿用现有部署方式。

Official account connection is admin-only and separate from API Key access. Login, device discovery, local association and disconnect are available through /api/v1/devices/{device_id}/official/connection. Passwords are never persisted; server-side sessions are stored as restricted secret files. No official widget mutation or other extra RPC write is enabled by connecting.

Photo sources / 照片数据源

Gallery supports read-only NAS folders and HTTPS WebDAV sources, random photo preview, and device-bound scheduled random photos. Source definitions are persisted in SQLite. See setup and limits. NAS 本地目录需要显式只读挂载；网盘先支持通用 WebDAV，也可使用同步到 NAS 的目录。

Build networking and cache

The Dockerfile retains the pinned Python base, locked Python dependencies, fonts-noto-cjk, Debian repository signature checks, and offline artifact SHA256 verification. Both production and CI use scripts/build-verify.sh, which enables BuildKit and retains docker compose build --pull.

APT mirror hosts are configurable through DEBIAN_MIRROR and DEBIAN_SECURITY_MIRROR (hosts only, without scheme or path). Defaults are deb.debian.org; HTTPS is used for both Debian and security repositories. For mainland-China NAS builds, use the shared profile:

. ./deploy/build-cn.env
./scripts/build-verify.sh

CI loads this same profile, using Tsinghua TUNA for both repositories unless Gitea repository variables DEBIAN_MIRROR / DEBIAN_SECURITY_MIRROR override it. Production may also set the two hosts directly in .env when calling the build script without the profile. An alternate mainland mirror is mirrors.ustc.edu.cn; availability and throughput depend on the Runner network. No fallback disables signature verification or removes packages.

Layer order stays: pinned base → APT/font dependencies → requirements.lock → pip dependencies → application/vendor source → build metadata. A source-only change therefore reuses the APT and pip layers, provided the same builder, base digest, mirror arguments, and optional offline artifacts are used. Changing the lockfile rebuilds pip and later layers, not APT. Changing the mirror or APT instructions rebuilds APT and later layers.

Previously an APT layer miss redownloaded indexes/packages, and pip used --no-cache-dir. Now locked BuildKit mounts retain APT indexes, downloaded .deb files, and pip downloads/wheels outside the final image. Debian's docker-clean hook is disabled so it cannot discard cached packages. Re-executed APT steps still run apt-get update and validate repository signatures; cached packages and pip artifacts can avoid repeat large downloads. The initial build after this Dockerfile change will miss dependency layers; subsequent builds benefit. No measured speedup is claimed before CI completes.

Caches belong to the Docker builder, not the ephemeral CI workspace. Keep the NAS Runner's builder storage across jobs; removing the builder, pruning caches, or using a fresh Docker daemon loses the benefit. Cache mounts are disposable: an empty cache still builds correctly. --pull remains enabled; a pinned base digest check alone does not invalidate dependency layers. To deliberately refresh installed APT versions, use docker compose build --pull --no-cache.

CI with a host Docker socket

Run python scripts/ci-compose-verify.py after loading the build mirror profile. The script derives a temporary Compose document from production settings, with an absolute client-side build context, a UUID project name, a unique CI image tag, and project-scoped data / config named volumes. It removes production bind mounts, the fixed container name, host ports, and the production .env reference. A fresh test token, mock driver, and disabled device writes are explicitly configured. Docker initializes empty named volumes from the image's owned /data and /config directories; the service remains UID/GID 10333.

Build contexts are sent by the Docker client, so the job workspace can remain inside the job container. Runtime storage is managed by the daemon; no matching NAS host /workspace directory is needed. HTTP readiness and the unmodified nine-endpoint smoke script run inside the service with compose exec -T. Restart is followed by another readiness check and the same smoke test.

A finally block removes the test containers, network, volumes, and unique image tag on success or failure. The workflow records the private temporary Compose path in GITHUB_ENV and repeats cleanup in an always() step, covering normal cancellation/interruption. Failed cleanup retains the manifest so the fallback can retry. Abrupt host/daemon loss or a forcibly terminated entire job can prevent both cleanup paths; no CI script can guarantee cleanup in that case. No global prune or production resource removal is performed. Keep builder caches: only the run-specific image tag is removed.

CI Job pip indexes versus Docker build indexes

The Job bootstrap runs python scripts/ci-install.py before Ruff/mypy/tests. It needs only the Python standard library, so it runs before PyYAML or tooling is installed. CI_PIP_INDEX_URL and CI_PIP_FALLBACK_INDEX_URL are optional Gitea repository variables. Defaults are the TUNA HTTPS PyPI simple endpoint and official https://pypi.org/simple. Each is attempted separately; the fallback repeats the unchanged editable .[test] installation, including PEP 517 build isolation and setuptools>=75. No simultaneous extra index is used, no build dependency is removed, and both failures fail CI.

Logs show inherited index/config/proxy hosts and the selected index host for each attempt, with URL credentials and query strings removed. Bootstrap disables inherited pip config files via PIP_CONFIG_FILE=/dev/null, explicitly sets the selected index, and clears no-index/extra-index/find-links, insecure trusted-host, and no-build-isolation environment settings. Constraints, TLS certificate environment settings, and proxy settings remain in effect.

These CI_PIP_* settings apply only to the Job's pip install. They are not passed into Docker build arguments or persisted into the production image. The Dockerfile keeps its own cached/offline dependency installation and logs the Docker pip index host for online installs. Debian mirror variables affect APT only and are loaded later in the Docker verification step.

If dependency installation fails, the always() cleanup step still executes. It skips the Python/PyYAML cleanup script when no CI Compose manifest exists, as no Docker verification resources have been created at that point.

19 KiB Raw Blame History Unescape Escape