Add documented KoKonna failure policy, bounded retries and durable recovery escalation [skip ci]

docs/KOKONNA_RECOVERY.md

@@ -0,0 +1,61 @@
+# KoKonna 错误、重试与阻塞处理
+
+核对日期：2026-10-16。依据：[KoKonna OpenAPI 官方文档](https://kokonna.art/pages/openapi)、[Requests 异常定义](https://requests.readthedocs.io/en/latest/_modules/requests/exceptions/)。本页区分厂商契约与 Frame Flow 的本地策略；没有文档保证的行为不作为自动重发依据。
+
+## 官方接口契约
+
+- 设备 API Key 使用 Bearer 鉴权；单设备每分钟上限 20 次，请求超限返回 429。
+- 上传成功返回新图片 ID 和计数，并立即激活新图；接口不提供幂等键或上传状态查询契约。
+- 删除可能改变当前图片。切换与删除都不能视为天然可重放。
+- 404 可表示方法不存在；图片下载时也可表示目标图片不存在。
+- 500 既可能是通用服务故障，也可能明确表示设备密钥无效、图片不存在或图片 ID 参数错误。实现只匹配官方明确列出的签名，其他 500 不猜测是否产生副作用。
+- 上传请求体上限 50 MB；本服务原图限制仍为 20 MiB，并保留像素上限和 base64 检查。
+- 官方文档未承诺服务处理时长、超时后的取消、事务回滚或 Retry-After 返回；不能靠等待固定时间证明写入没有执行。
+- Requests 将连接建立超时定义为可安全重试；读取超时不提供相同保证。一般连接异常也不能推定请求未发送。
+
+## 已落实的本地策略
+
+| 结果 | 处理 | 自动重试 | 设备阻塞 |
+| --- | --- | --- | --- |
+| 未配置密钥 | CONFIG_REQUIRED | 无 | 无 |
+| 明确的设备密钥无效 | CONFIG_REQUIRED，保存固定错误分类 | 无 | 无 |
+| 明确的图片不存在 | NOT_FOUND | 无 | 无 |
+| 明确的图片 ID 参数错误 | INVALID_REQUEST | 无 | 无 |
+| 明确的方法不存在 | UNSUPPORTED_CAPABILITY | 无 | 无 |
+| 其他 HTTP 4xx 拒绝 | 保存状态码，结束失败 | 除 429 外无 | 无 |
+| 本地请求预算耗尽 | 等待预算窗口 | 同一请求最多 3 次尝试 | 不创建 UNKNOWN |
+| HTTP 429 | RATE_LIMITED，尊重服务端冷却时间 | 最多 3 次尝试 | 耗尽后失败，不创建 UNKNOWN |
+| ConnectTimeout | 确定请求未发送 | 读写均最多 3 次尝试 | 耗尽后失败，不创建 UNKNOWN |
+| 只读查询的临时网络错误、通用 5xx、无效 JSON | CLOUD_UNAVAILABLE | 最多 3 次尝试 | 不创建 UNKNOWN |
+| TLS 异常 | 保存固定分类 | 无 | 写请求仍保守标为 UNKNOWN |
+| 写请求读取超时、一般连接异常、通用 5xx | 无法确定写入结果 | 不重发，只读核对 | UNKNOWN，保留保护 |
+| 写请求无效响应或无有效上传 ID | 无法确认成功 | 不重发，只读核对 | UNKNOWN，保留保护 |
+
+“最多 3 次”包含首次请求，重试发生在同一传输请求内，不重新渲染、不重新执行整个操作。普通退避为 1、2 秒；每次实际请求都计入本地 18 次/分钟预算。429 优先使用 Retry-After 秒数或 HTTP 日期；缺失/无效时按 60 秒。单次等待不超过 60 秒，超过此值保留冷却并结束当前尝试，不提前请求。没有无限循环，也不在重启后重放不确定的写入。
+
+操作错误的 details 保存 attempts、retryable、write_outcome，以及 transport_error / http_status / failure_kind 等固定分类。不会保存异常原文、完整下载 URL、带密钥的厂商消息或响应正文。是否重试由结果类别决定，不能仅依据 HTTP 500。
+
+## 未知写入的核对与升级
+
+上传前记录资产、哈希、字节数、名称、上传前图库和时间。响应丢失后只读查找并校验唯一候选；图片相同且设备已同步才能记为成功。已证明上传、但设备持续停留原图时，沿用两次稳定观察的失败关闭规则。
+
+没有候选、候选不唯一、内容不符、设备未同步、查询失败或缺少历史证据，都有独立恢复原因。图库没有目标图片不等于厂商保证从未接受请求，因此不自动把它改为失败或重传。
+
+`GET /api/v1/operations/{operation_id}` 的 `recovery` 返回：
+
+- `state`：checking / manual_review / resolved。
+- `reason`：gallery_match_missing、gallery_match_ambiguous、content_mismatch、waiting_device_sync、target_mismatch、read_unavailable、reconciliation_error、missing_upload_evidence 或 manual_reconciliation_required。
+- `first_checked_at`、`last_checked_at`、`checks`、`requires_manual_review`。
+
+只读核对每条操作至多每分钟一次。操作持续未知超过 sync_timeout + 300 秒（默认约 15 分钟）时，持久化一次 `operation.blocked` 事件和 `MANUAL_REVIEW_REQUIRED` 历史记录；服务重启、重复检查或原因改变均不会重复升级同一条未解决操作。既有检查继续运行，不自动取消设备保护。
+
+管理员可在 Webhooks 中订阅 `operation.blocked`；未配置订阅不会自动向外发送通知。事件和 API 记录始终保留，但它们不能替代已配置的通知渠道。单个任务失败、单条未知操作的元数据异常均不应中断其他任务处理。
+
+## 人工处理
+
+1. 根据 operation_id 查看原始错误、恢复原因、当前设备和图库；不点击重新执行来猜测结果。
+2. 调用现有 reconcile 接口并传 `acknowledge=false` 进行核对。
+3. 无法证明成功、但管理员确认可以结束旧操作时，传 `acknowledge=true`：结束为失败，保留原始目标、上传意图和此前错误，记录观察到的图片与人工处理历史。该接口不会重发设备写入。
+4. 有目标一致且同步完成的证据才记为成功。处理后恢复原计划，错过的任务不补发。
+
+仍可能因厂商无法给出明确写入结果而需要人工处理；本策略解决错误误判、可安全重试的暂时失败和长期阻塞缺乏解释的问题，不承诺网络故障时所有设备任务都能自动成功。

docs/REAL_DEVICE_TEST.md

@@ -41,3 +41,7 @@ minute apart record `DISPLAY_NOT_CONFIRMED` and unblock future scheduled work.
 Temporary overrides retain their restore behavior and are not automatically
 closed as failed. Legacy operations without durable upload evidence require
 manual acknowledgement; absence from the gallery alone never proves failure.
+
+### 错误与有限重试
+
+完整错误矩阵、重试边界、持久化恢复状态和人工处理见 [KoKonna 恢复策略](KOKONNA_RECOVERY.md)。自动重试只允许只读请求、连接建立超时或明确限流拒绝；读取超时等未知写入不会重发。

docs/RESEARCH.md

@@ -1,6 +1,7 @@
 # KoKonna API research / 接口调查
 
-Checked 2026-10-01 against https://kokonna.art/pages/openapi.
+Rechecked 2026-10-16 against https://kokonna.art/pages/openapi.
+错误分类、重试和未知写入处理见 [KoKonna 恢复策略](KOKONNA_RECOVERY.md)。
 Base: https://api.kokonna.art/openapi. No fallback to legacy galaxyguide.cn.
 
 | Official endpoint | Frame Flow capability |
@@ -29,7 +30,7 @@ headroom at 18. Device state has a 60-second cache, gallery is loaded on demand.
 Uploads automatically activate a new image; there is no upload-without-activation
 flag. Gallery-only uploads are therefore local staging, not a fictitious cloud
 operation. Maximum upstream request size is 50 MB; this service accepts 20 MiB
-image files with bounded decoded pixels. No non-idempotent cloud write is retried.
+image files with bounded decoded pixels. Only proven-unsent connections or explicit rate-limit rejections may retry a write, within a three-attempt budget. Ambiguous writes are never replayed.
 Cloud acknowledgement is not physical refresh confirmation; /device synced and
 imageId drive the operation state. Unconfirmed or interrupted writes become UNKNOWN
 and block later device writes until read-only reconciliation/explicit acknowledgement.

docs/openapi.json

@@ -2,7 +2,7 @@
   "openapi": "3.1.0",
   "info": {
     "title": "Pigway Frame Flow API",
-    "version": "0.1.0"
+    "version": "0.1.4"
   },
   "paths": {
     "/api/v1/devices/{device_id}/widgets/selection": {
@@ -6397,7 +6397,15 @@
           "content": {
             "application/json": {
               "schema": {
-                "$ref": "#/components/schemas/TemplateContent"
+                "anyOf": [
+                  {
+                    "$ref": "#/components/schemas/TemplateContent"
+                  },
+                  {
+                    "$ref": "#/components/schemas/AutoCollageContent"
+                  }
+                ],
+                "title": "Payload"
               }
             }
           }
@@ -14298,6 +14306,87 @@
         ],
         "title": "AssetBatch"
       },
+      "AutoCollageContent": {
+        "properties": {
+          "type": {
+            "type": "string",
+            "const": "auto_collage",
+            "title": "Type"
+          },
+          "source_id": {
+            "anyOf": [
+              {
+                "type": "string",
+                "pattern": "^[a-z0-9-]{1,60}$"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Source Id"
+          },
+          "count": {
+            "type": "integer",
+            "maximum": 12.0,
+            "minimum": 1.0,
+            "title": "Count",
+            "default": 4
+          },
+          "title": {
+            "type": "string",
+            "maxLength": 100,
+            "title": "Title",
+            "default": ""
+          },
+          "message": {
+            "type": "string",
+            "maxLength": 500,
+            "title": "Message",
+            "default": ""
+          },
+          "orientation": {
+            "type": "string",
+            "enum": [
+              "auto",
+              "portrait",
+              "landscape"
+            ],
+            "title": "Orientation",
+            "default": "auto"
+          },
+          "style": {
+            "type": "string",
+            "enum": [
+              "grid",
+              "hero",
+              "poster"
+            ],
+            "title": "Style",
+            "default": "grid"
+          },
+          "use_ai": {
+            "type": "boolean",
+            "title": "Use Ai",
+            "default": true
+          },
+          "prompt": {
+            "type": "string",
+            "maxLength": 1500,
+            "title": "Prompt",
+            "default": ""
+          },
+          "profile": {
+            "$ref": "#/components/schemas/Profile",
+            "default": "color"
+          }
+        },
+        "additionalProperties": false,
+        "type": "object",
+        "required": [
+          "type"
+        ],
+        "title": "AutoCollageContent"
+      },
       "Body_import_images_api_v1_layout_images_post": {
         "properties": {
           "files": {
@@ -14778,6 +14867,9 @@
               {
                 "$ref": "#/components/schemas/PhotoSourceContent"
               },
+              {
+                "$ref": "#/components/schemas/AutoCollageContent"
+              },
               {
                 "$ref": "#/components/schemas/ProviderContent"
               },
@@ -14789,6 +14881,7 @@
             "discriminator": {
               "propertyName": "type",
               "mapping": {
+                "auto_collage": "#/components/schemas/AutoCollageContent",
                 "html": "#/components/schemas/FutureContent",
                 "image": "#/components/schemas/ImageContent",
                 "image_url": "#/components/schemas/URLContent",
@@ -14852,6 +14945,7 @@
           "UNSUPPORTED_CAPABILITY",
           "RENDER_FAILED",
           "UPLOAD_FAILED",
+          "DISPLAY_NOT_CONFIRMED",
           "WRITE_RESULT_UNKNOWN",
           "RATE_LIMITED",
           "INTERNAL_ERROR",
@@ -15062,7 +15156,7 @@
           "version": {
             "type": "string",
             "title": "Version",
-            "default": "0.1.0"
+            "default": "0.1.4"
           }
         },
         "type": "object",
@@ -15324,6 +15418,9 @@
                   {
                     "$ref": "#/components/schemas/PhotoSourceContent"
                   },
+                  {
+                    "$ref": "#/components/schemas/AutoCollageContent"
+                  },
                   {
                     "$ref": "#/components/schemas/ProviderContent"
                   },
@@ -15334,6 +15431,7 @@
                 "discriminator": {
                   "propertyName": "type",
                   "mapping": {
+                    "auto_collage": "#/components/schemas/AutoCollageContent",
                     "html": "#/components/schemas/FutureContent",
                     "image": "#/components/schemas/ImageContent",
                     "image_url": "#/components/schemas/URLContent",
@@ -15469,6 +15567,9 @@
                   {
                     "$ref": "#/components/schemas/PhotoSourceContent"
                   },
+                  {
+                    "$ref": "#/components/schemas/AutoCollageContent"
+                  },
                   {
                     "$ref": "#/components/schemas/ProviderContent"
                   },
@@ -15479,6 +15580,7 @@
                 "discriminator": {
                   "propertyName": "type",
                   "mapping": {
+                    "auto_collage": "#/components/schemas/AutoCollageContent",
                     "html": "#/components/schemas/FutureContent",
                     "image": "#/components/schemas/ImageContent",
                     "image_url": "#/components/schemas/URLContent",
@@ -15787,6 +15889,16 @@
       },
       "Operation": {
         "properties": {
+          "recovery": {
+            "anyOf": [
+              {
+                "$ref": "#/components/schemas/RecoveryState"
+              },
+              {
+                "type": "null"
+              }
+            ]
+          },
           "operation_id": {
             "type": "string",
             "title": "Operation Id"
@@ -16808,6 +16920,49 @@
         ],
         "title": "Reconciliation"
       },
+      "RecoveryState": {
+        "properties": {
+          "state": {
+            "type": "string",
+            "enum": [
+              "checking",
+              "manual_review",
+              "resolved"
+            ],
+            "title": "State"
+          },
+          "reason": {
+            "type": "string",
+            "title": "Reason"
+          },
+          "first_checked_at": {
+            "type": "string",
+            "title": "First Checked At"
+          },
+          "last_checked_at": {
+            "type": "string",
+            "title": "Last Checked At"
+          },
+          "checks": {
+            "type": "integer",
+            "title": "Checks"
+          },
+          "requires_manual_review": {
+            "type": "boolean",
+            "title": "Requires Manual Review"
+          }
+        },
+        "type": "object",
+        "required": [
+          "state",
+          "reason",
+          "first_checked_at",
+          "last_checked_at",
+          "checks",
+          "requires_manual_review"
+        ],
+        "title": "RecoveryState"
+      },
       "Region": {
         "properties": {
           "asset_id": {
@@ -17178,7 +17333,7 @@
           "version": {
             "type": "string",
             "title": "Version",
-            "default": "0.1.0"
+            "default": "0.1.4"
           },
           "api_version": {
             "type": "string",