abearxiong
9da3d14752
- Implemented JimengService for image generation with API integration. - Created OSSService for handling image uploads to S3. - Developed PBService for managing PocketBase interactions. - Added task management for image creation and downloading using BullMQ. - Introduced routes for creating image generation tasks. - Implemented logging and error handling for image processing. - Added configuration management for Redis and other services. - Created scripts for testing image generation and PocketBase integration. - Updated package dependencies and added new scripts for worker management.
190 lines
3.7 KiB
Markdown
190 lines
3.7 KiB
Markdown
# 图片生成任务系统使用指南
|
||
|
||
基于 BullMQ 和 Redis 的异步图片生成任务系统。
|
||
|
||
## 系统架构
|
||
|
||
```
|
||
触发任务 → BullMQ 队列 → Worker 处理 → jimeng API 生成图片 → 上传 TOS → 更新 storage
|
||
```
|
||
|
||
## 前置要求
|
||
|
||
1. **Redis 服务**：确保 Redis 已启动
|
||
```bash
|
||
# 检查 Redis 状态
|
||
redis-cli ping
|
||
```
|
||
|
||
2. **环境变量配置**：确保 `.env` 文件已配置
|
||
- Redis 连接信息
|
||
- jimeng API 密钥
|
||
- TOS 配置
|
||
|
||
3. **PM2**：用于管理 Worker 进程
|
||
```bash
|
||
npm install -g pm2
|
||
```
|
||
|
||
## 使用方法
|
||
|
||
### 1. 启动 Worker
|
||
|
||
```bash
|
||
# 启动 Worker
|
||
bun run worker:start
|
||
|
||
# 查看日志
|
||
bun run worker:logs
|
||
|
||
# 查看 Worker 状态
|
||
bun run worker:status
|
||
|
||
# 重启 Worker
|
||
bun run worker:restart
|
||
|
||
# 停止 Worker
|
||
bun run worker:stop
|
||
```
|
||
|
||
### 2. 添加任务
|
||
|
||
#### 单个任务
|
||
```bash
|
||
bun run queue:single <promptId>
|
||
|
||
# 示例
|
||
bun run queue:single aadpldhvpwpdpwrp
|
||
```
|
||
|
||
#### 批量任务（所有待处理）
|
||
```bash
|
||
bun run queue:pending
|
||
```
|
||
|
||
### 3. 查询任务状态
|
||
|
||
```bash
|
||
bun run queue:status <jobId>
|
||
|
||
# 示例
|
||
bun run queue:status aadpldhvpwpdpdpwrp
|
||
```
|
||
|
||
## 任务流程
|
||
|
||
1. **添加任务**：从 storage 读取提示词，添加到 BullMQ 队列
|
||
2. **Worker 处理**：
|
||
- 验证提示词存在于 storage
|
||
- 调用 jimeng API 生成图片
|
||
- 下载图片到内存
|
||
- 上传到 TOS 服务器
|
||
- 更新 storage 文件（添加 imageUrl）
|
||
3. **结果返回**：任务完成或失败的信息
|
||
|
||
## 队列配置
|
||
|
||
- **并发数**：5（可通过 `QUEUE_CONCURRENCY` 环境变量调整）
|
||
- **重试次数**：3 次
|
||
- **重试策略**：指数退避（起始 5 秒）
|
||
- **失败暂停**：连续失败 10 个任务后自动暂停队列
|
||
|
||
## 文件结构
|
||
|
||
```
|
||
src/
|
||
├── config/
|
||
│ ├── redis.config.ts # Redis 配置
|
||
│ └── queue.config.ts # 队列配置
|
||
├── services/
|
||
│ ├── storage.service.ts # Storage 操作
|
||
│ ├── jimeng.service.ts # jimeng API
|
||
│ └── oss.service.ts # TOS 上传
|
||
├── queue/
|
||
│ ├── types.ts # 类型定义
|
||
│ ├── connection.ts # Redis 连接
|
||
│ ├── queue.ts # BullMQ 队列
|
||
│ ├── processor.ts # 任务处理器
|
||
│ └── index.ts # 导出
|
||
workers/
|
||
└── image-worker.ts # Worker 进程
|
||
scripts/
|
||
├── queue-single.ts # 单个任务脚本
|
||
├── queue-pending.ts # 批量任务脚本
|
||
└── check-status.ts # 状态查询脚本
|
||
```
|
||
|
||
## 环境变量
|
||
|
||
```bash
|
||
# Redis
|
||
REDIS_HOST=localhost
|
||
REDIS_PORT=6379
|
||
REDIS_PASSWORD=
|
||
REDIS_DB=0
|
||
|
||
# jimeng API
|
||
JIMENG_API_KEY=4e962fc85078d5bfc02c9882bfe659eb
|
||
JIMENG_API_URL=https://jimeng-api.kevisual.cn/v1
|
||
JIMENG_TIMEOUT=30000
|
||
|
||
# TOS
|
||
TOS_BUCKET=kevisual-images
|
||
TOS_ACCESS_KEY_ID=
|
||
TOS_ACCESS_KEY_SECRET=
|
||
TOS_REGION=cn-beijing
|
||
TOS_ENDPOINT=tos-cn-beijing.volces.com
|
||
|
||
# Queue
|
||
QUEUE_CONCURRENCY=5
|
||
QUEUE_MAX_FAILED=10
|
||
```
|
||
|
||
## 日志
|
||
|
||
Worker 日志位于 `logs/` 目录：
|
||
- `worker-error.log` - 错误日志
|
||
- `worker-out.log` - 输出日志
|
||
|
||
## 常见问题
|
||
|
||
### 1. Worker 启动失败
|
||
|
||
检查 Redis 是否运行：
|
||
```bash
|
||
redis-cli ping
|
||
```
|
||
|
||
检查环境变量是否配置正确：
|
||
```bash
|
||
cat .env
|
||
```
|
||
|
||
### 2. 任务失败
|
||
|
||
查看 Worker 日志：
|
||
```bash
|
||
bun run worker:logs
|
||
```
|
||
|
||
查询任务状态：
|
||
```bash
|
||
bun run queue:status <promptId>
|
||
```
|
||
|
||
### 3. 队列暂停
|
||
|
||
如果连续失败 10 个任务，队列会自动暂停。需要：
|
||
1. 排查失败原因
|
||
2. 手动恢复队列（代码中暂未实现，可使用 PM2 重启）
|
||
```bash
|
||
bun run worker:restart
|
||
```
|
||
|
||
## 下一步
|
||
|
||
- 添加 Bull Board 进行可视化监控
|
||
- 添加 HTTP API 代替 CLI 脚本
|
||
- 添加 Prometheus 指标
|
||
- 支持多 Worker 实例
|