Compare commits

..
15 Commits
Author SHA1 Message Date
greg edc755fdb8 Merge branch 'main' into fix/port-ark-survival-ascended-crossplay
Validate game templates / validate (push) Successful in 19s
Validate game templates / boot-test (push) Successful in 5m58s
Validate game templates / publish (push) Skipped
2026-10-26 00:19:41 +00:00
openhands ef4d72621d Merge pull request 'fix(build): clear a killed boot test's container and volume first' (#16) from fix/worker-finish-job-on-stop into main
Builder image / publish (push) Successful in 21s
Validate game templates / validate (push) Successful in 24s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Reviewed-on: #16
2026-10-25 22:33:31 +00:00
gregandClaude Opus 5.5 e9e78610d6 fix(build): clear a killed boot test's container and volume first
Validate game templates / validate (push) Successful in 20s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
When the worker is restarted mid-build (watchtower updating its image), the
boot test's container and volume are left behind, and the next boot test of
that game deploys under the same name. Remove them before each boot test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:32:43 -04:00
openhands ea2c0b1eeb Merge pull request 'feat(worker): one draft per game; a rebuild replaces the earlier one' (#15) from fix/worker-branch-per-run into main
Builder image / publish (push) Successful in 22s
Validate game templates / validate (push) Successful in 24s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Reviewed-on: #15
2026-10-25 22:18:55 +00:00
gregandClaude Opus 5.5 5d7dee3305 feat(worker): one draft per game; a rebuild replaces the earlier one
Validate game templates / validate (push) Successful in 20s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
The draft branch was named after the job. Rerunning a build pushed onto the
earlier run's branch, which had review commits, so the push was rejected
and the finished build failed at its last step. Separate branches per run
would pile up open PRs for the same game instead.

A game now has one draft at a time, on draft/<id>. Before pushing, the
worker closes the game's open draft PRs (draft/<id>, and draft/<id>-<job>
as earlier versions named them) and deletes their branches, then
force-pushes draft/<id> and opens the new PR, which says which PR it
replaces. Other games' drafts and anyone's non-draft branches are never
touched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:28 -04:00
openhands f1d074bb11 Merge pull request 'feat(builder): documented launch options only, and no game-specific code' (#14) from feat/documented-launch-options into main
Builder image / publish (push) Successful in 22s
Validate game templates / validate (push) Successful in 24s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Reviewed-on: #14
2026-10-25 22:01:37 +00:00
gregandClaude Opus 5.5 8623ba2d7b feat(builder): documented launch options only, and no game-specific code
Validate game templates / validate (push) Successful in 20s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Launch options must be documented. A draft passed an option its server
doesn't have; servers ignore unknown options, so it booted and passed every
boot test while its setting did nothing. build now reads the pages research
based the variant on, in full, and the drafter rejects any launch option in
the startup command that none of them, nor the image's own scripts, mention.
Shell tools' options (mkdir -p, [ -n ...]) are skipped; the check is off
without sources.

The builder is game-agnostic:
- AUTHORING.md (the model's rules), the research prompt and the schema
  describe the rules without naming a game.
- The draft prompt no longer includes real templates from the repo. Its
  format example is one invented server (no real command, options, ports or
  app id), checked by TestExampleTemplateFollowsTheRules.
- Comments, help texts, BUILDER.md and test fixtures use invented names and
  values.
- TestBuilderIsGameAgnostic reads every game's name and id from games/ and
  fails if any appears in the builder's code, tests, rules, schemas or docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:58:53 -04:00
greg dd92b6be4c Merge pull request 'fix(worker): start with a repository-only token' (#12) from fix/worker-repo-scope-token into main
Builder image / publish (push) Successful in 22s
Validate game templates / validate (push) Successful in 24s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Reviewed-on: #12
2026-10-25 21:18:40 +00:00
gregandClaude Opus 5.5 0c868f7bac fix(worker): start with a repository-only token
Validate game templates / validate (push) Successful in 20s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
The start-up check read the token's account with GET /user, which needs the
read:user scope. A token with only repository scope, which is all the worker
needs, got 403 and the worker exited in a restart loop. On 403 it now
names the commit author from BOT_NAME (default gbr-bot) and still checks
push access on the games repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:18:22 -04:00
greg 425c242ae7 Merge pull request 'fix(worker): commit as the token's account and check it can push' (#11) from fix/worker-bot-identity into main
Builder image / publish (push) Successful in 21s
Validate game templates / validate (push) Successful in 24s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Reviewed-on: #11
2026-10-25 21:07:49 +00:00
gregandClaude Opus 5.5 f3a2a1f099 fix(worker): commit as the token's account and check it can push
Validate game templates / validate (push) Successful in 21s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
The worker committed as a hard-coded "gbr-bot", and a token without Write
access only showed up when a build finished and its push failed. On start it
now reads the token's account (GET /user) for the commit author, and
refuses to run if that account can't push to the games repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:28:54 -04:00
greg 929e7d68d0 Merge pull request 'feat(builder): research, build and a worker for the AI builder' (#6) from feat/builder-worker into main
Validate game templates / validate (push) Successful in 31s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Builder image / publish (push) Successful in 42s
Reviewed-on: #6
2026-10-25 20:22:45 +00:00
gregandClaude Opus 5.5 5dfc0cc794 fix(builder): drafts don't use CI secrets, and a missing one is fed back
Validate game templates / validate (push) Successful in 20s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
The second Valheim build stopped before its boot test: told that
test.yaml's password "looks like a credential; map it under 'secrets'",
the model mapped it to a CI secret nobody created, and boottest-payload's
missing-secret exit ended the build.

- The validator now says to remove a test password first (boot tests fill
  password settings with a throwaway value); secrets only when the server
  can't start without a real one.
- Drafts with test.yaml 'secrets' fail the draft checks.
- build turns boottest-payload's missing-secret exit into feedback for the
  model instead of an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:52:18 -04:00
gregandClaude Opus 5.5 c2b22d3955 feat(draft): catch split text settings and owner-editable game ports
Validate game templates / validate (push) Successful in 19s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
Two problems a draft can boot fine with but break once an owner changes a
setting, both in the builder's Valheim draft:

- splitSettings: text settings used unquoted on the command line, directly
  or through ARGS="-name ${SERVER_NAME}"; ./server $ARGS, so "GBRHosting -
  My Server" reaches the game as several arguments. Suggests quoting or a
  bash array.
- portSettings: game-port settings whose default isn't in defaultPorts, or
  that owners can change although nodes always publish the same container
  port. Internal ports (RCON, REST) and {PUBLIC_PORT_n} values are left out.

Both run on drafts only; AUTHORING.md gets the matching rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 15:43:27 -04:00
gregandClaude Opus 5.5 c75190d8c4 feat(builder): research, build and a worker for the AI builder
Validate game templates / validate (push) Successful in 19s
Validate game templates / boot-test (push) Skipped
Validate game templates / publish (push) Skipped
- research: searches SearXNG, reads the best pages through the reader, has the
  model list the server's variants, then checks every Docker image and Steam
  app it named for real before an admin sees it.
- build: first boot of the variant's images (skipping ones that need root),
  a draft by the model, then the draft deployed with the node-agent's
  BootTest like CI does; crashes, error lines and saves outside /server go
  back to the model, three boots at most.
- worker: polls the panel's builder queue (outbound only), runs research or
  build in a fresh clone, streams the output as the job log, and opens the
  result as a pull request from the bot account.
- Dockerfile.builder, a workflow publishing gbr-builder:latest, and
  BUILDER.md. Everything about where it runs comes from environment
  variables; deployment files live outside this public repo.

draft now keeps its conversation in a drafter, so build can send boot
failures back to the same model session. The boot-test runner comment no
longer names its VM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 12:51:29 -04:00
23 changed files with 2483 additions and 170 deletions

No files matched your search

+37
View File
@@ -0,0 +1,37 @@
name: Builder image
# Publishes the AI builder worker image (gbrgames worker, see BUILDER.md).
on:
push:
branches: ["main"]
paths:
- "cmd/**"
- "go.mod"
- "go.sum"
- "Dockerfile.builder"
workflow_dispatch:
env:
REGISTRY: www.gitea.gbreed.com
IMAGE_NAME: gbr-builder
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Login to Gitea Container Registry
run: |
echo "${{ secrets.GIT_SERVER_TOKEN }}" | docker login ${{ env.REGISTRY }} \
-u ${{ secrets.GIT_TOKEN_USERNAME }} \
--password-stdin
- name: Build and Push Docker Image
run: |
IMAGE_PATH="${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}"
SHORT_SHA=$(echo "${{ github.sha }}" | cut -c1-7)
docker build -f Dockerfile.builder -t $IMAGE_PATH:$SHORT_SHA -t $IMAGE_PATH:latest .
docker push $IMAGE_PATH:$SHORT_SHA
docker push $IMAGE_PATH:latest
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
boot-test: boot-test:
needs: validate needs: validate
if: needs.validate.outputs.games != '' if: needs.validate.outputs.games != ''
# A dedicated runner on VM 116 whose game containers can reach the internet but not the LAN # A dedicated runner whose game containers can reach the internet but not private networks
runs-on: game-test runs-on: game-test
steps: steps:
- name: Checkout Code - name: Checkout Code
+13 -4
View File
@@ -41,22 +41,31 @@ Know this before writing anything:
- `defaultPorts` lists **only what players connect to** (game and query ports). RCON, REST and admin - `defaultPorts` lists **only what players connect to** (game and query ports). RCON, REST and admin
ports are **never** published. ports are **never** published.
- Use `"7777/udp"` or `"8211:8211/udp"`. Every port needs its protocol. - Use `"7777/udp"` or `"8211:8211/udp"`. Every port needs its protocol.
- Nodes always publish these container ports (only the host side moves when servers share a node).
So the game's own port is **not a setting**: write it into the startup command or config file. A
game that must announce the port players use gets `{PUBLIC_PORT_<port>}`.
### The container user ### The container user
- Run `gbrgames inspect-image -game <id>`. If the image is built around its own user (it declares - Run `gbrgames inspect-image -game <id>`. If the image is built around its own user (it declares
`USER`, or locks a home folder to a uid other than 1000), set `containerUser` to that `uid:gid`. `USER`, or locks a home folder to a uid other than 1000), set `containerUser` to that `uid:gid`, as
Example: ARK's image needs `25000:25000`. the inspector reports it.
### The startup command ### The startup command
- Read settings under the names the agent gives them (see above). `${RCON_PORT}` when the setting is - Read settings under the names the agent gives them (see above). `${RCON_PORT}` when the setting is
`RCONPort` silently ignores the owner's value; it arrives as `RCONPORT`. `RCONPort` silently ignores the owner's value; it arrives as `RCONPORT`.
- Use `${VAR:-default}` for anything that may be unset. - Use `${VAR:-default}` for anything that may be unset.
- Only use launch options the game's documentation lists, spelled as it spells them. Servers ignore options
they don't know, so an invented one boots fine and its setting does nothing.
- Quote text settings where they're used: `-name "${SERVER_NAME}"`. `ARGS="-name ${SERVER_NAME}"; ./server $ARGS`
splits "GBRHosting - My Server" into words; build optional arguments in a bash array instead
(`args=(-name "$SERVER_NAME"); [ -n "$PASSWORD" ] && args+=(-password "$PASSWORD"); ./server "${args[@]}"`).
- If the image installs the game itself on start, make sure a failed install is retried rather than - If the image installs the game itself on start, make sure a failed install is retried rather than
starting a server that isn't there (see ARK's template). starting a server that isn't there: log in to SteamCMD once (`+login anonymous +quit`) before the
install, and retry the install a few times until the server's executable exists.
### Memory ### Memory
- `requiredMemoryMb` is the container's hard limit. A JVM or engine heap must stay about **1 GB below** - `requiredMemoryMb` is the container's hard limit. A JVM or engine heap must stay about **1 GB below**
it: Minecraft's `MEMORY` is `3G` for `4096` MB. A heap equal to the limit is killed on start. it: a `3G` heap for `4096` MB. A heap equal to the limit is killed on start.
### Settings (`defaultConfig` and `configSchema`) ### Settings (`defaultConfig` and `configSchema`)
- Take keys and default values from what the server really writes: run - Take keys and default values from what the server really writes: run
+67
View File
@@ -0,0 +1,67 @@
# AI builder (gbrgames worker)
The panel's **AI builder** (Admin → Games → AI Config Generator) is run by a worker we host ourselves.
The panel only keeps a queue; every result is a pull request to this repo, which CI validates and
boot-tests like any other.
```
Panel Builder worker (gbrgames worker, Docker)
───── ─────────────────────────────────────────
admin starts a job ──► queued ◄─poll── research: web search + page reader + model
progress / log / variants ◄─post── build: first boot → draft → boot with the node-agent
"Review draft PR" → branch draft/<id>-<job> + PR as the bot ──► CI (validate + boot test)
```
1. **Research** (`gbrgames research -name N`): searches the web, reads the best pages, and has the model list
the server's variants (edition, image or SteamCMD app, startup, ports). Every image and Steam app is
checked for real (`docker manifest inspect`, anonymous `steamcmd +app_info_print`) before the admin
sees it.
2. The admin picks a variant in the panel.
3. **Build** (`gbrgames build -name N -variant @v.json`): first boot of the image untouched (`first-boot`), a
draft by the model (`draft`, rules in `AUTHORING.md`), then the draft is deployed with the node-agent's
`BootTest`, the same check CI runs. Failures (crash, error lines, saves written outside `/server`) go
back to the model, three boots at most.
4. The worker pushes `games/<id>/` to a branch and opens a pull request as the bot account. A draft that
validates but never booted still gets a PR, titled `WIP:`, with what's left listed. Nothing reaches
players until a person merges it and enables the game.
The builder is **game-agnostic**: everything about a game comes from that game's research, first boot and
boot tests. No game's name, id or code may appear in the builder's source, its rules for the model
(`AUTHORING.md`), the schemas or this file, and the model is never shown another game's template (its format
example is an invented server). `TestBuilderIsGameAgnostic` fails the build if one does.
The worker only connects out (to the panel, Gitea, the model and the search services).
## Requirements
A Docker host with the isolated `gbr-boottest` network (`RUNNER.md`, step 1), so servers booted by the
builder can't reach private networks. Enough memory for the largest server it builds (the biggest need ~16 GB) and
disk for SteamCMD downloads. The image `gbr-builder:latest` is published by
`.gitea/workflows/builder-image.yml` whenever `cmd/` changes on main.
## Configuration
All through environment variables; nothing about where it runs lives in this repo.
| Variable | What |
| --- | --- |
| `PANEL_URL` | The panel the worker takes jobs from |
| `BUILDER_TOKEN` | Shared secret the panel checks on `/api/v1/builder/**` (same value in the panel's env) |
| `GITEA_TOKEN` | The bot account's token (repository read/write; the account needs Write on this repo): pushes draft branches, opens PRs as that account. Checked when the worker starts |
| `OLLAMA_URL` | Ollama for the model (or `MODEL_PROVIDER=gemini` with `GEMINI_API_KEY`) |
| `SEARXNG_URL`, `READER_URL` | A SearXNG instance (JSON API) and a page-to-text reader (`READER_URL/<page url>`) |
| `MODEL`, `POLL_INTERVAL`, `WORKER_NAME`, `WORK_DIR` | Optional: model name, poll interval (20s), name shown in the panel, work folder |
| `BOT_NAME` | Optional: commit author when the token can't read its own account (no `read:user` scope); default `gbr-bot` |
## Running a build by hand
On any machine with Docker and the `gbr-boottest` network, with `OLLAMA_URL`, `SEARXNG_URL` and
`READER_URL` set:
```bash
go run ./cmd/gbrgames research -name "<game>" -out research.json
# pick a variant from research.json into v.json, then:
go run ./cmd/gbrgames build -name "<game>" -variant @v.json -out build.json
```
The draft lands in `games/<id>/`, first-boot reports in `firstboot-out/`.
+14
View File
@@ -0,0 +1,14 @@
# The AI builder worker (gbrgames worker); see BUILDER.md.
FROM golang:1.24-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY cmd ./cmd
RUN CGO_ENABLED=0 go build -o /gbrgames ./cmd/gbrgames
FROM alpine:3.22
# docker CLI for first boots and boot tests (through the host's socket), git to clone and open PRs
RUN apk add --no-cache docker-cli git ca-certificates
COPY --from=build /gbrgames /usr/local/bin/gbrgames
WORKDIR /work
ENTRYPOINT ["gbrgames", "worker"]
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// The builder must work the same for every game: no game's name, id or code in its source, its rules for
// the model (AUTHORING.md), the schemas sent with every draft, or its docs. Games are read from games/,
// so a newly merged template is covered as soon as it lands.
func TestBuilderIsGameAgnostic(t *testing.T) {
root := filepath.Join("..", "..")
templates, _ := filepath.Glob(filepath.Join(root, "games", "*", "game.yaml"))
if len(templates) == 0 {
t.Fatal("no templates found to check against")
}
var names []string
for _, f := range templates {
g, err := loadMap(f)
if err != nil {
continue
}
for _, key := range []string{"id", "name"} {
if s, _ := g[key].(string); len(s) > 2 {
names = append(names, s)
}
}
}
files, _ := filepath.Glob("*.go")
files = append(files, filepath.Join(root, "AUTHORING.md"), filepath.Join(root, "BUILDER.md"))
schemas, _ := filepath.Glob(filepath.Join(root, "schema", "*.json"))
files = append(files, schemas...)
for _, f := range files {
raw, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
text := string(raw)
for _, n := range names {
re := regexp.MustCompile(`(?i)(^|[^A-Za-z0-9])` + regexp.QuoteMeta(n) + `($|[^A-Za-z0-9])`)
if loc := re.FindStringIndex(text); loc != nil {
line := strings.Count(text[:loc[0]], "\n") + 1
t.Errorf("%s:%d mentions %q: the builder must not name or copy any game", f, line, n)
}
}
}
}
+401
View File
@@ -0,0 +1,401 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
// build turns one researched variant into a template that has really booted: first boot of the image
// untouched (facts), a draft by the model, then the draft deployed with the node-agent's own code, exactly
// as CI's boot test does. What goes wrong in that boot (a crash, an error line, saves outside the volume)
// goes back to the model, a few rounds at most. The result is a games/<id>/ folder ready for a pull request.
type buildResult struct {
GameID string `json:"gameId"`
Dir string `json:"dir"`
Image string `json:"image"`
Passed bool `json:"passed"`
Rounds int `json:"rounds"`
Verified []string `json:"verified"`
Problems []string `json:"problems"`
Summary string `json:"summary"`
}
const defaultAgentImage = "www.gitea.gbreed.com/gbreed-public/gbr-node-agent:latest"
func runBuild(args []string) error {
fs := flag.NewFlagSet("build", flag.ExitOnError)
gamesDir := fs.String("games", "games", "directory holding one folder per game")
schemaDir := fs.String("schemas", "schema", "directory holding the JSON Schemas")
gameName := fs.String("name", "", "the game's name")
id := fs.String("id", "", "folder/id for the new game (default: from -name)")
variantArg := fs.String("variant", "", "the chosen variant: a research result variant as JSON, or @file")
notes := fs.String("notes", "", "what the admin asked for")
provider := fs.String("provider", "ollama", "ollama or gemini")
modelName := fs.String("model", "", "model name (default as for draft)")
attempts := fs.Int("attempts", 3, "fix rounds for validation problems, each time")
bootRounds := fs.Int("boot-rounds", 3, "boots of the draft before giving up")
out := fs.String("out", "", "write the result JSON here")
fs.Parse(args)
if *gameName == "" || *variantArg == "" {
return errors.New("build needs -name and -variant")
}
raw := []byte(*variantArg)
if strings.HasPrefix(*variantArg, "@") {
var err error
if raw, err = os.ReadFile((*variantArg)[1:]); err != nil {
return err
}
}
var v variant
if err := json.Unmarshal(raw, &v); err != nil {
return fmt.Errorf("-variant: %w", err)
}
if *id == "" {
*id = gameID(*gameName)
}
agentImage := os.Getenv("AGENT_IMAGE")
if agentImage == "" {
agentImage = defaultAgentImage
}
res, err := build(buildOptions{gamesDir: *gamesDir, schemaDir: *schemaDir, name: *gameName, id: *id, variant: v,
notes: *notes, provider: *provider, model: *modelName, attempts: *attempts, bootRounds: *bootRounds, agentImage: agentImage})
if res != nil && *out != "" {
raw, _ := json.MarshalIndent(res, "", " ")
if werr := os.WriteFile(*out, raw, 0o644); werr != nil && err == nil {
err = werr
}
}
if err != nil {
return err
}
if !res.Passed {
return fmt.Errorf("the draft in %s still has problems:\n- %s", res.Dir, strings.Join(res.Problems, "\n- "))
}
fmt.Printf("[build] %s booted with the node-agent and is ready for a pull request\n", res.Dir)
return nil
}
type buildOptions struct {
gamesDir, schemaDir, name, id string
variant variant
notes, provider, model string
attempts, bootRounds int
agentImage string
}
func build(o buildOptions) (*buildResult, error) {
if len(o.variant.Images) == 0 {
return nil, errors.New("the variant has no image to build from")
}
res := &buildResult{GameID: o.id, Dir: filepath.Join(o.gamesDir, o.id)}
res.Verified = append(res.Verified, o.variant.Verified...)
// 1. First boot of each candidate image until one runs as a non-root user
var report *bootReport
for _, image := range o.variant.Images {
steam := isSteamCMDImage(image) && o.variant.SteamAppID != ""
if isSteamCMDImage(image) && !steam {
continue // the SteamCMD image alone runs nothing
}
res.Image = image
if steam && o.variant.StartupHint == "" {
fmt.Printf("[build] no startup command known for Steam app %s; drafting without a first boot\n", o.variant.SteamAppID)
report = nil
break
}
fmt.Printf("[build] first boot of %s\n", image)
r, err := firstBoot(o.id+"-probe", image, o.variant.SteamAppID, steam, o.variant.StartupHint)
if err != nil {
return nil, err
}
if why := imageUnusable(r); why != "" {
fmt.Printf("[build] %s can't be used: %s\n", image, why)
res.Problems = append(res.Problems, why)
res.Image = ""
continue
}
report = r
res.Verified = append(res.Verified, fmt.Sprintf("first boot of %s: %s", image, r.Outcome))
break
}
if res.Image == "" {
return res, errors.New("no image of this variant runs as a non-root user")
}
res.Problems = nil
// 2. Draft until validation and the image checks pass
steamApp := ""
if isSteamCMDImage(res.Image) {
steamApp = o.variant.SteamAppID
}
d, err := newDrafter(draftJob{GamesDir: o.gamesDir, SchemaDir: o.schemaDir, Name: o.name, ID: o.id, Image: res.Image,
SteamApp: steamApp, Notes: variantNotes(o.variant, o.notes), Report: report, Provider: o.provider, Model: o.model, NoPull: true,
Docs: sourceText(o.variant.Sources)})
if err != nil {
return nil, err
}
problems, err := d.refine(o.attempts, nil)
res.Rounds = d.rounds
if err != nil {
return res, err
}
if len(problems) > 0 {
res.Problems = problems
res.Summary = buildSummary(o, res)
return res, nil
}
// 3. Boot the draft the way nodes run it; failures go back to the model
wantOutside := len(outsideDirs(report)) > 0
for boot := 1; boot <= o.bootRounds; boot++ {
fmt.Printf("[build] boot %d of the draft with the node-agent\n", boot)
bootProblems, notes, err := bootCheck(o.gamesDir, o.id, o.agentImage, wantOutside)
if err != nil {
return res, err
}
if len(bootProblems) == 0 {
res.Passed = true
res.Verified = append(res.Verified, notes...)
break
}
fmt.Printf("[build] boot %d failed:\n", boot)
for _, p := range bootProblems {
fmt.Println(" - " + firstLine(p))
}
res.Problems = bootProblems
if boot == o.bootRounds {
break
}
problems, err := d.refine(o.attempts, bootProblems)
res.Rounds = d.rounds
if err != nil {
return res, err
}
if len(problems) > 0 {
res.Problems = problems
break
}
}
if res.Passed {
res.Problems = nil
}
res.Summary = buildSummary(o, res)
return res, nil
}
func isSteamCMDImage(image string) bool {
return image == "cm2network/steamcmd" || strings.HasPrefix(image, "cm2network/steamcmd:")
}
func variantNotes(v variant, admin string) string {
var parts []string
if v.Title != "" {
parts = append(parts, "variant: "+v.Title+" ("+v.Description+")")
}
if v.StartupHint != "" {
parts = append(parts, "documented startup: "+v.StartupHint)
}
if len(v.Ports) > 0 {
parts = append(parts, "documented ports: "+strings.Join(v.Ports, ", "))
}
if v.Notes != "" {
parts = append(parts, v.Notes)
}
if admin != "" {
parts = append(parts, "the admin asked for: "+admin)
}
return strings.Join(parts, "; ")
}
// firstBoot runs first-boot and reads back its report.
func firstBoot(name, image, steamApp string, steam bool, startup string) (*bootReport, error) {
outDir := filepath.Join("firstboot-out", name)
args := []string{"-image", image, "-out", outDir}
if steam {
args = append(args, "-steam-app", steamApp, "-startup", startup)
}
if err := runFirstBoot(args); err != nil {
return nil, err
}
return readBootReport(filepath.Join(outDir, "report.json"))
}
// bootCheck deploys the template with the node-agent's BootTest, the same check CI runs, and when the
// game wrote files outside its volume on the first boot, boots it once more to see that it no longer does.
func bootCheck(gamesDir, id, agentImage string, wantOutside bool) (problems, verified []string, err error) {
self, err := os.Executable()
if err != nil {
return nil, nil, err
}
// A boot test that was killed (worker restarted mid-build) leaves its container and volume behind,
// and the next deploy under the same name would trip over them
runCmd("docker", "rm", "-f", "boottest-"+id)
runCmd("docker", "volume", "rm", "-f", "boottest-"+id+"-data")
payloadCmd := exec.Command(self, "boottest-payload", "-games", gamesDir, id)
var stderr bytes.Buffer
payloadCmd.Stderr = &stderr
payload, err := payloadCmd.Output()
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) && exitErr.ExitCode() == exitMissingSecret {
// The builder has no CI secrets: that's the template's problem to fix, not a reason to stop
return []string{"the boot test can't run: " + strings.TrimSpace(stderr.String()) + ". Remove 'secrets' from " +
"test.yaml; boot tests fill password settings with a throwaway value"}, nil, nil
}
return nil, nil, fmt.Errorf("boottest-payload: %v: %s", err, strings.TrimSpace(stderr.String()))
}
if out, err := runCmd("docker", "pull", "-q", agentImage); err != nil {
return nil, nil, fmt.Errorf("docker pull %s: %v: %s", agentImage, err, out)
}
cmd := exec.Command("docker", "run", "--rm", "-i", "-v", "/var/run/docker.sock:/var/run/docker.sock",
"--entrypoint", "java", agentImage, "-Dloader.main=com.gbreed.gbr.nodeagent.BootTest", "-cp", "app.jar",
"org.springframework.boot.loader.launch.PropertiesLauncher")
cmd.Stdin = bytes.NewReader(payload)
var output bytes.Buffer
cmd.Stdout = &teeWriter{&output}
cmd.Stderr = cmd.Stdout
runErr := cmd.Run()
if runErr != nil {
var exitErr *exec.ExitError
if !errors.As(runErr, &exitErr) {
return nil, nil, runErr
}
return []string{bootFailure(output.String())}, nil, nil
}
verified = append(verified, "boot test with the node-agent: "+bootVerdict(output.String()))
if wantOutside {
fmt.Println("[build] first boot wrote saves outside the volume; booting the draft to see that it no longer does")
r, err := firstBootGame(gamesDir, id)
if err != nil {
return nil, nil, err
}
if dirs := outsideDirs(r); len(dirs) > 0 {
return []string{fmt.Sprintf("booting your template, the server still wrote its own files outside the server volume, "+
"in %s, so your startup command or settings did not move them. Check the exact launch option name and "+
"syntax this server documents for its save/data folder (a wrong option is usually ignored silently) "+
"and point it at /server", strings.Join(dirs, ", "))}, nil, nil
}
verified = append(verified, "the server keeps its files inside /server")
}
return nil, verified, nil
}
func firstBootGame(gamesDir, id string) (*bootReport, error) {
outDir := filepath.Join("firstboot-out", id+"-draft")
if err := runFirstBoot([]string{"-games", gamesDir, "-game", id, "-out", outDir}); err != nil {
return nil, err
}
return readBootReport(filepath.Join(outDir, "report.json"))
}
// bootFailure turns BootTest's output into feedback: its verdict and the server's last lines, which hold
// the actual error (an unknown option, a missing file, a port in use).
func bootFailure(output string) string {
verdict, tail := bootVerdict(output), output
if i := strings.Index(output, "---- last "); i >= 0 {
tail = output[i:]
}
lines := strings.Split(strings.TrimSpace(tail), "\n")
if len(lines) > 60 {
lines = append(lines[:1], lines[len(lines)-59:]...)
}
return "booting your template with the node-agent (the real deploy code) failed: " + verdict +
". Find the cause in the server's output below and fix the template (startup command, settings, " +
"ready pattern, memory):\n" + strings.Join(lines, "\n")
}
func bootVerdict(output string) string {
for _, l := range strings.Split(output, "\n") {
for _, k := range []string{"[boot-test] FAILED: ", "[boot-test] PASSED: "} {
if i := strings.Index(l, k); i >= 0 {
return strings.TrimSpace(l[i+len("[boot-test] "):])
}
}
}
return "no verdict"
}
func firstLine(s string) string {
line, _, _ := strings.Cut(s, "\n")
return line
}
// teeWriter shows the boot as it happens (the worker streams it to the panel) while keeping it for the verdict.
type teeWriter struct{ buf *bytes.Buffer }
func (t *teeWriter) Write(p []byte) (int, error) {
os.Stdout.Write(p)
return t.buf.Write(p)
}
// buildSummary is the pull request's description: where the facts came from and what was really checked.
func buildSummary(o buildOptions, res *buildResult) string {
var b strings.Builder
fmt.Fprintf(&b, "Drafted by the GBR AI builder. **Review before merging**: check the settings, "+
"ports and descriptions against the game's documentation.\n\n")
fmt.Fprintf(&b, "- Game: **%s**", o.name)
if o.variant.Title != "" {
fmt.Fprintf(&b, " — %s", o.variant.Title)
}
fmt.Fprintf(&b, "\n- Image: `%s`", res.Image)
if o.variant.SteamAppID != "" && isSteamCMDImage(res.Image) {
fmt.Fprintf(&b, ", Steam app %s", o.variant.SteamAppID)
}
fmt.Fprintf(&b, "\n- Model rounds: %d\n", res.Rounds)
if len(res.Verified) > 0 {
b.WriteString("\n**Checked for real**\n")
for _, v := range res.Verified {
fmt.Fprintf(&b, "- %s\n", v)
}
}
if len(res.Problems) > 0 {
b.WriteString("\n**Still open** (the builder could not fix these)\n")
for _, p := range res.Problems {
fmt.Fprintf(&b, "- %s\n", firstLine(p))
}
}
if o.variant.Notes != "" {
fmt.Fprintf(&b, "\n**Notes from research:** %s\n", o.variant.Notes)
}
if len(o.variant.Sources) > 0 {
b.WriteString("\n**Sources**\n")
for _, s := range o.variant.Sources {
fmt.Fprintf(&b, "- %s\n", s)
}
}
return b.String()
}
// sourceText reads the pages research based the variant on, in full, so the drafter can check that every
// launch option the model uses is documented there. Empty (check skipped) without a reader.
func sourceText(sources []string) string {
reader := os.Getenv("READER_URL")
if reader == "" || len(sources) == 0 {
fmt.Println("[build] no research sources to check launch options against (READER_URL or sources missing)")
return ""
}
var b strings.Builder
read := 0
for _, s := range sources {
text, err := readPageLimit(reader, s, 200_000)
if err != nil {
fmt.Printf("[build] could not read %s: %v\n", s, err)
continue
}
read++
b.WriteString(text)
b.WriteString("\n")
}
fmt.Printf("[build] read %d of %d research sources to check launch options against\n", read, len(sources))
return b.String()
}
+50
View File
@@ -0,0 +1,50 @@
package main
import (
"strings"
"testing"
)
func TestBootFailureKeepsVerdictAndServerOutput(t *testing.T) {
out := "[boot-test] deploying boottest-example (cm2network/steamcmd)\n" +
"[boot-test] container started; waiting for ready pattern /Server ready/\n" +
"[boot-test] FAILED: container exited before it was ready\n" +
"---- last 2 log lines ----\n" +
"Unknown argument --save-dir\n" +
"Usage: example_server ...\n"
got := bootFailure(out)
if !strings.Contains(got, "failed: FAILED: container exited before it was ready") {
t.Errorf("verdict missing: %s", got)
}
if !strings.Contains(got, "Unknown argument --save-dir") || strings.Contains(got, "deploying boottest") {
t.Errorf("want only the server's last lines, got: %s", got)
}
if v := bootVerdict("[boot-test] PASSED: READY: Server ready"); v != "PASSED: READY: Server ready" {
t.Errorf("verdict = %q", v)
}
}
func TestOutsideDirsSkipsSteamNoise(t *testing.T) {
r := &bootReport{ContainerChanges: []string{
"A /home/steam/.config/example-game/worlds/w.db",
"A /home/steam/Steam/logs/connection_log.txt",
"C /home/steam/.steam/registry.vdf",
}}
got := outsideDirs(r)
if len(got) != 1 || got[0] != "/home/steam/.config/example-game/worlds" {
t.Errorf("got %v", got)
}
if outsideDirs(nil) != nil {
t.Error("nil report should have no dirs")
}
}
func TestVariantNotes(t *testing.T) {
v := variant{Title: "Crossplay", Description: "PC + Xbox", StartupHint: "./example_server -crossplay", Ports: []string{"30000/udp"}}
got := variantNotes(v, "needs Xbox")
for _, want := range []string{"variant: Crossplay (PC + Xbox)", "documented startup: ./example_server -crossplay", "30000/udp", "the admin asked for: needs Xbox"} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in %q", want, got)
}
}
}
+222 -118
View File
@@ -13,9 +13,10 @@ import (
"path" "path"
"path/filepath" "path/filepath"
"regexp" "regexp"
"sort"
"strings" "strings"
"time" "time"
"github.com/santhosh-tekuri/jsonschema/v6"
) )
// draft writes a first template for a new game with an AI model: it gives the model AUTHORING.md as its // draft writes a first template for a new game with an AI model: it gives the model AUTHORING.md as its
@@ -37,7 +38,7 @@ func runDraft(args []string) error {
fs := flag.NewFlagSet("draft", flag.ExitOnError) fs := flag.NewFlagSet("draft", flag.ExitOnError)
gamesDir := fs.String("games", "games", "directory holding one folder per game") gamesDir := fs.String("games", "games", "directory holding one folder per game")
schemaDir := fs.String("schemas", "schema", "directory holding the JSON Schemas") schemaDir := fs.String("schemas", "schema", "directory holding the JSON Schemas")
gameName := fs.String("name", "", "the game's name, e.g. \"Valheim\"") gameName := fs.String("name", "", "the game's name, as players know it")
id := fs.String("id", "", "folder/id for the new game (default: from -name)") id := fs.String("id", "", "folder/id for the new game (default: from -name)")
image := fs.String("image", "", "Docker image the server runs in") image := fs.String("image", "", "Docker image the server runs in")
steamApp := fs.String("steam-app", "", "SteamCMD app id, if the server is installed with SteamCMD") steamApp := fs.String("steam-app", "", "SteamCMD app id, if the server is installed with SteamCMD")
@@ -52,102 +53,171 @@ func runDraft(args []string) error {
if *gameName == "" || *image == "" { if *gameName == "" || *image == "" {
return errors.New("draft needs -name and -image") return errors.New("draft needs -name and -image")
} }
if *id == "" { var report *bootReport
*id = strings.Trim(regexp.MustCompile(`[^a-z0-9]+`).ReplaceAllString(strings.ToLower(*gameName), "-"), "-") if *reportPath != "" {
var err error
if report, err = readBootReport(*reportPath); err != nil {
return err
} }
dir := filepath.Join(*gamesDir, *id)
if _, err := os.Stat(dir); err == nil {
return fmt.Errorf("%s already exists; drafts only create new games", dir)
} }
d, err := newDrafter(draftJob{GamesDir: *gamesDir, SchemaDir: *schemaDir, Name: *gameName, ID: *id, Image: *image,
m, err := newModel(*provider, *modelName) SteamApp: *steamApp, Notes: *notes, Report: report, Provider: *provider, Model: *modelName, NoPull: *noPull})
if err != nil { if err != nil {
return err return err
} }
if problems, err := d.refine(*attempts, nil); err != nil {
return err
} else if len(problems) > 0 {
return fmt.Errorf("draft still has problems after %d fix rounds; it is in %s for a person to finish", *attempts, d.dir)
}
fmt.Println("[draft] next: review it, then boot-test it (build / CI) before opening a PR")
return nil
}
type draftJob struct {
GamesDir, SchemaDir string
Name, ID, Image string
SteamApp, Notes string
Report *bootReport
Provider, Model string
NoPull bool
// Text of the pages research based the variant on; when set, every launch option in the startup
// command must appear in it (or in the image's scripts)
Docs string
}
// drafter holds one conversation with the model about one game, so later problems (a failed boot) can
// be sent back to it with everything it was told before.
type drafter struct {
job draftJob
dir, id string
m model
facts *imageFacts
gameSchema, testSchema *jsonschema.Schema
answerSchema map[string]any
messages []chatMessage
rounds int
}
func newDrafter(j draftJob) (*drafter, error) {
id := j.ID
if id == "" {
id = gameID(j.Name)
}
d := &drafter{job: j, id: id, dir: filepath.Join(j.GamesDir, id)}
if _, err := os.Stat(d.dir); err == nil {
return nil, fmt.Errorf("%s already exists; drafts only create new games", d.dir)
}
var err error
if d.m, err = newModel(j.Provider, j.Model); err != nil {
return nil, err
}
rules, err := os.ReadFile("AUTHORING.md") rules, err := os.ReadFile("AUTHORING.md")
if err != nil { if err != nil {
return fmt.Errorf("AUTHORING.md (the model's rules): %w", err) return nil, fmt.Errorf("AUTHORING.md (the model's rules): %w", err)
} }
gameSchema, err := compileSchema(filepath.Join(*schemaDir, "game.schema.json")) if d.gameSchema, err = compileSchema(filepath.Join(j.SchemaDir, "game.schema.json")); err != nil {
if err != nil { return nil, err
return err
} }
testSchema, err := compileSchema(filepath.Join(*schemaDir, "test.schema.json")) if d.testSchema, err = compileSchema(filepath.Join(j.SchemaDir, "test.schema.json")); err != nil {
if err != nil { return nil, err
return err
} }
answerSchema, err := draftAnswerSchema(*schemaDir) if d.answerSchema, err = draftAnswerSchema(j.SchemaDir); err != nil {
if err != nil { return nil, err
return err
} }
if !j.NoPull {
if !*noPull { fmt.Printf("[draft] pulling %s\n", j.Image)
fmt.Printf("[draft] pulling %s\n", *image) if out, err := runCmd("docker", "pull", "-q", j.Image); err != nil {
if out, err := runCmd("docker", "pull", "-q", *image); err != nil { return nil, fmt.Errorf("docker pull: %v: %s", err, out)
return fmt.Errorf("docker pull: %v: %s", err, out)
} }
} }
facts, err := inspectImage(*image, nil) if d.facts, err = inspectImage(j.Image, nil); err != nil {
if err != nil { return nil, err
return err
} }
var report *bootReport if why := imageUnusable(j.Report); why != "" {
if *reportPath != "" { return nil, errors.New(why)
raw, err := os.ReadFile(*reportPath)
if err != nil {
return err
} }
report = &bootReport{} d.messages = []chatMessage{
if err := json.Unmarshal(raw, report); err != nil {
return fmt.Errorf("%s: %w", *reportPath, err)
}
if why := imageUnusable(report); why != "" {
return errors.New(why)
}
}
messages := []chatMessage{
{Role: "system", Content: draftSystemPrompt(string(rules))}, {Role: "system", Content: draftSystemPrompt(string(rules))},
{Role: "user", Content: draftRequest(*gameName, *id, *image, *steamApp, *notes, facts, report, exampleTemplates(*gamesDir, 2))}, {Role: "user", Content: draftRequest(j.Name, id, j.Image, j.SteamApp, j.Notes, d.facts, j.Report)},
}
return d, nil
} }
for round := 0; round <= *attempts; round++ { func gameID(name string) string {
fmt.Printf("[draft] round %d: asking %s\n", round+1, m.name()) return strings.Trim(regexp.MustCompile(`[^a-z0-9]+`).ReplaceAllString(strings.ToLower(name), "-"), "-")
}
// refine asks for a (corrected) template until it passes validation and the image checks, at most
// attempts+1 answers. feedback, when given, is what went wrong since the last answer (e.g. its boot).
// It returns the problems still left, nil when the draft passes.
func (d *drafter) refine(attempts int, feedback []string) ([]string, error) {
if len(feedback) > 0 {
d.ask(feedback)
}
var problems []string
for round := 0; round <= attempts; round++ {
d.rounds++
fmt.Printf("[draft] round %d: asking %s\n", d.rounds, d.m.name())
start := time.Now() start := time.Now()
answer, err := m.chat(messages, answerSchema) answer, err := d.m.chat(d.messages, d.answerSchema)
if err != nil { if err != nil {
return err return nil, err
} }
fmt.Printf("[draft] answer in %.0fs\n", time.Since(start).Seconds()) fmt.Printf("[draft] answer in %.0fs\n", time.Since(start).Seconds())
messages = append(messages, chatMessage{Role: "assistant", Content: answer}) d.messages = append(d.messages, chatMessage{Role: "assistant", Content: answer})
problems := writeDraft(dir, *id, answer, report) if problems = d.check(answer); len(problems) == 0 {
if len(problems) == 0 { fmt.Printf("[draft] %s passes validation and the image checks\n", d.dir)
problems = validateGame(dir, *id, gameSchema, testSchema) return nil, nil
}
if len(problems) == 0 {
if g, err := loadMap(filepath.Join(dir, "game.yaml")); err == nil {
imageProblems, _ := checkTemplate(g, facts)
problems = append(problems, imageProblems...)
t, _ := loadMap(filepath.Join(dir, "test.yaml"))
problems = append(problems, draftCompleteness(g, t)...)
problems = append(problems, unreadSettings(g, facts)...)
problems = append(problems, savesOutsideVolumes(g, report)...)
}
}
if len(problems) == 0 {
fmt.Printf("[draft] %s passes validation and the image checks\n", dir)
fmt.Println("[draft] next: review it, then boot-test it (first-boot / CI) before opening a PR")
return nil
} }
fmt.Printf("[draft] %d problem(s):\n", len(problems)) fmt.Printf("[draft] %d problem(s):\n", len(problems))
for _, p := range problems { for _, p := range problems {
fmt.Println(" - " + p) fmt.Println(" - " + p)
} }
messages = append(messages, chatMessage{Role: "user", Content: "Your template has these problems. Fix all of them and " + if round < attempts {
d.ask(problems)
}
}
return problems, nil
}
func (d *drafter) ask(problems []string) {
d.messages = append(d.messages, chatMessage{Role: "user", Content: "Your template has these problems. Fix all of them and " +
"answer with the complete corrected JSON object (both \"game\" and \"test\"):\n- " + strings.Join(problems, "\n- ")}) "answer with the complete corrected JSON object (both \"game\" and \"test\"):\n- " + strings.Join(problems, "\n- ")})
} }
return fmt.Errorf("draft still has problems after %d fix rounds; it is in %s for a person to finish", *attempts, dir)
func (d *drafter) check(answer string) []string {
problems := writeDraft(d.dir, d.id, answer, d.job.Report)
if len(problems) == 0 {
problems = validateGame(d.dir, d.id, d.gameSchema, d.testSchema)
}
if len(problems) == 0 {
if g, err := loadMap(filepath.Join(d.dir, "game.yaml")); err == nil {
imageProblems, _ := checkTemplate(g, d.facts)
problems = append(problems, imageProblems...)
t, _ := loadMap(filepath.Join(d.dir, "test.yaml"))
problems = append(problems, draftCompleteness(g, t)...)
problems = append(problems, unreadSettings(g, d.facts)...)
problems = append(problems, savesOutsideVolumes(g, d.job.Report)...)
problems = append(problems, splitSettings(g)...)
problems = append(problems, portSettings(g)...)
problems = append(problems, undocumentedOptions(g, d.job.Docs, d.facts)...)
}
}
return problems
}
func readBootReport(p string) (*bootReport, error) {
raw, err := os.ReadFile(p)
if err != nil {
return nil, err
}
r := &bootReport{}
if err := json.Unmarshal(raw, r); err != nil {
return nil, fmt.Errorf("%s: %w", p, err)
}
return r, nil
} }
func draftSystemPrompt(rules string) string { func draftSystemPrompt(rules string) string {
@@ -203,6 +273,11 @@ func draftCompleteness(game, test map[string]any) []string {
} else if v, _ := n.Int64(); v < 1024 { } else if v, _ := n.Int64(); v < 1024 {
problems = append(problems, "requiredMemoryMb is below 1024") problems = append(problems, "requiredMemoryMb is below 1024")
} }
// Secrets only exist as CI settings a person adds; a draft with one can't be boot-tested anywhere
if secrets, _ := test["secrets"].(map[string]any); len(secrets) > 0 {
problems = append(problems, "test.yaml has 'secrets' ("+strings.Join(sortedKeys(secrets), ", ")+"), which no boot test of a "+
"new game has: remove them. Password settings get a throwaway value in boot tests automatically")
}
if _, ok := game["readyPattern"]; !ok { if _, ok := game["readyPattern"]; !ok {
if _, ok := test["settleSec"]; !ok { if _, ok := test["settleSec"]; !ok {
problems = append(problems, "no readyPattern and no test.settleSec: boot tests need one of them") problems = append(problems, "no readyPattern and no test.settleSec: boot tests need one of them")
@@ -265,26 +340,11 @@ func closestVar(name string, vars []string) string {
var outsideNoise = []string{"/Steam/logs/", "/.steam/", "/steamcmd/", "/.cache/", "/tmp/", "/.local/share/umu"} var outsideNoise = []string{"/Steam/logs/", "/.steam/", "/steamcmd/", "/.cache/", "/tmp/", "/.local/share/umu"}
// savesOutsideVolumes turns first boot's "written outside the volumes" into a hard check: anything the // savesOutsideVolumes turns first boot's "written outside the volumes" into a hard check: anything the
// game itself wrote there (Valheim's worlds and admin lists in ~/.config/unity3d/...) is lost every time the // game itself wrote there (worlds or player lists under the user's home folder) is lost every time the
// node recreates the container. A template passes once its startup command or settings point the game at // node recreates the container. A template passes once its startup command or settings point the game at
// /server or /data; a boot test then shows whether that really worked. // /server or /data; a boot test then shows whether that really worked.
func savesOutsideVolumes(game map[string]any, report *bootReport) []string { func savesOutsideVolumes(game map[string]any, report *bootReport) []string {
if report == nil { dirs := outsideDirs(report)
return nil
}
dirs := map[string]bool{}
for _, c := range report.ContainerChanges {
_, p, _ := strings.Cut(c, " ")
noise := false
for _, n := range outsideNoise {
if strings.Contains(p, n) {
noise = true
}
}
if !noise && p != "" {
dirs[path.Dir(p)] = true
}
}
if len(dirs) == 0 { if len(dirs) == 0 {
return nil return nil
} }
@@ -302,7 +362,28 @@ func savesOutsideVolumes(game map[string]any, report *bootReport) []string {
return []string{fmt.Sprintf("first boot: the server wrote its own files outside the server volume, in %s. Those are lost "+ return []string{fmt.Sprintf("first boot: the server wrote its own files outside the server volume, in %s. Those are lost "+
"whenever the node recreates the container (every start). Make the game keep them under /server, e.g. with "+ "whenever the node recreates the container (every start). Make the game keep them under /server, e.g. with "+
"its save-directory launch option or by pointing that folder at /server, and include it in backupPaths", "its save-directory launch option or by pointing that folder at /server, and include it in backupPaths",
strings.Join(sortedKeysBool(dirs), ", "))} strings.Join(dirs, ", "))}
}
// outsideDirs lists the folders outside the volumes where the game itself wrote files during a boot.
func outsideDirs(report *bootReport) []string {
if report == nil {
return nil
}
dirs := map[string]bool{}
for _, c := range report.ContainerChanges {
_, p, _ := strings.Cut(c, " ")
noise := false
for _, n := range outsideNoise {
if strings.Contains(p, n) {
noise = true
}
}
if !noise && p != "" {
dirs[path.Dir(p)] = true
}
}
return sortedKeysBool(dirs)
} }
// imageUnusable recognises a first boot where the image died within seconds because it needs root (or // imageUnusable recognises a first boot where the image died within seconds because it needs root (or
@@ -325,7 +406,7 @@ func imageUnusable(r *bootReport) string {
*r.ExitCode, r.RanSeconds, r.User, denied[0]) *r.ExitCode, r.RanSeconds, r.User, denied[0])
} }
func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts, report *bootReport, examples string) string { func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts, report *bootReport) string {
var b strings.Builder var b strings.Builder
fmt.Fprintf(&b, "Write the template for **%s**.\n- id: %s\n- defaultDockerImage: %s\n", gameName, id, image) fmt.Fprintf(&b, "Write the template for **%s**.\n- id: %s\n- defaultDockerImage: %s\n", gameName, id, image)
if steamApp != "" { if steamApp != "" {
@@ -369,44 +450,67 @@ func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts
} else { } else {
b.WriteString("\n(No first-boot report: keep readyPattern out unless the facts show the ready line.)\n") b.WriteString("\n(No first-boot report: keep readyPattern out unless the facts show the ready line.)\n")
} }
b.WriteString("\n## Working templates from this repo, for format and style (not content)\n") b.WriteString("\n## Format example\nAn invented server, to show the format and house style only. Its command, options, " +
b.WriteString(examples) "ports and app id belong to no real game: never copy them; take every value from the facts above.\n```yaml\n")
b.WriteString(exampleTemplate)
b.WriteString("```\n")
return b.String() return b.String()
} }
// exampleTemplates picks the shortest templates in the repo as examples: real, validated, and small // exampleTemplate shows the model the format only. It's invented on purpose: the builder must stay
// enough to leave the model room for the facts. // game-agnostic, so the model never sees another game's template (whose options it would copy). Its
func exampleTemplates(gamesDir string, n int) string { // command, options, port and app id belong to no real server; validated in draft_test.go.
entries, _ := os.ReadDir(gamesDir) const exampleTemplate = `id: example-server
type ex struct { name: Example Server
id string defaultDockerImage: cm2network/steamcmd
body string defaultStartupCommand: args=(-name "${SERVER_NAME}" -world "${WORLD_NAME:-world}" -port 30000 -datadir /server/saves); [ -n "${JOIN_PASSWORD}" ] && args+=(-password "${JOIN_PASSWORD}"); ./example_server "${args[@]}"
} requiredMemoryMb: 4096
var all []ex requiredDiskMb: 8192
for _, e := range entries { defaultPorts:
raw, err := os.ReadFile(filepath.Join(gamesDir, e.Name(), "game.yaml")) - 30000/udp
if err == nil { - 30001/udp
all = append(all, ex{e.Name(), string(raw)}) installMethod: STEAMCMD
} installConfig:
} appId: "1000000"
sort.Slice(all, func(i, j int) bool { return len(all[i].body) < len(all[j].body) }) defaultConfig:
var b strings.Builder JoinPassword: ""
for i := 0; i < n && i < len(all); i++ { ServerName: GBRHosting - {SERVER_NAME}
fmt.Fprintf(&b, "\n### games/%s/game.yaml\n```yaml\n%s```\n", all[i].id, all[i].body) WorldName: world
} configSchema:
return b.String() type: object
} properties:
JoinPassword:
title: Join Password
description: Password players enter to join. Leave empty for a public server.
type: string
default: ""
category: Security & Access
x-savedValue:
key: GAME_JOIN_PASSWORD
label: Game join password
secret: true
ServerName:
title: Server Name
description: The name shown in the server browser.
type: string
default: GBRHosting - {SERVER_NAME}
category: System & General
showOnDeploy: true
WorldName:
title: World Name
description: The world to load. A new name starts a new world.
type: string
default: world
category: World & Environment
backupPaths:
- saves
readyPattern: 'Server is ready for connections'
`
// writeDraft turns the model's JSON answer into games/<id>/game.yaml and test.yaml, using the importer's // writeDraft turns the model's JSON answer into games/<id>/game.yaml and test.yaml, using the importer's
// writer so drafts look like every other template. // writer so drafts look like every other template.
func writeDraft(dir, id, answer string, report *bootReport) []string { func writeDraft(dir, id, answer string, report *bootReport) []string {
answer = strings.TrimSpace(answer) answer = trimToJSON(answer)
if i := strings.Index(answer, "{"); i > 0 {
answer = answer[i:] // some models still wrap JSON in prose or code fences
}
if i := strings.LastIndex(answer, "}"); i >= 0 {
answer = answer[:i+1]
}
var parsed struct { var parsed struct {
Game map[string]any `json:"game"` Game map[string]any `json:"game"`
Test map[string]any `json:"test"` Test map[string]any `json:"test"`
+60 -16
View File
@@ -2,14 +2,16 @@ package main
import ( import (
"encoding/json" "encoding/json"
"os"
"path/filepath"
"strings" "strings"
"testing" "testing"
) )
func TestDraftCompletenessNamesMissingSchemaEntries(t *testing.T) { func TestDraftCompletenessNamesMissingSchemaEntries(t *testing.T) {
// The Valheim draft that went round in circles: settings in defaultConfig, no configSchema // A draft that went round in circles: settings in defaultConfig, no configSchema
game := map[string]any{ game := map[string]any{
"defaultPorts": []any{"2456/udp"}, "defaultPorts": []any{"30000/udp"},
"requiredMemoryMb": json.Number("4096"), "requiredMemoryMb": json.Number("4096"),
"defaultConfig": map[string]any{"ServerName": "x", "ServerPassword": "", "MaxPlayers": json.Number("8")}, "defaultConfig": map[string]any{"ServerName": "x", "ServerPassword": "", "MaxPlayers": json.Number("8")},
} }
@@ -44,28 +46,28 @@ func TestUnreadSettings(t *testing.T) {
} }
func TestSavesOutsideVolumes(t *testing.T) { func TestSavesOutsideVolumes(t *testing.T) {
// Valheim's real first boot: its own files in ~/.config, plus Steam client logs (noise) // A real first boot: the game's own files in ~/.config, plus Steam client logs (noise)
valheim := &bootReport{ContainerChanges: []string{ home := &bootReport{ContainerChanges: []string{
"A /home/steam/.config/unity3d/IronGate/Valheim/adminlist.txt", "A /home/steam/.config/example-game/adminlist.txt",
"A /home/steam/.config/unity3d/IronGate/Valheim/bannedlist.txt", "A /home/steam/.config/example-game/bannedlist.txt",
"A /home/steam/Steam/logs/connection_log_2456.txt", "A /home/steam/Steam/logs/connection_log_30000.txt",
}} }}
game := map[string]any{"defaultStartupCommand": "./valheim_server.x86_64 -nographics -batchmode -world X"} game := map[string]any{"defaultStartupCommand": "./example_server -nographics -batchmode -world X"}
problems := savesOutsideVolumes(game, valheim) problems := savesOutsideVolumes(game, home)
if len(problems) != 1 || !strings.Contains(problems[0], "/home/steam/.config/unity3d/IronGate/Valheim") || strings.Contains(problems[0], "Steam/logs") { if len(problems) != 1 || !strings.Contains(problems[0], "/home/steam/.config/example-game") || strings.Contains(problems[0], "Steam/logs") {
t.Fatalf("got %v", problems) t.Fatalf("got %v", problems)
} }
// Fixed with a save-directory option under /server // Fixed with a save-directory option under /server
game["defaultStartupCommand"] = "./valheim_server.x86_64 -nographics -batchmode -world X -savedir /server/saves" game["defaultStartupCommand"] = "./example_server -nographics -batchmode -world X -datadir /server/saves"
if p := savesOutsideVolumes(game, valheim); p != nil { if p := savesOutsideVolumes(game, home); p != nil {
t.Errorf("redirected template still flagged: %v", p) t.Errorf("redirected template still flagged: %v", p)
} }
// ARK's real first boot: only Proton's cache, which is not game data // A Proton server's first boot: only Proton's cache, which is not game data
ark := &bootReport{ContainerChanges: []string{"A /home/gameserver/.cache/umu-protonfixes/protonfixes_test.log"}} proton := &bootReport{ContainerChanges: []string{"A /home/gameserver/.cache/umu-protonfixes/protonfixes_test.log"}}
if p := savesOutsideVolumes(map[string]any{}, ark); p != nil { if p := savesOutsideVolumes(map[string]any{}, proton); p != nil {
t.Errorf("ARK proton cache flagged: %v", p) t.Errorf("proton cache flagged: %v", p)
} }
} }
@@ -84,3 +86,45 @@ func TestImageUnusable(t *testing.T) {
t.Errorf("long run flagged: %q", why) t.Errorf("long run flagged: %q", why)
} }
} }
func TestDraftCompletenessRejectsSecrets(t *testing.T) {
// A draft that couldn't be boot-tested: the password mapped to a CI secret nobody created
test := map[string]any{"settleSec": json.Number("120"), "secrets": map[string]any{"Password": "EXAMPLE_TEST_PASSWORD"}}
problems := strings.Join(draftCompleteness(map[string]any{}, test), "\n")
if !strings.Contains(problems, "test.yaml has 'secrets' (Password)") {
t.Errorf("got:\n%s", problems)
}
}
// The model copies the example's format, so it must pass everything a draft has to pass
func TestExampleTemplateFollowsTheRules(t *testing.T) {
dir := filepath.Join(t.TempDir(), "example-server")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "game.yaml"), []byte(exampleTemplate), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "test.yaml"), []byte("timeoutSec: 2400\n"), 0o644); err != nil {
t.Fatal(err)
}
gs, err := compileSchema(filepath.Join("..", "..", "schema", "game.schema.json"))
if err != nil {
t.Fatal(err)
}
ts, err := compileSchema(filepath.Join("..", "..", "schema", "test.schema.json"))
if err != nil {
t.Fatal(err)
}
problems := validateGame(dir, "example-server", gs, ts)
g, err := loadMap(filepath.Join(dir, "game.yaml"))
if err != nil {
t.Fatal(err)
}
problems = append(problems, draftCompleteness(g, map[string]any{})...)
problems = append(problems, splitSettings(g)...)
problems = append(problems, portSettings(g)...)
for _, p := range problems {
t.Error(p)
}
}
+325
View File
@@ -0,0 +1,325 @@
package main
import (
"encoding/json"
"fmt"
"path"
"regexp"
"sort"
"strconv"
"strings"
)
// Checks a booting draft can still fail: things that only go wrong once an owner changes a setting, so
// neither the validator nor a boot test with the defaults sees them.
// splitSettings finds text settings that reach the server's command line unquoted, so a value with spaces
// ("GBRHosting - My Server") arrives as several arguments and the game only sees the first word. Covers
// direct use (-name ${SERVER_NAME}) and the common ARGS="-name ${SERVER_NAME}"; ./server $ARGS, where the
// quotes in the assignment don't survive the unquoted $ARGS.
func splitSettings(game map[string]any) []string {
startup, _ := game["defaultStartupCommand"].(string)
if startup == "" {
return nil
}
textVars := map[string]string{} // env name -> setting key
for key, p := range schemaProperties(game) {
if prop, _ := p.(map[string]any); prop["type"] == "string" {
textVars[envName(key)] = key
}
}
if len(textVars) == 0 {
return nil
}
s := scanShell(unwrapShell(startup))
split := map[string]string{} // setting key -> how it gets split
var reach func(holder string, seen map[string]bool) []string
reach = func(holder string, seen map[string]bool) []string {
if seen[holder] {
return nil
}
seen[holder] = true
var out []string
for _, v := range s.held[holder] {
if _, ok := textVars[v]; ok {
out = append(out, v)
}
out = append(out, reach(v, seen)...)
}
return out
}
for _, e := range s.uses {
if e.quoted {
continue
}
if key, ok := textVars[e.name]; ok {
split[key] = "${" + e.name + "} is not in double quotes"
}
if len(s.held[e.name]) > 0 {
for _, v := range reach(e.name, map[string]bool{}) {
if _, done := split[textVars[v]]; !done {
split[textVars[v]] = fmt.Sprintf("${%s} is inside $%s, which is used without double quotes", v, e.name)
}
}
}
}
var problems []string
for _, key := range sortedKeysString(split) {
problems = append(problems, fmt.Sprintf("setting %s is split into words on the command line (%s): a value with spaces like "+
"\"GBRHosting - My Server\" reaches the game as several arguments. Quote it where it is used (-name \"${%s}\"), or build "+
"optional arguments in a bash array: args=(-name \"$%s\"); [ -n \"$X\" ] && args+=(-x \"$X\"); ./server \"${args[@]}\"",
key, split[key], envName(key), envName(key)))
}
return problems
}
var shellWrap = regexp.MustCompile(`^\s*(?:ba)?sh\s+-c\s+'(.*)'\s*$`)
// unwrapShell looks inside a startup command that is one bash -c '...', which would otherwise be
// all single-quoted text.
func unwrapShell(cmd string) string {
if m := shellWrap.FindStringSubmatch(cmd); m != nil && !strings.Contains(m[1], "'") {
return m[1]
}
return cmd
}
type shellUse struct {
name string
quoted bool
}
type shellScan struct {
uses []shellUse // expansions where the value is used: words, arguments
held map[string][]string // variable -> variables its assigned value contains
}
var (
assignWord = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_]*)\+?=`)
varName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*`)
nestedVar = regexp.MustCompile(`\$\{?([A-Za-z_][A-Za-z0-9_]*)`)
)
// scanShell is just enough of a shell parser for splitSettings: quotes, $VAR / ${VAR...} expansions, and
// assignments (NAME=..., NAME+=..., NAME=(...) arrays).
func scanShell(cmd string) *shellScan {
s := &shellScan{held: map[string][]string{}}
var single, double, wordStart = false, false, true
assign, inArray := "", false
record := func(name string, quoted bool) {
switch {
case assign != "" && !inArray:
// NAME="...$X..." : no splitting yet, but $NAME carries X along
s.held[assign] = append(s.held[assign], name)
case inArray && quoted:
s.held[assign] = append(s.held[assign], name) // safe unless the array is expanded unquoted
default:
s.uses = append(s.uses, shellUse{name, quoted})
}
}
for i := 0; i < len(cmd); i++ {
c := cmd[i]
if wordStart && !single && !double {
if m := assignWord.FindString(cmd[i:]); m != "" {
assign = strings.TrimRight(strings.TrimSuffix(m, "="), "+")
i += len(m)
if i < len(cmd) && cmd[i] == '(' {
inArray = true
} else {
i--
}
wordStart = false
continue
}
}
wordStart = false
switch {
case c == '\\' && !single:
i++
case c == '\'' && !double:
single = !single
case c == '"' && !single:
double = !double
case c == '$' && !single && i+1 < len(cmd):
if cmd[i+1] == '{' {
end := matchingBrace(cmd, i+1)
body := cmd[i+2 : end]
name := varName.FindString(strings.TrimLeft(body, "#!"))
if name != "" {
record(name, double)
}
// Defaults can hold more expansions: ${A:-${B}}
for _, m := range nestedVar.FindAllStringSubmatch(body[len(name):], -1) {
record(m[1], double)
}
i = end
} else if name := varName.FindString(cmd[i+1:]); name != "" {
record(name, double)
i += len(name)
}
case single || double:
case inArray && c == ')':
inArray, assign = false, ""
case inArray:
case c == ' ' || c == '\t' || c == '\n' || c == ';' || c == '&' || c == '|' || c == '(' || c == ')':
assign, wordStart = "", true
}
}
return s
}
func matchingBrace(s string, open int) int {
depth := 0
for i := open; i < len(s); i++ {
switch s[i] {
case '{':
depth++
case '}':
depth--
if depth == 0 {
return i
}
}
}
return len(s) - 1
}
var (
portSetting = regexp.MustCompile(`(?i)(port$|(^|_)port(_|$))`)
notAPort = regexp.MustCompile(`(?i)(sup|re|ex|im|trans)port$`)
// Reached over the Docker network, never published (AUTHORING.md)
internalPort = regexp.MustCompile(`(?i)rcon|rest|api|admin|telnet|web|http|metrics`)
)
// portSettings checks settings that set a port players connect to against defaultPorts. The node always
// publishes defaultPorts' container ports (moving only the host side when servers share a node), so a
// game listening anywhere else can't be reached: a default that isn't published is broken from the
// start, and an editable one breaks the moment an owner changes it.
func portSettings(game map[string]any) []string {
published := map[int]bool{}
ports, _ := game["defaultPorts"].([]any)
for _, p := range ports {
n, _ := strconv.Atoi(strings.Split(containerPort(fmt.Sprint(p)), "/")[0])
published[n] = true
}
defaults, _ := game["defaultConfig"].(map[string]any)
props := schemaProperties(game)
var problems []string
keys := make([]string, 0, len(defaults))
for k := range defaults {
keys = append(keys, k)
}
sort.Strings(keys)
for _, key := range keys {
if !portSetting.MatchString(key) || notAPort.MatchString(key) || internalPort.MatchString(key) {
continue
}
port, ok := portNumber(defaults[key])
if !ok {
continue // {PUBLIC_PORT_n} and other placeholders are filled per server
}
_, editable := props[key]
switch {
case !published[port]:
problems = append(problems, fmt.Sprintf("setting %s defaults to port %d, which defaultPorts does not publish: players can't reach it. "+
"Use one of the published ports, or add this port to defaultPorts if players connect to it", key, port))
case editable:
problems = append(problems, fmt.Sprintf("setting %s lets owners change the port, but the node always publishes container port %d "+
"(defaultPorts), so any other value makes the server unreachable. Don't make it a setting: remove %s from "+
"defaultConfig and configSchema and write %d directly in the startup command or config file. If the game must "+
"announce the port players use, give it {PUBLIC_PORT_%d} instead", key, port, key, port, port))
}
}
return problems
}
func portNumber(v any) (int, bool) {
var s string
switch n := v.(type) {
case json.Number:
s = n.String()
case string:
s = n
case int, int64, float64:
s = fmt.Sprint(n)
default:
return 0, false
}
p, err := strconv.Atoi(strings.TrimSpace(s))
return p, err == nil && p > 0 && p < 65536
}
var (
optionToken = regexp.MustCompile(`(?:^|[\s("'])(-{1,2}[A-Za-z][A-Za-z0-9_.]*(?:-[A-Za-z0-9_.]+)*)`)
segmentSep = regexp.MustCompile(`;|&&|\|\||\||\n`)
testOps = map[string]bool{"eq": true, "ne": true, "lt": true, "gt": true, "le": true, "ge": true, "nt": true, "ot": true, "ef": true}
// Options of these are the shell's and its tools', not the game's
shellTools = map[string]bool{"mkdir": true, "cp": true, "mv": true, "ln": true, "rm": true, "chmod": true, "chown": true,
"wget": true, "curl": true, "tar": true, "unzip": true, "test": true, "[": true, "[[": true, "echo": true, "printf": true,
"sleep": true, "find": true, "sed": true, "grep": true, "cd": true, "export": true, "touch": true, "bash": true, "sh": true,
"cat": true, "head": true, "tail": true, "ls": true, "set": true, "ulimit": true, "env": true, "id": true}
)
// undocumentedOptions finds launch options in the startup command that none of the researched pages (nor
// the image's own scripts) mention. Servers silently ignore options they don't know, so a made-up one
// boots fine and passes every boot test while its setting does nothing.
func undocumentedOptions(game map[string]any, docs string, facts *imageFacts) []string {
if strings.TrimSpace(docs) == "" {
return nil
}
startup, _ := game["defaultStartupCommand"].(string)
corpus := docs
if facts != nil {
for _, s := range facts.Scripts {
corpus += "\n" + s
}
}
seen := map[string]bool{}
var missing []string
for _, opt := range launchOptions(unwrapShell(startup)) {
name := strings.TrimLeft(opt, "-")
key := strings.ToLower(name)
if seen[key] {
continue
}
seen[key] = true
documented := regexp.MustCompile(`(?i)(^|[^A-Za-z0-9_])-{1,2}` + regexp.QuoteMeta(name) + `($|[^A-Za-z0-9_])`)
if !documented.MatchString(corpus) {
missing = append(missing, opt)
}
}
if len(missing) == 0 {
return nil
}
return []string{fmt.Sprintf("the startup command passes %s, which none of the pages researched for this game mention. "+
"Servers silently ignore options they don't know, so it boots fine but does nothing: remove it (and any setting that "+
"only feeds it), or spell it exactly as the documentation does", strings.Join(missing, ", "))}
}
// launchOptions lists the -options passed to the game, skipping the shell's own (mkdir -p, [ -n ... ]).
func launchOptions(cmd string) []string {
var out []string
for _, seg := range segmentSep.Split(cmd, -1) {
fields := strings.Fields(strings.TrimLeft(strings.TrimSpace(seg), "({"))
i := 0
for i < len(fields) && (fields[i] == "exec" || fields[i] == "then" || fields[i] == "do" || fields[i] == "else" ||
(assignWord.MatchString(fields[i]) && !strings.Contains(fields[i], "=("))) {
i++
}
if i == len(fields) {
continue
}
if word := path.Base(fields[i]); shellTools[word] {
continue
}
for _, m := range optionToken.FindAllStringSubmatch(strings.Join(fields[i:], " "), -1) {
name := strings.TrimLeft(m[1], "-")
if len(name) < 2 || testOps[name] {
continue
}
out = append(out, m[1])
}
}
return out
}
+151
View File
@@ -0,0 +1,151 @@
package main
import (
"encoding/json"
"strings"
"testing"
)
func textSettings(keys ...string) map[string]any {
props := map[string]any{}
for _, k := range keys {
props[k] = map[string]any{"type": "string"}
}
return map[string]any{"properties": props}
}
func TestSplitSettingsCatchesAnArgsString(t *testing.T) {
// A real draft: quoted in the assignment, split by the unquoted $ARGS
game := map[string]any{
"defaultStartupCommand": `ARGS="-name ${SERVER_NAME:-Example Server} -port 30000 -world ${WORLD_NAME:-Dedicated}"; ` +
`[ -n "$PASSWORD" ] && ARGS="$ARGS -password $PASSWORD"; [ "$CROSSPLAY" = "true" ] && ARGS="$ARGS -crossplay"; ` +
`HOME=/server ./example_server $ARGS`,
"configSchema": textSettings("ServerName", "WorldName", "Password"),
}
problems := splitSettings(game)
got := strings.Join(problems, "\n")
if len(problems) != 3 {
t.Fatalf("want ServerName, WorldName and Password, got:\n%s", got)
}
for _, key := range []string{"setting Password", "setting ServerName", "setting WorldName", "inside $ARGS"} {
if !strings.Contains(got, key) {
t.Errorf("missing %q in:\n%s", key, got)
}
}
}
func TestSplitSettingsAcceptsQuotedAndArrays(t *testing.T) {
for _, startup := range []string{
`./example_server -name "${SERVER_NAME}" -world "$WORLD_NAME" -port 30000`,
`args=(-name "$SERVER_NAME" -world "${WORLD_NAME:-Dedicated}"); [ -n "$PASSWORD" ] && args+=(-password "$PASSWORD"); ./example_server "${args[@]}"`,
`bash -c 'exec ./server -name "$SERVER_NAME"'`,
} {
game := map[string]any{"defaultStartupCommand": startup, "configSchema": textSettings("ServerName", "WorldName", "Password")}
if p := splitSettings(game); len(p) != 0 {
t.Errorf("%s\n flagged: %v", startup, p)
}
}
}
func TestSplitSettingsDirectAndArrayMisuse(t *testing.T) {
cases := map[string]string{
`./server -name ${SERVER_NAME}`: "is not in double quotes",
`bash -c 'exec ./server -name $SERVER_NAME'`: "is not in double quotes",
`args=(-name "$SERVER_NAME"); ./server ${args[@]}`: "inside $args",
`args=(-name $SERVER_NAME); ./server "${args[@]}"`: "is not in double quotes",
`export OPTS="-name ${SERVER_NAME}" && ./run $OPTS`: "inside $OPTS",
}
for startup, want := range cases {
game := map[string]any{"defaultStartupCommand": startup, "configSchema": textSettings("ServerName")}
got := strings.Join(splitSettings(game), "\n")
if !strings.Contains(got, want) {
t.Errorf("%s\n want %q, got %q", startup, want, got)
}
}
// Numbers and booleans can't contain spaces
game := map[string]any{"defaultStartupCommand": `./server -port ${GAME_PORT}`, "configSchema": map[string]any{
"properties": map[string]any{"GamePort": map[string]any{"type": "integer"}}}}
if p := splitSettings(game); len(p) != 0 {
t.Errorf("integer setting flagged: %v", p)
}
}
func TestPortSettings(t *testing.T) {
game := map[string]any{
"defaultPorts": []any{"30000/udp", "30001/udp"},
"defaultConfig": map[string]any{
"GamePort": json.Number("30000"),
"QueryPort": json.Number("27015"),
"RCONPort": json.Number("25575"),
"PublicPort": "{PUBLIC_PORT_30000}",
"Transport": "udp",
"Export": json.Number("9"),
},
"configSchema": map[string]any{"properties": map[string]any{
"GamePort": map[string]any{"type": "integer"}, "QueryPort": map[string]any{"type": "integer"},
}},
}
got := strings.Join(portSettings(game), "\n")
if !strings.Contains(got, "setting GamePort lets owners change the port") || !strings.Contains(got, "write 30000 directly") {
t.Errorf("editable published port not caught:\n%s", got)
}
if !strings.Contains(got, "setting QueryPort defaults to port 27015, which defaultPorts does not publish") {
t.Errorf("unpublished port not caught:\n%s", got)
}
for _, key := range []string{"RCONPort", "PublicPort", "Transport", "Export"} {
if strings.Contains(got, "setting "+key+" ") {
t.Errorf("%s should be left alone:\n%s", key, got)
}
}
// A published port written by the template itself, not offered to owners, is fine
delete(game["configSchema"].(map[string]any)["properties"].(map[string]any), "GamePort")
delete(game["defaultConfig"].(map[string]any), "QueryPort")
if p := portSettings(game); len(p) != 0 {
t.Errorf("got %v", p)
}
}
func TestUndocumentedOptionsCatchesMaxPlayers(t *testing.T) {
// A real draft's startup: every option is documented except -maxplayers
game := map[string]any{"defaultStartupCommand": `args=(-name "${SERVER_NAME:-GBRHosting - Example}" -port 30000 -world "${WORLD_NAME:-Dedicated}" -datadir /server); ` +
`[ -n "${SERVER_PASSWORD}" ] && args+=(-password "${SERVER_PASSWORD}"); [ "${CROSSPLAY}" = "true" ] && args+=(-crossplay); ` +
`[ -n "${MAX_PLAYERS}" ] && args+=(-maxplayers "${MAX_PLAYERS}"); ./example_server "${args[@]}"`}
docs := "Launch with `-name`, `-port`, `-world`, `-password`, `-datadir <path>` and `-crossplay` to allow consoles."
got := undocumentedOptions(game, docs, nil)
if len(got) != 1 || !strings.Contains(got[0], "passes -maxplayers,") {
t.Fatalf("want only -maxplayers flagged, got %v", got)
}
// Once removed, nothing is left to flag
game["defaultStartupCommand"] = strings.Replace(game["defaultStartupCommand"].(string), `[ -n "${MAX_PLAYERS}" ] && args+=(-maxplayers "${MAX_PLAYERS}"); `, "", 1)
if got := undocumentedOptions(game, docs, nil); len(got) != 0 {
t.Errorf("got %v", got)
}
}
func TestUndocumentedOptionsIgnoresShellAndUsesImageScripts(t *testing.T) {
game := map[string]any{"defaultStartupCommand": `mkdir -p /server/.config; cp -f /a /b; [ -n "$X" ] && [ "$Y" -eq 1 ] && ` +
`exec ./server --listen-port=7777 -ServerPlatform=ALL -logs`}
facts := &imageFacts{Scripts: map[string]string{"/start.sh": `exec "$@" -logs`}}
got := strings.Join(undocumentedOptions(game, "Use --listen-port and -ServerPlatform=ALL for crossplay.", facts), "\n")
if got != "" {
t.Errorf("shell options, documented ones and ones the image's script uses must pass, got %q", got)
}
// Without sources the check is off (draft run by hand, or no reader configured)
game["defaultStartupCommand"] = "./server -madeup"
if p := undocumentedOptions(game, "", nil); p != nil {
t.Errorf("got %v", p)
}
if p := undocumentedOptions(game, "unrelated page", nil); len(p) != 1 {
t.Errorf("want -madeup flagged, got %v", p)
}
}
func TestUndocumentedOptionsWholeWords(t *testing.T) {
// "-port" in the docs doesn't document "-portal", and "-port" isn't documented by "-portal"
game := map[string]any{"defaultStartupCommand": "./server -port 1 -portal x"}
got := strings.Join(undocumentedOptions(game, "options: -port", nil), "")
if !strings.Contains(got, "passes -portal,") || strings.Contains(got, "-port,") {
t.Errorf("got %q", got)
}
}
+2 -2
View File
@@ -466,8 +466,8 @@ func (b *booter) copyOut(vol, mount string, files []string, report *bootReport)
return cmd.Wait() return cmd.Wait()
} }
// fileLines reads the copied log files, so ready and error lines that never reach the console (ARK // fileLines reads the copied log files, so ready and error lines that never reach the console (servers
// writes to ShooterGame.log) are found too. // that only log to a file) are found too.
func (b *booter) fileLines(report *bootReport) []logLine { func (b *booter) fileLines(report *bootReport) []logLine {
var out []logLine var out []logLine
for _, f := range report.FilesCreated { for _, f := range report.FilesCreated {
+14 -14
View File
@@ -6,21 +6,21 @@ import (
) )
func TestParseConfigFormats(t *testing.T) { func TestParseConfigFormats(t *testing.T) {
// Palworld's Unreal-style list // An Unreal-style option list
pal := parseConfig("/server/Pal/Saved/Config/LinuxServer/PalWorldSettings.ini", "[/Script/Pal.PalGameWorldSettings]\n"+ list := parseConfig("/server/Game/Saved/Config/LinuxServer/GameSettings.ini", "[/Script/Game.WorldSettings]\n"+
`OptionSettings=(Difficulty=None,ExpRate=1.000000,ServerName="Default Palworld Server",CrossplayPlatforms=(Steam,Xbox,PS5,Mac))`) `OptionSettings=(Difficulty=None,ExpRate=1.000000,ServerName="Default Server",CrossplayPlatforms=(Steam,Xbox,PS5,Mac))`)
want := map[string]string{"Difficulty": "None", "ExpRate": "1.000000", "ServerName": "Default Palworld Server", "CrossplayPlatforms": "(Steam,Xbox,PS5,Mac)"} want := map[string]string{"Difficulty": "None", "ExpRate": "1.000000", "ServerName": "Default Server", "CrossplayPlatforms": "(Steam,Xbox,PS5,Mac)"}
if len(pal) != len(want) { if len(list) != len(want) {
t.Fatalf("palworld: got %d settings: %+v", len(pal), pal) t.Fatalf("option list: got %d settings: %+v", len(list), list)
} }
for _, s := range pal { for _, s := range list {
if s.Section != "OptionSettings" || want[s.Key] != s.Value { if s.Section != "OptionSettings" || want[s.Key] != s.Value {
t.Errorf("palworld setting %+v", s) t.Errorf("option list setting %+v", s)
} }
} }
// Minecraft's server.properties // A server.properties file
mc := parseConfig("/data/server.properties", "#Minecraft server properties\nmax-players=20\nmotd=A Minecraft Server\nlevel-seed=\n") mc := parseConfig("/data/server.properties", "#Server properties\nmax-players=20\nmotd=A Server\nlevel-seed=\n")
if len(mc) != 3 || mc[0].Key != "max-players" || mc[0].Value != "20" || mc[2].Value != "" { if len(mc) != 3 || mc[0].Key != "max-players" || mc[0].Value != "20" || mc[2].Value != "" {
t.Errorf("properties: %+v", mc) t.Errorf("properties: %+v", mc)
} }
@@ -66,7 +66,7 @@ func TestAnalyzeLines(t *testing.T) {
{At: 2, Source: "console", Text: "Failed to load optional module"}, {At: 2, Source: "console", Text: "Failed to load optional module"},
{At: 9, Source: "console", Text: "[09:00:01 INFO]: Done (8.1s)! For help, type \"help\""}, {At: 9, Source: "console", Text: "[09:00:01 INFO]: Done (8.1s)! For help, type \"help\""},
{At: 9, Source: "console", Text: "[09:00:01 INFO]: Thread RCON Listener started"}, {At: 9, Source: "console", Text: "[09:00:01 INFO]: Thread RCON Listener started"},
// After the stop at 200s: the shutdown, as Minecraft's runner prints it // After the stop at 200s: the shutdown, as a wrapper script prints it
{At: 201, Source: "console", Text: "[06:01:39] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started"}, {At: 201, Source: "console", Text: "[06:01:39] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started"},
{At: 202, Source: "console", Text: "2026-10-25T06:01:39.976Z\tINFO\tmc-server-runner\tDone"}, {At: 202, Source: "console", Text: "2026-10-25T06:01:39.976Z\tINFO\tmc-server-runner\tDone"},
}, []logLine{{At: -1, Source: "/server/x.log", Text: "Listening on 0.0.0.0:7777"}}, 200) }, []logLine{{At: -1, Source: "/server/x.log", Text: "Listening on 0.0.0.0:7777"}}, 200)
@@ -91,11 +91,11 @@ func TestAnalyzeLines(t *testing.T) {
func TestConfigCandidates(t *testing.T) { func TestConfigCandidates(t *testing.T) {
for rel, want := range map[string]bool{ for rel, want := range map[string]bool{
"server.properties": true, "server.properties": true,
"Pal/Saved/Config/LinuxServer/x.ini": true, "Game/Saved/Config/LinuxServer/x.ini": true,
"ShooterGame/Saved/Logs/ShooterGame.log": true, "Game/Saved/Logs/Server.log": true,
"steamapps/appmanifest_123.acf": false, "steamapps/appmanifest_123.acf": false,
"Engine/Config/BaseEngine.ini": false, "Engine/Config/BaseEngine.ini": false,
"plugins/Geyser/config.yml": false, "plugins/SomePlugin/config.yml": false,
"world/level.dat": false, "world/level.dat": false,
} { } {
if got := isConfigCandidate(rel, 1000); got != want { if got := isConfigCandidate(rel, 1000); got != want {
+4 -4
View File
@@ -278,8 +278,8 @@ func parseProbe(f *imageFacts, out string) {
} }
// suggestContainerUser works out whether the image only works as a specific user. Images either declare // suggestContainerUser works out whether the image only works as a specific user. Images either declare
// it (USER) or are built around an account whose home folder nobody else may enter (ARK's gameserver, // it (USER) or are built around an account whose home folder nobody else may enter (e.g. a home folder
// uid 25000, with /home/gameserver at 700). // at mode 700 owned by a uid other than 1000).
func suggestContainerUser(f *imageFacts) { func suggestContainerUser(f *imageFacts) {
if u := strings.TrimSpace(f.User); u != "" && u != "root" && u != "0" { if u := strings.TrimSpace(f.User); u != "" && u != "root" && u != "0" {
resolved := resolveUser(f, u) resolved := resolveUser(f, u)
@@ -364,7 +364,7 @@ func checkTemplate(game map[string]any, f *imageFacts) (problems, notes []string
} }
if cmd, _ := game["defaultStartupCommand"].(string); cmd != "" { if cmd, _ := game["defaultStartupCommand"].(string); cmd != "" {
for _, m := range assignedVars.FindAllStringSubmatch(cmd, -1) { for _, m := range assignedVars.FindAllStringSubmatch(cmd, -1) {
provided[m[1]] = true // set by the command itself, e.g. export ASA_START_PARAMS=... provided[m[1]] = true // set by the command itself, e.g. export START_PARAMS=...
} }
withDefault := map[string]bool{} withDefault := map[string]bool{}
for _, m := range defaultedRef.FindAllStringSubmatch(cmd, -1) { for _, m := range defaultedRef.FindAllStringSubmatch(cmd, -1) {
@@ -410,7 +410,7 @@ func containerPort(p string) string {
return p return p
} }
// startupScriptPaths picks absolute paths out of a template's startup command (e.g. ARK's // startupScriptPaths picks absolute paths out of a template's startup command (e.g. an image's
// /usr/bin/start_server) so the probe reads those scripts too. // /usr/bin/start_server) so the probe reads those scripts too.
func startupScriptPaths(game map[string]any) []string { func startupScriptPaths(game map[string]any) []string {
cmd, _ := game["defaultStartupCommand"].(string) cmd, _ := game["defaultStartupCommand"].(string)
+8 -8
View File
@@ -21,7 +21,7 @@ func TestEnvNameMatchesNodeAgent(t *testing.T) {
} }
} }
// Probe output as ARK's image produces it (trimmed) // Probe output of an image built around its own user (trimmed)
const arkProbe = probeMarker + ` passwd const arkProbe = probeMarker + ` passwd
root:x:0:0:root:/root:/bin/bash root:x:0:0:root:/root:/bin/bash
gameserver:x:25000:25000::/home/gameserver:/bin/bash gameserver:x:25000:25000::/home/gameserver:/bin/bash
@@ -34,7 +34,7 @@ bash|/usr/bin/bash
` + probeMarker + ` script /usr/bin/start_server ` + probeMarker + ` script /usr/bin/start_server
#!/bin/bash #!/bin/bash
cd /home/gameserver/steamcmd cd /home/gameserver/steamcmd
echo "$ASA_START_PARAMS ${ENABLE_DEBUG} $HOME" echo "$START_PARAMS ${ENABLE_DEBUG} $HOME"
` `
func arkFacts() *imageFacts { func arkFacts() *imageFacts {
@@ -55,7 +55,7 @@ func TestProbeParsing(t *testing.T) {
if !strings.Contains(f.Scripts["/usr/bin/start_server"], "steamcmd") { if !strings.Contains(f.Scripts["/usr/bin/start_server"], "steamcmd") {
t.Fatalf("script not captured: %q", f.Scripts) t.Fatalf("script not captured: %q", f.Scripts)
} }
if want := []string{"ASA_START_PARAMS", "ENABLE_DEBUG"}; !reflect.DeepEqual(f.EnvRead, want) { if want := []string{"ENABLE_DEBUG", "START_PARAMS"}; !reflect.DeepEqual(f.EnvRead, want) {
t.Errorf("envRead = %v, want %v (HOME is common and left out)", f.EnvRead, want) t.Errorf("envRead = %v, want %v (HOME is common and left out)", f.EnvRead, want)
} }
} }
@@ -63,7 +63,7 @@ func TestProbeParsing(t *testing.T) {
func TestSuggestContainerUser(t *testing.T) { func TestSuggestContainerUser(t *testing.T) {
// Locked home owned by another uid: use that account, with its own group // Locked home owned by another uid: use that account, with its own group
if f := arkFacts(); f.SuggestedContainerUser != "25000:25000" { if f := arkFacts(); f.SuggestedContainerUser != "25000:25000" {
t.Errorf("ARK: suggested %q, want 25000:25000", f.SuggestedContainerUser) t.Errorf("suggested %q, want 25000:25000", f.SuggestedContainerUser)
} }
// uid 1000 images need nothing // uid 1000 images need nothing
@@ -92,9 +92,9 @@ func TestCheckTemplate(t *testing.T) {
f := arkFacts() f := arkFacts()
game := map[string]any{ game := map[string]any{
"defaultConfig": map[string]any{"RCONPort": 27020, "MaxPlayers": 70}, "defaultConfig": map[string]any{"RCONPort": 27020, "MaxPlayers": 70},
// Mirrors ARK's old command: ASA_START_PARAMS is assigned, SERVER_MAP has a default, RCON_PORT is the // A command where START_PARAMS is assigned, SERVER_MAP has a default, RCON_PORT is the
// misnamed one, MAX_PLAYERS is set by a setting, MISSING has no default at all // misnamed one, MAX_PLAYERS is set by a setting, MISSING has no default at all
"defaultStartupCommand": `export ASA_START_PARAMS="${ASA_START_PARAMS:-${SERVER_MAP:-TheIsland}?RCONPort=${RCON_PORT:-27020} -Max=${MAX_PLAYERS:-70} ${MISSING}}"`, "defaultStartupCommand": `export START_PARAMS="${START_PARAMS:-${SERVER_MAP:-DefaultMap}?RCONPort=${RCON_PORT:-27020} -Max=${MAX_PLAYERS:-70} ${MISSING}}"`,
} }
problems, _ := checkTemplate(game, f) problems, _ := checkTemplate(game, f)
joined := strings.Join(problems, "\n") joined := strings.Join(problems, "\n")
@@ -103,7 +103,7 @@ func TestCheckTemplate(t *testing.T) {
t.Errorf("missing problem %q in:\n%s", want, joined) t.Errorf("missing problem %q in:\n%s", want, joined)
} }
} }
for _, unwanted := range []string{"ASA_START_PARAMS", "SERVER_MAP", "MAX_PLAYERS}"} { for _, unwanted := range []string{"START_PARAMS", "SERVER_MAP", "MAX_PLAYERS}"} {
if strings.Contains(joined, unwanted) { if strings.Contains(joined, unwanted) {
t.Errorf("false problem about %s in:\n%s", unwanted, joined) t.Errorf("false problem about %s in:\n%s", unwanted, joined)
} }
@@ -114,7 +114,7 @@ func TestCheckTemplate(t *testing.T) {
// With the fixes, only the deliberately missing variable is left; 25000:1000 counts (same uid) // With the fixes, only the deliberately missing variable is left; 25000:1000 counts (same uid)
game["containerUser"] = "25000:1000" game["containerUser"] = "25000:1000"
game["defaultStartupCommand"] = `export ASA_START_PARAMS="${ASA_START_PARAMS:-x}?RCONPort=${RCONPORT:-27020}"` game["defaultStartupCommand"] = `export START_PARAMS="${START_PARAMS:-x}?RCONPort=${RCONPORT:-27020}"`
if problems, _ := checkTemplate(game, f); len(problems) != 0 { if problems, _ := checkTemplate(game, f); len(problems) != 0 {
t.Errorf("fixed template still has problems: %v", problems) t.Errorf("fixed template still has problems: %v", problems)
} }
+10 -1
View File
@@ -7,6 +7,9 @@
// go run ./cmd/gbrgames inspect-image -game ID # read the image's facts and check the template against them // go run ./cmd/gbrgames inspect-image -game ID # read the image's facts and check the template against them
// go run ./cmd/gbrgames first-boot -game ID # start the server once, untouched; record its log and files // go run ./cmd/gbrgames first-boot -game ID # start the server once, untouched; record its log and files
// go run ./cmd/gbrgames draft -name N -image I # have an AI model write a new game's template from those facts // go run ./cmd/gbrgames draft -name N -image I # have an AI model write a new game's template from those facts
// go run ./cmd/gbrgames research -name N # find how a game's server runs: variants, images, Steam app
// go run ./cmd/gbrgames build -name N -variant V # first boot, draft, and boot the draft until it works
// go run ./cmd/gbrgames worker # take research/build jobs from the panel (see BUILDER.md)
package main package main
import ( import (
@@ -34,6 +37,12 @@ func main() {
err = runFirstBoot(os.Args[2:]) err = runFirstBoot(os.Args[2:])
case "draft": case "draft":
err = runDraft(os.Args[2:]) err = runDraft(os.Args[2:])
case "research":
err = runResearch(os.Args[2:])
case "build":
err = runBuild(os.Args[2:])
case "worker":
err = runWorker(os.Args[2:])
default: default:
usage() usage()
} }
@@ -44,6 +53,6 @@ func main() {
} }
func usage() { func usage() {
fmt.Fprintln(os.Stderr, "usage: gbrgames validate [-games DIR] | import [-games DIR] [-all] EXPORT.json | export [-repo R] [-commit SHA] [-out FILE] | boottest-payload [-out FILE] ID | inspect-image [-game ID] [-json] [-no-pull] [IMAGE] | first-boot [-game ID | -image IMAGE] [-env K=V] [-max D] [-quiet D] [-out DIR] | draft -name NAME -image IMAGE [-steam-app ID] [-first-boot REPORT] [-provider ollama|gemini] [-model M]") fmt.Fprintln(os.Stderr, "usage: gbrgames validate [-games DIR] | import [-games DIR] [-all] EXPORT.json | export [-repo R] [-commit SHA] [-out FILE] | boottest-payload [-out FILE] ID | inspect-image [-game ID] [-json] [-no-pull] [IMAGE] | first-boot [-game ID | -image IMAGE] [-env K=V] [-max D] [-quiet D] [-out DIR] | draft -name NAME -image IMAGE [-steam-app ID] [-first-boot REPORT] [-provider ollama|gemini] [-model M] | research -name NAME [-notes N] [-out FILE] | build -name NAME -variant JSON|@FILE [-out FILE] | worker [-once]")
os.Exit(2) os.Exit(2)
} }
+403
View File
@@ -0,0 +1,403 @@
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"net/url"
"os"
"regexp"
"sort"
"strings"
"time"
)
// research finds out how a game's dedicated server can be run before anything is drafted: it searches the
// web (a self-hosted SearXNG), reads the most useful pages (a reader that turns pages into text), and has
// the model list the server's variants (editions, crossplay builds, docker image or SteamCMD app). Every
// image and Steam app it names is then checked for real, so a variant shown to an admin exists.
type variant struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
Recommended bool `json:"recommended"`
Images []string `json:"images"`
SteamAppID string `json:"steamAppId"`
StartupHint string `json:"startupHint"`
Ports []string `json:"ports"`
Notes string `json:"notes"`
Sources []string `json:"sources"`
// Filled by the checks, not the model
Verified []string `json:"verified,omitempty"`
Problems []string `json:"problems,omitempty"`
}
type researchResult struct {
GameName string `json:"gameName"`
Supported bool `json:"supported"`
Reason string `json:"reason,omitempty"`
Message string `json:"message"`
Variants []variant `json:"variants"`
Sources []string `json:"sources"`
}
var researchSchema = map[string]any{
"type": "object",
"required": []string{"supported", "reason", "message", "variants"},
"properties": map[string]any{
"supported": map[string]any{"type": "boolean"},
"reason": map[string]any{"enum": []string{"ok", "unreleased", "single_player", "no_dedicated_server", "sunset", "unknown"}},
"message": map[string]any{"type": "string"},
"variants": map[string]any{
"type": "array",
"items": map[string]any{
"type": "object",
"required": []string{"id", "title", "description", "recommended", "images", "steamAppId", "startupHint", "ports", "notes", "sources"},
"properties": map[string]any{
"id": map[string]any{"type": "string"},
"title": map[string]any{"type": "string"},
"description": map[string]any{"type": "string"},
"recommended": map[string]any{"type": "boolean"},
"images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"steamAppId": map[string]any{"type": "string"},
"startupHint": map[string]any{"type": "string"},
"ports": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"notes": map[string]any{"type": "string"},
"sources": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
},
},
},
},
}
func runResearch(args []string) error {
fs := flag.NewFlagSet("research", flag.ExitOnError)
gameName := fs.String("name", "", "the game's name, as players know it")
notes := fs.String("notes", "", "what the admin asked for (crossplay, a modded edition...)")
provider := fs.String("provider", "ollama", "ollama or gemini")
modelName := fs.String("model", "", "model name (default as for draft)")
pages := fs.Int("pages", 6, "how many search results to read")
out := fs.String("out", "", "write the result JSON here (default stdout)")
fs.Parse(args)
if *gameName == "" {
return errors.New("research needs -name")
}
m, err := newModel(*provider, *modelName)
if err != nil {
return err
}
res, err := research(m, *gameName, *notes, *pages)
if err != nil {
return err
}
raw, _ := json.MarshalIndent(res, "", " ")
if *out == "" {
fmt.Println(string(raw))
return nil
}
return os.WriteFile(*out, raw, 0o644)
}
func research(m model, gameName, notes string, pages int) (*researchResult, error) {
searx, reader := os.Getenv("SEARXNG_URL"), os.Getenv("READER_URL")
if searx == "" || reader == "" {
return nil, errors.New("set SEARXNG_URL (e.g. http://host:8080) and READER_URL (e.g. http://host:3000)")
}
queries := []string{
gameName + " dedicated server docker image",
gameName + " dedicated server steamcmd app id linux",
gameName + " dedicated server setup ports launch options",
gameName + " crossplay console dedicated server",
}
var hits []searchHit
for _, q := range queries {
fmt.Printf("[research] searching: %s\n", q)
h, err := searxSearch(searx, q)
if err != nil {
fmt.Printf("[research] search failed: %v\n", err)
continue
}
hits = append(hits, h...)
}
if len(hits) == 0 {
return nil, errors.New("no search results (is SearXNG reachable?)")
}
picked := pickPages(hits, pages)
var b strings.Builder
fmt.Fprintf(&b, "Game: **%s**\n", gameName)
if notes != "" {
fmt.Fprintf(&b, "The admin asked for: %s\n", notes)
}
b.WriteString("\n## Search results\n")
for _, h := range dedupeHits(hits) {
fmt.Fprintf(&b, "- %s — %s: %.200s\n", h.URL, h.Title, h.Content)
}
var sources []string
for _, h := range picked {
fmt.Printf("[research] reading %s\n", h.URL)
text, err := readPage(reader, h.URL)
if err != nil {
fmt.Printf("[research] could not read it: %v\n", err)
continue
}
sources = append(sources, h.URL)
fmt.Fprintf(&b, "\n## Page: %s\n%s\n", h.URL, text)
}
messages := []chatMessage{
{Role: "system", Content: researchPrompt},
{Role: "user", Content: b.String()},
}
fmt.Printf("[research] asking %s\n", m.name())
answer, err := m.chat(messages, researchSchema)
if err != nil {
return nil, err
}
res := &researchResult{}
if err := json.Unmarshal([]byte(trimToJSON(answer)), res); err != nil {
return nil, fmt.Errorf("the model's answer is not valid JSON: %w", err)
}
res.GameName = gameName
res.Sources = sources
verifyVariants(res)
return res, nil
}
const researchPrompt = "You research how to host a game's dedicated server for GBR Hosting, a game server host. Servers run " +
"in Docker on Linux, as a non-root user, with a volume at /server. From the search results and pages given, list " +
"the ways this game's dedicated server can be run (variants): separate editions of the game with their own server, " +
"crossplay builds, or notably different server images. For each variant give:\n" +
"- images: Docker Hub images that run this server, best first (prefer images that run as a non-root user, and the " +
"plain SteamCMD route cm2network/steamcmd when the game ships a Linux dedicated server on Steam). Only images that " +
"appear in the pages.\n" +
"- steamAppId: the Steam app id of the DEDICATED SERVER (not the game itself), if installed via SteamCMD; else \"\".\n" +
"- startupHint: the command that starts the server from its install folder /server, with the launch options " +
"for name, port, world and save location exactly as the pages document them; \"\" if unknown.\n" +
"- ports: every port players connect to, as \"port/proto\".\n" +
"- notes: anything a template author must know (Windows-only server needing Proton, crossplay flags, mods).\n" +
"- sources: the page URLs that support it.\n" +
"Mark one variant recommended. Use only what the pages say or what you know for certain; leave fields empty " +
"instead of guessing. If no dedicated server can be hosted (single-player only, unreleased, official servers " +
"only, shut down), set supported false, the matching reason, no variants, and explain in message. Otherwise " +
"reason \"ok\" and a one-sentence message."
type searchHit struct {
URL string `json:"url"`
Title string `json:"title"`
Content string `json:"content"`
}
func searxSearch(base, query string) ([]searchHit, error) {
u := strings.TrimRight(base, "/") + "/search?format=json&q=" + url.QueryEscape(query)
raw, err := httpGet(u, 30*time.Second)
if err != nil {
return nil, err
}
var resp struct {
Results []searchHit `json:"results"`
}
if err := json.Unmarshal(raw, &resp); err != nil {
return nil, fmt.Errorf("unreadable search answer: %.200s", raw)
}
if len(resp.Results) > 8 {
resp.Results = resp.Results[:8]
}
return resp.Results, nil
}
// Pages that hold the facts a template needs, read first
var usefulPage = regexp.MustCompile(`(?i)hub\.docker\.com|github\.com|steamdb\.info|developer\.valvesoftware|wiki|fandom|reddit\.com/r/\w*(server|admin)`)
// pickPages reads one page per site, docs and image pages first, so a handful of reads covers the most ground.
func pickPages(hits []searchHit, n int) []searchHit {
hits = dedupeHits(hits)
sort.SliceStable(hits, func(i, j int) bool {
return usefulPage.MatchString(hits[i].URL) && !usefulPage.MatchString(hits[j].URL)
})
seen := map[string]bool{}
var out []searchHit
for _, h := range hits {
u, err := url.Parse(h.URL)
if err != nil || seen[u.Host] {
continue
}
seen[u.Host] = true
out = append(out, h)
if len(out) == n {
break
}
}
return out
}
func dedupeHits(hits []searchHit) []searchHit {
seen := map[string]bool{}
var out []searchHit
for _, h := range hits {
if h.URL == "" || seen[h.URL] {
continue
}
seen[h.URL] = true
out = append(out, h)
}
return out
}
func readPage(reader, pageURL string) (string, error) {
return readPageLimit(reader, pageURL, 9000)
}
// readPageLimit reads a page as text through the reader, cut to limit bytes.
func readPageLimit(reader, pageURL string, limit int) (string, error) {
raw, err := httpGet(strings.TrimRight(reader, "/")+"/"+pageURL, 60*time.Second)
if err != nil {
return "", err
}
text := string(raw)
if len(text) > limit {
text = text[:limit] + "\n...(truncated)"
}
return text, nil
}
func httpGet(u string, timeout time.Duration) ([]byte, error) {
client := &http.Client{Timeout: timeout}
resp, err := client.Get(u)
if err != nil {
return nil, err
}
defer resp.Body.Close()
raw, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, err
}
if resp.StatusCode >= 300 {
return nil, fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
return raw, nil
}
// trimToJSON cuts prose or code fences some models put around a JSON answer.
func trimToJSON(answer string) string {
answer = strings.TrimSpace(answer)
if i := strings.Index(answer, "{"); i > 0 {
answer = answer[i:]
}
if i := strings.LastIndex(answer, "}"); i >= 0 {
answer = answer[:i+1]
}
return answer
}
// verifyVariants checks every image and Steam app the model named: images must exist in their registry,
// Steam apps must exist and be readable anonymously (what SteamCMD installs need). What fails is dropped
// and noted; a variant left with nothing runnable can't be recommended.
func verifyVariants(res *researchResult) {
var apps []string
for _, v := range res.Variants {
if v.SteamAppID != "" {
apps = append(apps, v.SteamAppID)
}
}
names := steamAppNames(apps)
anyRunnable := false
for i := range res.Variants {
v := &res.Variants[i]
var images []string
for _, img := range v.Images {
img = strings.TrimSpace(img)
if img == "" {
continue
}
if out, err := runCmd("docker", "manifest", "inspect", img); err != nil {
v.Problems = append(v.Problems, fmt.Sprintf("image %s not found: %.120s", img, out))
continue
}
v.Verified = append(v.Verified, "image "+img+" exists")
images = append(images, img)
}
v.Images = images
if v.SteamAppID != "" {
if name, ok := names[v.SteamAppID]; ok {
v.Verified = append(v.Verified, fmt.Sprintf("Steam app %s is %q", v.SteamAppID, name))
if !hasImage(v.Images, "cm2network/steamcmd") {
v.Images = append(v.Images, "cm2network/steamcmd")
}
} else {
v.Problems = append(v.Problems, "Steam app "+v.SteamAppID+" not found for anonymous SteamCMD")
v.SteamAppID = ""
}
}
if len(v.Images) == 0 {
v.Problems = append(v.Problems, "no image or Steam app that could be checked; it can't be built")
v.Recommended = false
} else {
anyRunnable = true
}
}
if !anyRunnable && res.Supported {
res.Supported = false
res.Reason = "unknown"
res.Message = "No variant has a Docker image or Steam app that could be confirmed. " + res.Message
}
// Exactly one recommendation, among the runnable ones
rec := -1
for i, v := range res.Variants {
if v.Recommended && rec < 0 && len(v.Images) > 0 {
rec = i
}
res.Variants[i].Recommended = false
}
if rec < 0 {
for i, v := range res.Variants {
if len(v.Images) > 0 {
rec = i
break
}
}
}
if rec >= 0 {
res.Variants[rec].Recommended = true
}
}
func hasImage(images []string, name string) bool {
for _, i := range images {
if i == name || strings.HasPrefix(i, name+":") {
return true
}
}
return false
}
var appName = regexp.MustCompile(`"common"\s*\{[^}]*?"name"\s+"([^"]+)"`)
// steamAppNames asks SteamCMD (anonymously, in its official image) for each app's info: an app that
// anonymous SteamCMD can't see can't be installed by the node-agent either.
func steamAppNames(apps []string) map[string]string {
names := map[string]string{}
if len(apps) == 0 {
return names
}
for _, app := range apps {
if !regexp.MustCompile(`^\d+$`).MatchString(app) {
continue
}
out, err := runCmd("docker", "run", "--rm", "--network", bootTestNetwork, "cm2network/steamcmd", "bash", "-c",
"/home/steam/steamcmd/steamcmd.sh +login anonymous +app_info_update 1 +app_info_print "+app+" +quit")
if err != nil {
fmt.Printf("[research] steamcmd for app %s: %v\n", app, err)
continue
}
if m := appName.FindStringSubmatch(out); m != nil {
names[app] = m[1]
}
}
return names
}
+41
View File
@@ -0,0 +1,41 @@
package main
import "testing"
func TestPickPagesPrefersDocsAndOnePerSite(t *testing.T) {
hits := []searchHit{
{URL: "https://blog.example.com/example-server"},
{URL: "https://github.com/a/example-docker"},
{URL: "https://github.com/b/other"},
{URL: "https://hub.docker.com/r/x/example"},
{URL: "https://blog.example.com/example-server"},
}
got := pickPages(hits, 3)
if len(got) != 3 {
t.Fatalf("got %d pages", len(got))
}
if got[0].URL != "https://github.com/a/example-docker" || got[1].URL != "https://hub.docker.com/r/x/example" || got[2].URL != "https://blog.example.com/example-server" {
t.Errorf("order: %v", got)
}
}
func TestTrimToJSON(t *testing.T) {
if got := trimToJSON("Here you go:\n```json\n{\"a\": 1}\n```"); got != `{"a": 1}` {
t.Errorf("got %q", got)
}
}
func TestAppNameFromAppInfo(t *testing.T) {
out := `"1000000"
{
"common"
{
"name" "Example Dedicated Server"
"type" "Tool"
}
}`
m := appName.FindStringSubmatch(out)
if m == nil || m[1] != "Example Dedicated Server" {
t.Errorf("got %v", m)
}
}
+2 -1
View File
@@ -168,7 +168,8 @@ func validateGame(dir, folder string, gameSchema, testSchema *jsonschema.Schema)
add("test.yaml config.%s is not a key of this game", key) add("test.yaml config.%s is not a key of this game", key)
} }
if sensitiveKey.MatchString(key) { if sensitiveKey.MatchString(key) {
add("test.yaml config.%s looks like a credential; map it under 'secrets' to a Gitea secret instead", key) add("test.yaml config.%s looks like a credential; remove it (boot tests fill password settings with a throwaway "+
"value), or, only if the server can't start without a real one, map it under 'secrets' to a CI secret", key)
} }
} }
for _, key := range sortedKeys(secrets) { for _, key := range sortedKeys(secrets) {
+491
View File
@@ -0,0 +1,491 @@
package main
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"sync"
"time"
)
// worker runs the AI builder. It asks the panel for queued jobs (it only connects out,
// like node-agents do), runs research or build in a fresh clone of this repo, streams the output to the
// panel as the job's log, and finishes a build with a pull request opened as the bot account. CI then
// validates and boot-tests that pull request like any other.
//
// Panel API (headers X-Builder-Token and X-Builder-Worker):
//
// POST /api/v1/builder/jobs/claim {worker} -> 200 builderJob | 204 nothing queued
// POST /api/v1/builder/jobs/{id}/progress {stage, lines} -> 200 {cancelled}
// POST /api/v1/builder/jobs/{id}/result {research | build+prUrl | error}
type builderJob struct {
ID string `json:"id"`
Kind string `json:"kind"` // "research" or "build"
GameName string `json:"gameName"`
Notes string `json:"notes"`
Variant json.RawMessage `json:"variant"`
}
type jobResult struct {
Research *researchResult `json:"research,omitempty"`
Build *buildResult `json:"build,omitempty"`
PRURL string `json:"prUrl,omitempty"`
Error string `json:"error,omitempty"`
}
type workerConfig struct {
panel, token string
gitea, repo, botToken string
workDir, name string
poll time.Duration
provider, model string
botLogin, botEmail string // the token's account, used as the commit author
}
// checkBot reads which account GITEA_TOKEN belongs to and makes sure it can push to the games repo, so a
// wrong token or missing permission shows when the worker starts, not at the end of a 30-minute build.
func (c *workerConfig) checkBot() error {
var user struct {
Login string `json:"login"`
Email string `json:"email"`
}
if err := c.giteaGet("/user", &user); err != nil {
if !strings.HasPrefix(err.Error(), "HTTP 403") {
return fmt.Errorf("GITEA_TOKEN: %w", err)
}
// A repository-only token (all the worker needs) may not read its own account: name the author from
// BOT_NAME instead. Whether it can push is still checked below.
user.Login = envOr("BOT_NAME", "gbr-bot")
}
var repo struct {
Permissions struct {
Push bool `json:"push"`
} `json:"permissions"`
}
if err := c.giteaGet("/repos/"+c.repo, &repo); err != nil {
return fmt.Errorf("%s: %w", c.repo, err)
}
if !repo.Permissions.Push {
return fmt.Errorf("the account %s can't push to %s: give it Write access there", user.Login, c.repo)
}
c.botLogin, c.botEmail = user.Login, user.Email
if c.botEmail == "" {
c.botEmail = user.Login + "@users.noreply"
}
return nil
}
func (c *workerConfig) giteaGet(p string, out any) error {
req, err := http.NewRequest(http.MethodGet, c.gitea+"/api/v1"+p, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "token "+c.botToken)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 300 {
raw, _ := io.ReadAll(resp.Body)
return fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
return json.NewDecoder(resp.Body).Decode(out)
}
func runWorker(args []string) error {
fs := flag.NewFlagSet("worker", flag.ExitOnError)
once := fs.Bool("once", false, "handle at most one job, then exit")
fs.Parse(args)
host, _ := os.Hostname()
c := workerConfig{
panel: strings.TrimRight(os.Getenv("PANEL_URL"), "/"), token: os.Getenv("BUILDER_TOKEN"),
gitea: strings.TrimRight(envOr("GITEA_URL", "https://www.gitea.gbreed.com"), "/"), repo: envOr("GAMES_REPO", "gbreed-public/gbr-games"),
botToken: os.Getenv("GITEA_TOKEN"), workDir: envOr("WORK_DIR", "/work"), name: envOr("WORKER_NAME", host),
provider: envOr("MODEL_PROVIDER", "ollama"), model: os.Getenv("MODEL"),
}
var err error
if c.poll, err = time.ParseDuration(envOr("POLL_INTERVAL", "20s")); err != nil {
return err
}
for _, v := range []struct{ name, value string }{{"PANEL_URL", c.panel}, {"BUILDER_TOKEN", c.token}, {"GITEA_TOKEN", c.botToken}} {
if v.value == "" {
return fmt.Errorf("set %s", v.name)
}
}
if err := c.checkBot(); err != nil {
return err
}
fmt.Printf("[worker] %s polling %s every %s; pull requests as %s\n", c.name, c.panel, c.poll, c.botLogin)
for {
job, err := c.claim()
if err != nil {
fmt.Printf("[worker] claim: %v\n", err)
}
if job != nil {
fmt.Printf("[worker] job %s: %s %q\n", job.ID, job.Kind, job.GameName)
res := c.handle(job)
if err := c.post("/jobs/"+job.ID+"/result", res, nil); err != nil {
fmt.Printf("[worker] posting the result of %s: %v\n", job.ID, err)
}
if *once {
return nil
}
continue
}
if *once {
return nil
}
time.Sleep(c.poll)
}
}
func envOr(name, def string) string {
if v := os.Getenv(name); v != "" {
return v
}
return def
}
func (c *workerConfig) claim() (*builderJob, error) {
var job builderJob
status, err := c.request("/jobs/claim", map[string]string{"worker": c.name}, &job)
if err != nil || status == http.StatusNoContent {
return nil, err
}
return &job, nil
}
func (c *workerConfig) post(p string, body, out any) error {
_, err := c.request(p, body, out)
return err
}
func (c *workerConfig) request(p string, body, out any) (int, error) {
payload, err := json.Marshal(body)
if err != nil {
return 0, err
}
req, err := http.NewRequest(http.MethodPost, c.panel+"/api/v1/builder"+p, bytes.NewReader(payload))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-Builder-Token", c.token)
req.Header.Set("X-Builder-Worker", c.name)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode >= 300 {
return resp.StatusCode, fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
if out != nil && len(bytes.TrimSpace(raw)) > 0 {
if err := json.Unmarshal(raw, out); err != nil {
return resp.StatusCode, err
}
}
return resp.StatusCode, nil
}
// handle runs one job in its own clone and returns what to report; it never panics the worker.
func (c *workerConfig) handle(job *builderJob) *jobResult {
dir := filepath.Join(c.workDir, "job-"+job.ID)
os.RemoveAll(dir)
defer os.RemoveAll(dir)
if out, err := runCmd("git", "clone", "-q", "--depth", "1", c.gitea+"/"+c.repo+".git", dir); err != nil {
return &jobResult{Error: "git clone: " + out}
}
self, err := os.Executable()
if err != nil {
return &jobResult{Error: err.Error()}
}
resultFile := filepath.Join(dir, "job-result.json")
modelArgs := []string{"-provider", c.provider}
if c.model != "" {
modelArgs = append(modelArgs, "-model", c.model)
}
switch job.Kind {
case "research":
args := append([]string{"research", "-name", job.GameName, "-notes", job.Notes, "-out", resultFile}, modelArgs...)
if err := c.runStreamed(job.ID, dir, self, args, "RESEARCH"); err != nil {
return &jobResult{Error: err.Error()}
}
res := &researchResult{}
if err := readJSON(resultFile, res); err != nil {
return &jobResult{Error: err.Error()}
}
return &jobResult{Research: res}
case "build":
variantFile := filepath.Join(dir, "job-variant.json")
if err := os.WriteFile(variantFile, job.Variant, 0o644); err != nil {
return &jobResult{Error: err.Error()}
}
args := append([]string{"build", "-name", job.GameName, "-notes", job.Notes, "-variant", "@" + variantFile, "-out", resultFile}, modelArgs...)
runErr := c.runStreamed(job.ID, dir, self, args, "FIRST_BOOT")
res := &buildResult{}
if err := readJSON(resultFile, res); err != nil {
msg := "the build produced no result"
if runErr != nil {
msg = runErr.Error()
}
return &jobResult{Error: msg}
}
// A draft that validates but didn't boot still goes up as a draft PR: a person can finish it, and
// CI shows the same boot failure
if _, err := os.Stat(filepath.Join(dir, res.Dir, "game.yaml")); err != nil {
return &jobResult{Build: res, Error: "no template was written: " + strings.Join(res.Problems, "; ")}
}
c.progress(job.ID, "OPENING_PR", []string{"[worker] opening the pull request"})
pr, err := c.openPR(dir, job, res)
if err != nil {
return &jobResult{Build: res, Error: "opening the pull request: " + err.Error()}
}
return &jobResult{Build: res, PRURL: pr}
}
return &jobResult{Error: "unknown job kind " + job.Kind}
}
// Output prefixes that mark a new stage of a build, shown in the panel's progress bar
var stagePrefixes = []struct{ prefix, stage string }{
{"[build] first boot", "FIRST_BOOT"},
{"[first-boot]", ""},
{"[draft]", "DRAFTING"},
{"[build] boot ", "BOOT_TEST"},
}
// runStreamed runs a gbrgames command and sends its output to the panel every few seconds (which also
// tells the panel the worker is alive). The panel can answer "cancelled" to stop the job.
func (c *workerConfig) runStreamed(jobID, dir, self string, args []string, stage string) error {
cmd := exec.Command(self, args...)
cmd.Dir = dir
pr, pw := io.Pipe()
cmd.Stdout, cmd.Stderr = pw, pw
if err := cmd.Start(); err != nil {
return err
}
var mu sync.Mutex
var pending []string
var lastErr string
done := make(chan struct{})
go func() {
sc := bufio.NewScanner(pr)
sc.Buffer(make([]byte, 1024*1024), 1024*1024)
for sc.Scan() {
line := strings.TrimRight(ansi.ReplaceAllString(sc.Text(), ""), "\r ")
fmt.Println(line)
mu.Lock()
pending = append(pending, line)
for _, s := range stagePrefixes {
if s.stage != "" && strings.HasPrefix(line, s.prefix) {
stage = s.stage
}
}
if strings.HasPrefix(line, "error: ") {
lastErr = strings.TrimPrefix(line, "error: ")
}
mu.Unlock()
}
close(done)
}()
wait := make(chan error, 1)
go func() {
err := cmd.Wait()
pw.Close()
<-done
wait <- err
}()
tick := time.NewTicker(5 * time.Second)
defer tick.Stop()
flush := func() bool {
mu.Lock()
lines, st := pending, stage
pending = nil
mu.Unlock()
return c.progress(jobID, st, lines)
}
for {
select {
case err := <-wait:
flush()
if err != nil {
if lastErr != "" {
return errors.New(lastErr)
}
return err
}
return nil
case <-tick.C:
if cancelled := flush(); cancelled {
cmd.Process.Kill()
<-wait
return errors.New("cancelled from the panel")
}
}
}
}
// progress sends log lines (possibly none, as a heartbeat) and reports whether the job was cancelled.
func (c *workerConfig) progress(jobID, stage string, lines []string) bool {
var resp struct {
Cancelled bool `json:"cancelled"`
}
if lines == nil {
lines = []string{}
}
if err := c.post("/jobs/"+jobID+"/progress", map[string]any{"stage": stage, "lines": lines}, &resp); err != nil {
fmt.Printf("[worker] progress: %v\n", err)
}
return resp.Cancelled
}
// openPR commits the new game folder to the game's draft branch as the bot and opens a pull request to
// main. A game has one draft at a time: an earlier draft's PR is closed and its branch deleted first.
func (c *workerConfig) openPR(dir string, job *builderJob, res *buildResult) (string, error) {
branch := draftBranch(res.GameID)
replaced, err := c.replaceOldDrafts(res.GameID)
if err != nil {
return "", fmt.Errorf("replacing the earlier draft: %w", err)
}
summary := res.Summary
for _, n := range replaced {
summary += fmt.Sprintf("\nReplaces #%d (closed, branch deleted).\n", n)
}
auth := "http.extraHeader=Authorization: token " + c.botToken
steps := []struct {
name string
args []string
}{
{"checkout", []string{"checkout", "-q", "-b", branch}},
{"add", []string{"add", filepath.ToSlash(res.Dir)}},
{"commit", []string{"-c", "user.name=" + c.botLogin, "-c", "user.email=" + c.botEmail, "commit", "-q", "-m",
fmt.Sprintf("feat(games): draft %s\n\nDrafted and boot-tested by the GBR builder (job %s).", job.GameName, job.ID)}},
// Forced: the branch is the bot's own, and whatever was on it was the draft being replaced
{"push", []string{"-c", auth, "push", "-q", "--force", "origin", branch}},
}
for _, s := range steps {
cmd := exec.Command("git", s.args...)
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil {
// The push command holds the token: name the step only, and mask it in git's output
return "", fmt.Errorf("git %s: %s", s.name, strings.ReplaceAll(string(out), c.botToken, "***"))
}
}
title := "feat(games): add " + job.GameName
if !res.Passed {
title = "WIP: " + title + " (did not boot yet)"
}
var pr struct {
HTMLURL string `json:"html_url"`
}
body := map[string]any{"title": title, "head": branch, "base": "main", "body": summary}
if err := c.giteaCall(http.MethodPost, "/repos/"+c.repo+"/pulls", body, &pr); err != nil {
return "", err
}
return pr.HTMLURL, nil
}
// replaceOldDrafts closes the game's open draft PRs and deletes their branches, so a rebuild leaves one
// draft to review instead of a pile. Only draft/ branches for this game are touched, never anyone's other
// work. Returns the numbers of the PRs it closed.
func (c *workerConfig) replaceOldDrafts(gameID string) ([]int, error) {
var pulls []struct {
Number int `json:"number"`
Head struct {
Ref string `json:"ref"`
} `json:"head"`
}
if err := c.giteaCall(http.MethodGet, "/repos/"+c.repo+"/pulls?state=open&limit=50", nil, &pulls); err != nil {
return nil, err
}
branches := map[string]bool{draftBranch(gameID): true}
var closed []int
for _, p := range pulls {
if !isDraftBranchOf(p.Head.Ref, gameID) {
continue
}
if err := c.giteaCall(http.MethodPatch, fmt.Sprintf("/repos/%s/pulls/%d", c.repo, p.Number), map[string]string{"state": "closed"}, nil); err != nil {
return closed, err
}
fmt.Printf("[worker] closed #%d, the earlier draft of %s\n", p.Number, gameID)
closed = append(closed, p.Number)
branches[p.Head.Ref] = true
}
for b := range branches {
// Gone already is fine: not every game has had a draft
if err := c.giteaCall(http.MethodDelete, "/repos/"+c.repo+"/branches/"+url.PathEscape(b), nil, nil); err != nil && !strings.HasPrefix(err.Error(), "HTTP 404") {
return closed, fmt.Errorf("deleting branch %s: %w", b, err)
}
}
return closed, nil
}
func draftBranch(gameID string) string {
return "draft/" + gameID
}
// isDraftBranchOf matches draft/<id>, and draft/<id>-<job> as the worker named them before.
func isDraftBranchOf(ref, gameID string) bool {
d := draftBranch(gameID)
if ref == d {
return true
}
rest, ok := strings.CutPrefix(ref, d+"-")
return ok && regexp.MustCompile(`^[0-9a-f]{8}$`).MatchString(rest)
}
// giteaCall sends one API call as the bot; out may be nil.
func (c *workerConfig) giteaCall(method, p string, body, out any) error {
var rd io.Reader
if body != nil {
payload, err := json.Marshal(body)
if err != nil {
return err
}
rd = bytes.NewReader(payload)
}
req, err := http.NewRequest(method, c.gitea+"/api/v1"+p, rd)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "token "+c.botToken)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode >= 300 {
return fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
if out != nil && len(bytes.TrimSpace(raw)) > 0 {
return json.Unmarshal(raw, out)
}
return nil
}
func readJSON(p string, out any) error {
raw, err := os.ReadFile(p)
if err != nil {
return err
}
return json.Unmarshal(raw, out)
}
+115
View File
@@ -0,0 +1,115 @@
package main
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestCheckBotNeedsPushAccess(t *testing.T) {
push := false
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "token secret" {
w.WriteHeader(http.StatusUnauthorized)
return
}
switch r.URL.Path {
case "/api/v1/user":
w.Write([]byte(`{"login":"git_ai_bot","email":"bot@example.com"}`))
case "/api/v1/repos/org/games":
if push {
w.Write([]byte(`{"permissions":{"push":true}}`))
} else {
w.Write([]byte(`{"permissions":{"push":false}}`))
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer srv.Close()
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
if err := c.checkBot(); err == nil || !strings.Contains(err.Error(), "git_ai_bot can't push to org/games") {
t.Errorf("read-only bot: got %v", err)
}
push = true
if err := c.checkBot(); err != nil {
t.Fatal(err)
}
if c.botLogin != "git_ai_bot" || c.botEmail != "bot@example.com" {
t.Errorf("identity = %s <%s>", c.botLogin, c.botEmail)
}
c.botToken = "wrong"
if err := c.checkBot(); err == nil || !strings.Contains(err.Error(), "GITEA_TOKEN") {
t.Errorf("bad token: got %v", err)
}
}
func TestCheckBotWithRepositoryOnlyToken(t *testing.T) {
// Gitea answers /user with 403 when the token only has repository scope
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/v1/user":
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(`{"message":"token does not have at least one of required scope(s), required=[read:user]"}`))
case "/api/v1/repos/org/games":
w.Write([]byte(`{"permissions":{"push":true}}`))
}
}))
defer srv.Close()
t.Setenv("BOT_NAME", "gbr-bot")
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
if err := c.checkBot(); err != nil {
t.Fatal(err)
}
if c.botLogin != "gbr-bot" || c.botEmail != "gbr-bot@users.noreply" {
t.Errorf("identity = %s <%s>", c.botLogin, c.botEmail)
}
}
func TestReplaceOldDraftsClosesOnlyThisGamesDrafts(t *testing.T) {
var calls []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls = append(calls, r.Method+" "+r.URL.Path)
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/repos/org/games/pulls":
w.Write([]byte(`[
{"number": 13, "head": {"ref": "draft/example-4b174b2d"}},
{"number": 20, "head": {"ref": "draft/example"}},
{"number": 21, "head": {"ref": "draft/example-other"}},
{"number": 22, "head": {"ref": "draft/other-4b174b2d"}},
{"number": 23, "head": {"ref": "fix/example-ports"}}
]`))
case r.Method == http.MethodDelete && r.URL.Path == "/api/v1/repos/org/games/branches/draft/example-4b174b2d":
w.WriteHeader(http.StatusNoContent)
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNotFound) // already gone is fine
default:
w.Write([]byte(`{}`))
}
}))
defer srv.Close()
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
closed, err := c.replaceOldDrafts("example")
if err != nil {
t.Fatal(err)
}
if len(closed) != 2 || closed[0] != 13 || closed[1] != 20 {
t.Errorf("closed %v, want [13 20]: not another game's draft, not draft/example-other, not someone's fix branch", closed)
}
got := strings.Join(calls, "\n")
for _, want := range []string{"PATCH /api/v1/repos/org/games/pulls/13", "PATCH /api/v1/repos/org/games/pulls/20",
"DELETE /api/v1/repos/org/games/branches/draft/example-4b174b2d", "DELETE /api/v1/repos/org/games/branches/draft/example"} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in:\n%s", want, got)
}
}
for _, unwanted := range []string{"pulls/21", "pulls/22", "pulls/23"} {
if strings.Contains(got, unwanted) {
t.Errorf("touched %s:\n%s", unwanted, got)
}
}
}
+1 -1
View File
@@ -17,7 +17,7 @@
"defaultDockerImage": { "description": "Image the game server runs in.", "type": "string", "minLength": 1 }, "defaultDockerImage": { "description": "Image the game server runs in.", "type": "string", "minLength": 1 },
"containerUser": { "containerUser": {
"description": "uid:gid the server runs as, for images built around their own user (ARK's needs 25000:25000). Defaults to 1000:1000; root is never allowed.", "description": "uid:gid the server runs as, for images built around their own user, as inspect-image reports it. Defaults to 1000:1000; root is never allowed.",
"type": ["string", "null"], "type": ["string", "null"],
"pattern": "^[1-9][0-9]{0,8}:[1-9][0-9]{0,8}$" "pattern": "^[1-9][0-9]{0,8}:[1-9][0-9]{0,8}$"
}, },