Merge branch 'main' into fix/port-palworld
Validate game templates / validate (push) Successful in 23s
Validate game templates / boot-test (push) Successful in 3m47s
Validate game templates / publish (push) Skipped

This commit is contained in:
greg committed 2026-10-26 00:34:58 +00:00
commit b1fb77f284
25 files changed
+2486 -187

No files matched your search

+37
View File
@@ -0,0 +1,37 @@
name: Builder image
# Publishes the AI builder worker image (gbrgames worker, see BUILDER.md).
on:
push:
branches: ["main"]
paths:
- "cmd/**"
- "go.mod"
- "go.sum"
- "Dockerfile.builder"
workflow_dispatch:
env:
REGISTRY: www.gitea.gbreed.com
IMAGE_NAME: gbr-builder
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Login to Gitea Container Registry
run: |
echo "${{ secrets.GIT_SERVER_TOKEN }}" | docker login ${{ env.REGISTRY }} \
-u ${{ secrets.GIT_TOKEN_USERNAME }} \
--password-stdin
- name: Build and Push Docker Image
run: |
IMAGE_PATH="${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}"
SHORT_SHA=$(echo "${{ github.sha }}" | cut -c1-7)
docker build -f Dockerfile.builder -t $IMAGE_PATH:$SHORT_SHA -t $IMAGE_PATH:latest .
docker push $IMAGE_PATH:$SHORT_SHA
docker push $IMAGE_PATH:latest
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
boot-test:
needs: validate
if: needs.validate.outputs.games != ''
# A dedicated runner on VM 116 whose game containers can reach the internet but not the LAN
# A dedicated runner whose game containers can reach the internet but not private networks
runs-on: game-test
steps:
- name: Checkout Code
+13 -4
View File
@@ -41,22 +41,31 @@ Know this before writing anything:
- `defaultPorts` lists **only what players connect to** (game and query ports). RCON, REST and admin
ports are **never** published.
- Use `"7777/udp"` or `"8211:8211/udp"`. Every port needs its protocol.
- Nodes always publish these container ports (only the host side moves when servers share a node).
So the game's own port is **not a setting**: write it into the startup command or config file. A
game that must announce the port players use gets `{PUBLIC_PORT_<port>}`.
### The container user
- Run `gbrgames inspect-image -game <id>`. If the image is built around its own user (it declares
`USER`, or locks a home folder to a uid other than 1000), set `containerUser` to that `uid:gid`.
Example: ARK's image needs `25000:25000`.
`USER`, or locks a home folder to a uid other than 1000), set `containerUser` to that `uid:gid`, as
the inspector reports it.
### The startup command
- Read settings under the names the agent gives them (see above). `${RCON_PORT}` when the setting is
`RCONPort` silently ignores the owner's value; it arrives as `RCONPORT`.
- Use `${VAR:-default}` for anything that may be unset.
- Only use launch options the game's documentation lists, spelled as it spells them. Servers ignore options
they don't know, so an invented one boots fine and its setting does nothing.
- Quote text settings where they're used: `-name "${SERVER_NAME}"`. `ARGS="-name ${SERVER_NAME}"; ./server $ARGS`
splits "GBRHosting - My Server" into words; build optional arguments in a bash array instead
(`args=(-name "$SERVER_NAME"); [ -n "$PASSWORD" ] && args+=(-password "$PASSWORD"); ./server "${args[@]}"`).
- If the image installs the game itself on start, make sure a failed install is retried rather than
starting a server that isn't there (see ARK's template).
starting a server that isn't there: log in to SteamCMD once (`+login anonymous +quit`) before the
install, and retry the install a few times until the server's executable exists.
### Memory
- `requiredMemoryMb` is the container's hard limit. A JVM or engine heap must stay about **1 GB below**
it: Minecraft's `MEMORY` is `3G` for `4096` MB. A heap equal to the limit is killed on start.
it: a `3G` heap for `4096` MB. A heap equal to the limit is killed on start.
### Settings (`defaultConfig` and `configSchema`)
- Take keys and default values from what the server really writes: run
+67
View File
@@ -0,0 +1,67 @@
# AI builder (gbrgames worker)
The panel's **AI builder** (Admin → Games → AI Config Generator) is run by a worker we host ourselves.
The panel only keeps a queue; every result is a pull request to this repo, which CI validates and
boot-tests like any other.
```
Panel Builder worker (gbrgames worker, Docker)
───── ─────────────────────────────────────────
admin starts a job ──► queued ◄─poll── research: web search + page reader + model
progress / log / variants ◄─post── build: first boot → draft → boot with the node-agent
"Review draft PR" → branch draft/<id>-<job> + PR as the bot ──► CI (validate + boot test)
```
1. **Research** (`gbrgames research -name N`): searches the web, reads the best pages, and has the model list
the server's variants (edition, image or SteamCMD app, startup, ports). Every image and Steam app is
checked for real (`docker manifest inspect`, anonymous `steamcmd +app_info_print`) before the admin
sees it.
2. The admin picks a variant in the panel.
3. **Build** (`gbrgames build -name N -variant @v.json`): first boot of the image untouched (`first-boot`), a
draft by the model (`draft`, rules in `AUTHORING.md`), then the draft is deployed with the node-agent's
`BootTest`, the same check CI runs. Failures (crash, error lines, saves written outside `/server`) go
back to the model, three boots at most.
4. The worker pushes `games/<id>/` to a branch and opens a pull request as the bot account. A draft that
validates but never booted still gets a PR, titled `WIP:`, with what's left listed. Nothing reaches
players until a person merges it and enables the game.
The builder is **game-agnostic**: everything about a game comes from that game's research, first boot and
boot tests. No game's name, id or code may appear in the builder's source, its rules for the model
(`AUTHORING.md`), the schemas or this file, and the model is never shown another game's template (its format
example is an invented server). `TestBuilderIsGameAgnostic` fails the build if one does.
The worker only connects out (to the panel, Gitea, the model and the search services).
## Requirements
A Docker host with the isolated `gbr-boottest` network (`RUNNER.md`, step 1), so servers booted by the
builder can't reach private networks. Enough memory for the largest server it builds (the biggest need ~16 GB) and
disk for SteamCMD downloads. The image `gbr-builder:latest` is published by
`.gitea/workflows/builder-image.yml` whenever `cmd/` changes on main.
## Configuration
All through environment variables; nothing about where it runs lives in this repo.
| Variable | What |
| --- | --- |
| `PANEL_URL` | The panel the worker takes jobs from |
| `BUILDER_TOKEN` | Shared secret the panel checks on `/api/v1/builder/**` (same value in the panel's env) |
| `GITEA_TOKEN` | The bot account's token (repository read/write; the account needs Write on this repo): pushes draft branches, opens PRs as that account. Checked when the worker starts |
| `OLLAMA_URL` | Ollama for the model (or `MODEL_PROVIDER=gemini` with `GEMINI_API_KEY`) |
| `SEARXNG_URL`, `READER_URL` | A SearXNG instance (JSON API) and a page-to-text reader (`READER_URL/<page url>`) |
| `MODEL`, `POLL_INTERVAL`, `WORKER_NAME`, `WORK_DIR` | Optional: model name, poll interval (20s), name shown in the panel, work folder |
| `BOT_NAME` | Optional: commit author when the token can't read its own account (no `read:user` scope); default `gbr-bot` |
## Running a build by hand
On any machine with Docker and the `gbr-boottest` network, with `OLLAMA_URL`, `SEARXNG_URL` and
`READER_URL` set:
```bash
go run ./cmd/gbrgames research -name "<game>" -out research.json
# pick a variant from research.json into v.json, then:
go run ./cmd/gbrgames build -name "<game>" -variant @v.json -out build.json
```
The draft lands in `games/<id>/`, first-boot reports in `firstboot-out/`.
+14
View File
@@ -0,0 +1,14 @@
# The AI builder worker (gbrgames worker); see BUILDER.md.
FROM golang:1.24-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY cmd ./cmd
RUN CGO_ENABLED=0 go build -o /gbrgames ./cmd/gbrgames
FROM alpine:3.22
# docker CLI for first boots and boot tests (through the host's socket), git to clone and open PRs
RUN apk add --no-cache docker-cli git ca-certificates
COPY --from=build /gbrgames /usr/local/bin/gbrgames
WORKDIR /work
ENTRYPOINT ["gbrgames", "worker"]
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// The builder must work the same for every game: no game's name, id or code in its source, its rules for
// the model (AUTHORING.md), the schemas sent with every draft, or its docs. Games are read from games/,
// so a newly merged template is covered as soon as it lands.
func TestBuilderIsGameAgnostic(t *testing.T) {
root := filepath.Join("..", "..")
templates, _ := filepath.Glob(filepath.Join(root, "games", "*", "game.yaml"))
if len(templates) == 0 {
t.Fatal("no templates found to check against")
}
var names []string
for _, f := range templates {
g, err := loadMap(f)
if err != nil {
continue
}
for _, key := range []string{"id", "name"} {
if s, _ := g[key].(string); len(s) > 2 {
names = append(names, s)
}
}
}
files, _ := filepath.Glob("*.go")
files = append(files, filepath.Join(root, "AUTHORING.md"), filepath.Join(root, "BUILDER.md"))
schemas, _ := filepath.Glob(filepath.Join(root, "schema", "*.json"))
files = append(files, schemas...)
for _, f := range files {
raw, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
text := string(raw)
for _, n := range names {
re := regexp.MustCompile(`(?i)(^|[^A-Za-z0-9])` + regexp.QuoteMeta(n) + `($|[^A-Za-z0-9])`)
if loc := re.FindStringIndex(text); loc != nil {
line := strings.Count(text[:loc[0]], "\n") + 1
t.Errorf("%s:%d mentions %q: the builder must not name or copy any game", f, line, n)
}
}
}
}
+401
View File
@@ -0,0 +1,401 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
// build turns one researched variant into a template that has really booted: first boot of the image
// untouched (facts), a draft by the model, then the draft deployed with the node-agent's own code, exactly
// as CI's boot test does. What goes wrong in that boot (a crash, an error line, saves outside the volume)
// goes back to the model, a few rounds at most. The result is a games/<id>/ folder ready for a pull request.
type buildResult struct {
GameID string `json:"gameId"`
Dir string `json:"dir"`
Image string `json:"image"`
Passed bool `json:"passed"`
Rounds int `json:"rounds"`
Verified []string `json:"verified"`
Problems []string `json:"problems"`
Summary string `json:"summary"`
}
const defaultAgentImage = "www.gitea.gbreed.com/gbreed-public/gbr-node-agent:latest"
func runBuild(args []string) error {
fs := flag.NewFlagSet("build", flag.ExitOnError)
gamesDir := fs.String("games", "games", "directory holding one folder per game")
schemaDir := fs.String("schemas", "schema", "directory holding the JSON Schemas")
gameName := fs.String("name", "", "the game's name")
id := fs.String("id", "", "folder/id for the new game (default: from -name)")
variantArg := fs.String("variant", "", "the chosen variant: a research result variant as JSON, or @file")
notes := fs.String("notes", "", "what the admin asked for")
provider := fs.String("provider", "ollama", "ollama or gemini")
modelName := fs.String("model", "", "model name (default as for draft)")
attempts := fs.Int("attempts", 3, "fix rounds for validation problems, each time")
bootRounds := fs.Int("boot-rounds", 3, "boots of the draft before giving up")
out := fs.String("out", "", "write the result JSON here")
fs.Parse(args)
if *gameName == "" || *variantArg == "" {
return errors.New("build needs -name and -variant")
}
raw := []byte(*variantArg)
if strings.HasPrefix(*variantArg, "@") {
var err error
if raw, err = os.ReadFile((*variantArg)[1:]); err != nil {
return err
}
}
var v variant
if err := json.Unmarshal(raw, &v); err != nil {
return fmt.Errorf("-variant: %w", err)
}
if *id == "" {
*id = gameID(*gameName)
}
agentImage := os.Getenv("AGENT_IMAGE")
if agentImage == "" {
agentImage = defaultAgentImage
}
res, err := build(buildOptions{gamesDir: *gamesDir, schemaDir: *schemaDir, name: *gameName, id: *id, variant: v,
notes: *notes, provider: *provider, model: *modelName, attempts: *attempts, bootRounds: *bootRounds, agentImage: agentImage})
if res != nil && *out != "" {
raw, _ := json.MarshalIndent(res, "", " ")
if werr := os.WriteFile(*out, raw, 0o644); werr != nil && err == nil {
err = werr
}
}
if err != nil {
return err
}
if !res.Passed {
return fmt.Errorf("the draft in %s still has problems:\n- %s", res.Dir, strings.Join(res.Problems, "\n- "))
}
fmt.Printf("[build] %s booted with the node-agent and is ready for a pull request\n", res.Dir)
return nil
}
type buildOptions struct {
gamesDir, schemaDir, name, id string
variant variant
notes, provider, model string
attempts, bootRounds int
agentImage string
}
func build(o buildOptions) (*buildResult, error) {
if len(o.variant.Images) == 0 {
return nil, errors.New("the variant has no image to build from")
}
res := &buildResult{GameID: o.id, Dir: filepath.Join(o.gamesDir, o.id)}
res.Verified = append(res.Verified, o.variant.Verified...)
// 1. First boot of each candidate image until one runs as a non-root user
var report *bootReport
for _, image := range o.variant.Images {
steam := isSteamCMDImage(image) && o.variant.SteamAppID != ""
if isSteamCMDImage(image) && !steam {
continue // the SteamCMD image alone runs nothing
}
res.Image = image
if steam && o.variant.StartupHint == "" {
fmt.Printf("[build] no startup command known for Steam app %s; drafting without a first boot\n", o.variant.SteamAppID)
report = nil
break
}
fmt.Printf("[build] first boot of %s\n", image)
r, err := firstBoot(o.id+"-probe", image, o.variant.SteamAppID, steam, o.variant.StartupHint)
if err != nil {
return nil, err
}
if why := imageUnusable(r); why != "" {
fmt.Printf("[build] %s can't be used: %s\n", image, why)
res.Problems = append(res.Problems, why)
res.Image = ""
continue
}
report = r
res.Verified = append(res.Verified, fmt.Sprintf("first boot of %s: %s", image, r.Outcome))
break
}
if res.Image == "" {
return res, errors.New("no image of this variant runs as a non-root user")
}
res.Problems = nil
// 2. Draft until validation and the image checks pass
steamApp := ""
if isSteamCMDImage(res.Image) {
steamApp = o.variant.SteamAppID
}
d, err := newDrafter(draftJob{GamesDir: o.gamesDir, SchemaDir: o.schemaDir, Name: o.name, ID: o.id, Image: res.Image,
SteamApp: steamApp, Notes: variantNotes(o.variant, o.notes), Report: report, Provider: o.provider, Model: o.model, NoPull: true,
Docs: sourceText(o.variant.Sources)})
if err != nil {
return nil, err
}
problems, err := d.refine(o.attempts, nil)
res.Rounds = d.rounds
if err != nil {
return res, err
}
if len(problems) > 0 {
res.Problems = problems
res.Summary = buildSummary(o, res)
return res, nil
}
// 3. Boot the draft the way nodes run it; failures go back to the model
wantOutside := len(outsideDirs(report)) > 0
for boot := 1; boot <= o.bootRounds; boot++ {
fmt.Printf("[build] boot %d of the draft with the node-agent\n", boot)
bootProblems, notes, err := bootCheck(o.gamesDir, o.id, o.agentImage, wantOutside)
if err != nil {
return res, err
}
if len(bootProblems) == 0 {
res.Passed = true
res.Verified = append(res.Verified, notes...)
break
}
fmt.Printf("[build] boot %d failed:\n", boot)
for _, p := range bootProblems {
fmt.Println(" - " + firstLine(p))
}
res.Problems = bootProblems
if boot == o.bootRounds {
break
}
problems, err := d.refine(o.attempts, bootProblems)
res.Rounds = d.rounds
if err != nil {
return res, err
}
if len(problems) > 0 {
res.Problems = problems
break
}
}
if res.Passed {
res.Problems = nil
}
res.Summary = buildSummary(o, res)
return res, nil
}
func isSteamCMDImage(image string) bool {
return image == "cm2network/steamcmd" || strings.HasPrefix(image, "cm2network/steamcmd:")
}
func variantNotes(v variant, admin string) string {
var parts []string
if v.Title != "" {
parts = append(parts, "variant: "+v.Title+" ("+v.Description+")")
}
if v.StartupHint != "" {
parts = append(parts, "documented startup: "+v.StartupHint)
}
if len(v.Ports) > 0 {
parts = append(parts, "documented ports: "+strings.Join(v.Ports, ", "))
}
if v.Notes != "" {
parts = append(parts, v.Notes)
}
if admin != "" {
parts = append(parts, "the admin asked for: "+admin)
}
return strings.Join(parts, "; ")
}
// firstBoot runs first-boot and reads back its report.
func firstBoot(name, image, steamApp string, steam bool, startup string) (*bootReport, error) {
outDir := filepath.Join("firstboot-out", name)
args := []string{"-image", image, "-out", outDir}
if steam {
args = append(args, "-steam-app", steamApp, "-startup", startup)
}
if err := runFirstBoot(args); err != nil {
return nil, err
}
return readBootReport(filepath.Join(outDir, "report.json"))
}
// bootCheck deploys the template with the node-agent's BootTest, the same check CI runs, and when the
// game wrote files outside its volume on the first boot, boots it once more to see that it no longer does.
func bootCheck(gamesDir, id, agentImage string, wantOutside bool) (problems, verified []string, err error) {
self, err := os.Executable()
if err != nil {
return nil, nil, err
}
// A boot test that was killed (worker restarted mid-build) leaves its container and volume behind,
// and the next deploy under the same name would trip over them
runCmd("docker", "rm", "-f", "boottest-"+id)
runCmd("docker", "volume", "rm", "-f", "boottest-"+id+"-data")
payloadCmd := exec.Command(self, "boottest-payload", "-games", gamesDir, id)
var stderr bytes.Buffer
payloadCmd.Stderr = &stderr
payload, err := payloadCmd.Output()
if err != nil {
var exitErr *exec.ExitError
if errors.As(err, &exitErr) && exitErr.ExitCode() == exitMissingSecret {
// The builder has no CI secrets: that's the template's problem to fix, not a reason to stop
return []string{"the boot test can't run: " + strings.TrimSpace(stderr.String()) + ". Remove 'secrets' from " +
"test.yaml; boot tests fill password settings with a throwaway value"}, nil, nil
}
return nil, nil, fmt.Errorf("boottest-payload: %v: %s", err, strings.TrimSpace(stderr.String()))
}
if out, err := runCmd("docker", "pull", "-q", agentImage); err != nil {
return nil, nil, fmt.Errorf("docker pull %s: %v: %s", agentImage, err, out)
}
cmd := exec.Command("docker", "run", "--rm", "-i", "-v", "/var/run/docker.sock:/var/run/docker.sock",
"--entrypoint", "java", agentImage, "-Dloader.main=com.gbreed.gbr.nodeagent.BootTest", "-cp", "app.jar",
"org.springframework.boot.loader.launch.PropertiesLauncher")
cmd.Stdin = bytes.NewReader(payload)
var output bytes.Buffer
cmd.Stdout = &teeWriter{&output}
cmd.Stderr = cmd.Stdout
runErr := cmd.Run()
if runErr != nil {
var exitErr *exec.ExitError
if !errors.As(runErr, &exitErr) {
return nil, nil, runErr
}
return []string{bootFailure(output.String())}, nil, nil
}
verified = append(verified, "boot test with the node-agent: "+bootVerdict(output.String()))
if wantOutside {
fmt.Println("[build] first boot wrote saves outside the volume; booting the draft to see that it no longer does")
r, err := firstBootGame(gamesDir, id)
if err != nil {
return nil, nil, err
}
if dirs := outsideDirs(r); len(dirs) > 0 {
return []string{fmt.Sprintf("booting your template, the server still wrote its own files outside the server volume, "+
"in %s, so your startup command or settings did not move them. Check the exact launch option name and "+
"syntax this server documents for its save/data folder (a wrong option is usually ignored silently) "+
"and point it at /server", strings.Join(dirs, ", "))}, nil, nil
}
verified = append(verified, "the server keeps its files inside /server")
}
return nil, verified, nil
}
func firstBootGame(gamesDir, id string) (*bootReport, error) {
outDir := filepath.Join("firstboot-out", id+"-draft")
if err := runFirstBoot([]string{"-games", gamesDir, "-game", id, "-out", outDir}); err != nil {
return nil, err
}
return readBootReport(filepath.Join(outDir, "report.json"))
}
// bootFailure turns BootTest's output into feedback: its verdict and the server's last lines, which hold
// the actual error (an unknown option, a missing file, a port in use).
func bootFailure(output string) string {
verdict, tail := bootVerdict(output), output
if i := strings.Index(output, "---- last "); i >= 0 {
tail = output[i:]
}
lines := strings.Split(strings.TrimSpace(tail), "\n")
if len(lines) > 60 {
lines = append(lines[:1], lines[len(lines)-59:]...)
}
return "booting your template with the node-agent (the real deploy code) failed: " + verdict +
". Find the cause in the server's output below and fix the template (startup command, settings, " +
"ready pattern, memory):\n" + strings.Join(lines, "\n")
}
func bootVerdict(output string) string {
for _, l := range strings.Split(output, "\n") {
for _, k := range []string{"[boot-test] FAILED: ", "[boot-test] PASSED: "} {
if i := strings.Index(l, k); i >= 0 {
return strings.TrimSpace(l[i+len("[boot-test] "):])
}
}
}
return "no verdict"
}
func firstLine(s string) string {
line, _, _ := strings.Cut(s, "\n")
return line
}
// teeWriter shows the boot as it happens (the worker streams it to the panel) while keeping it for the verdict.
type teeWriter struct{ buf *bytes.Buffer }
func (t *teeWriter) Write(p []byte) (int, error) {
os.Stdout.Write(p)
return t.buf.Write(p)
}
// buildSummary is the pull request's description: where the facts came from and what was really checked.
func buildSummary(o buildOptions, res *buildResult) string {
var b strings.Builder
fmt.Fprintf(&b, "Drafted by the GBR AI builder. **Review before merging**: check the settings, "+
"ports and descriptions against the game's documentation.\n\n")
fmt.Fprintf(&b, "- Game: **%s**", o.name)
if o.variant.Title != "" {
fmt.Fprintf(&b, " — %s", o.variant.Title)
}
fmt.Fprintf(&b, "\n- Image: `%s`", res.Image)
if o.variant.SteamAppID != "" && isSteamCMDImage(res.Image) {
fmt.Fprintf(&b, ", Steam app %s", o.variant.SteamAppID)
}
fmt.Fprintf(&b, "\n- Model rounds: %d\n", res.Rounds)
if len(res.Verified) > 0 {
b.WriteString("\n**Checked for real**\n")
for _, v := range res.Verified {
fmt.Fprintf(&b, "- %s\n", v)
}
}
if len(res.Problems) > 0 {
b.WriteString("\n**Still open** (the builder could not fix these)\n")
for _, p := range res.Problems {
fmt.Fprintf(&b, "- %s\n", firstLine(p))
}
}
if o.variant.Notes != "" {
fmt.Fprintf(&b, "\n**Notes from research:** %s\n", o.variant.Notes)
}
if len(o.variant.Sources) > 0 {
b.WriteString("\n**Sources**\n")
for _, s := range o.variant.Sources {
fmt.Fprintf(&b, "- %s\n", s)
}
}
return b.String()
}
// sourceText reads the pages research based the variant on, in full, so the drafter can check that every
// launch option the model uses is documented there. Empty (check skipped) without a reader.
func sourceText(sources []string) string {
reader := os.Getenv("READER_URL")
if reader == "" || len(sources) == 0 {
fmt.Println("[build] no research sources to check launch options against (READER_URL or sources missing)")
return ""
}
var b strings.Builder
read := 0
for _, s := range sources {
text, err := readPageLimit(reader, s, 200_000)
if err != nil {
fmt.Printf("[build] could not read %s: %v\n", s, err)
continue
}
read++
b.WriteString(text)
b.WriteString("\n")
}
fmt.Printf("[build] read %d of %d research sources to check launch options against\n", read, len(sources))
return b.String()
}
+50
View File
@@ -0,0 +1,50 @@
package main
import (
"strings"
"testing"
)
func TestBootFailureKeepsVerdictAndServerOutput(t *testing.T) {
out := "[boot-test] deploying boottest-example (cm2network/steamcmd)\n" +
"[boot-test] container started; waiting for ready pattern /Server ready/\n" +
"[boot-test] FAILED: container exited before it was ready\n" +
"---- last 2 log lines ----\n" +
"Unknown argument --save-dir\n" +
"Usage: example_server ...\n"
got := bootFailure(out)
if !strings.Contains(got, "failed: FAILED: container exited before it was ready") {
t.Errorf("verdict missing: %s", got)
}
if !strings.Contains(got, "Unknown argument --save-dir") || strings.Contains(got, "deploying boottest") {
t.Errorf("want only the server's last lines, got: %s", got)
}
if v := bootVerdict("[boot-test] PASSED: READY: Server ready"); v != "PASSED: READY: Server ready" {
t.Errorf("verdict = %q", v)
}
}
func TestOutsideDirsSkipsSteamNoise(t *testing.T) {
r := &bootReport{ContainerChanges: []string{
"A /home/steam/.config/example-game/worlds/w.db",
"A /home/steam/Steam/logs/connection_log.txt",
"C /home/steam/.steam/registry.vdf",
}}
got := outsideDirs(r)
if len(got) != 1 || got[0] != "/home/steam/.config/example-game/worlds" {
t.Errorf("got %v", got)
}
if outsideDirs(nil) != nil {
t.Error("nil report should have no dirs")
}
}
func TestVariantNotes(t *testing.T) {
v := variant{Title: "Crossplay", Description: "PC + Xbox", StartupHint: "./example_server -crossplay", Ports: []string{"30000/udp"}}
got := variantNotes(v, "needs Xbox")
for _, want := range []string{"variant: Crossplay (PC + Xbox)", "documented startup: ./example_server -crossplay", "30000/udp", "the admin asked for: needs Xbox"} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in %q", want, got)
}
}
}
+223 -119
View File
@@ -13,9 +13,10 @@ import (
"path"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
"github.com/santhosh-tekuri/jsonschema/v6"
)
// draft writes a first template for a new game with an AI model: it gives the model AUTHORING.md as its
@@ -37,7 +38,7 @@ func runDraft(args []string) error {
fs := flag.NewFlagSet("draft", flag.ExitOnError)
gamesDir := fs.String("games", "games", "directory holding one folder per game")
schemaDir := fs.String("schemas", "schema", "directory holding the JSON Schemas")
gameName := fs.String("name", "", "the game's name, e.g. \"Valheim\"")
gameName := fs.String("name", "", "the game's name, as players know it")
id := fs.String("id", "", "folder/id for the new game (default: from -name)")
image := fs.String("image", "", "Docker image the server runs in")
steamApp := fs.String("steam-app", "", "SteamCMD app id, if the server is installed with SteamCMD")
@@ -52,102 +53,171 @@ func runDraft(args []string) error {
if *gameName == "" || *image == "" {
return errors.New("draft needs -name and -image")
}
if *id == "" {
*id = strings.Trim(regexp.MustCompile(`[^a-z0-9]+`).ReplaceAllString(strings.ToLower(*gameName), "-"), "-")
var report *bootReport
if *reportPath != "" {
var err error
if report, err = readBootReport(*reportPath); err != nil {
return err
}
dir := filepath.Join(*gamesDir, *id)
if _, err := os.Stat(dir); err == nil {
return fmt.Errorf("%s already exists; drafts only create new games", dir)
}
m, err := newModel(*provider, *modelName)
d, err := newDrafter(draftJob{GamesDir: *gamesDir, SchemaDir: *schemaDir, Name: *gameName, ID: *id, Image: *image,
SteamApp: *steamApp, Notes: *notes, Report: report, Provider: *provider, Model: *modelName, NoPull: *noPull})
if err != nil {
return err
}
if problems, err := d.refine(*attempts, nil); err != nil {
return err
} else if len(problems) > 0 {
return fmt.Errorf("draft still has problems after %d fix rounds; it is in %s for a person to finish", *attempts, d.dir)
}
fmt.Println("[draft] next: review it, then boot-test it (build / CI) before opening a PR")
return nil
}
type draftJob struct {
GamesDir, SchemaDir string
Name, ID, Image string
SteamApp, Notes string
Report *bootReport
Provider, Model string
NoPull bool
// Text of the pages research based the variant on; when set, every launch option in the startup
// command must appear in it (or in the image's scripts)
Docs string
}
// drafter holds one conversation with the model about one game, so later problems (a failed boot) can
// be sent back to it with everything it was told before.
type drafter struct {
job draftJob
dir, id string
m model
facts *imageFacts
gameSchema, testSchema *jsonschema.Schema
answerSchema map[string]any
messages []chatMessage
rounds int
}
func newDrafter(j draftJob) (*drafter, error) {
id := j.ID
if id == "" {
id = gameID(j.Name)
}
d := &drafter{job: j, id: id, dir: filepath.Join(j.GamesDir, id)}
if _, err := os.Stat(d.dir); err == nil {
return nil, fmt.Errorf("%s already exists; drafts only create new games", d.dir)
}
var err error
if d.m, err = newModel(j.Provider, j.Model); err != nil {
return nil, err
}
rules, err := os.ReadFile("AUTHORING.md")
if err != nil {
return fmt.Errorf("AUTHORING.md (the model's rules): %w", err)
return nil, fmt.Errorf("AUTHORING.md (the model's rules): %w", err)
}
gameSchema, err := compileSchema(filepath.Join(*schemaDir, "game.schema.json"))
if err != nil {
return err
if d.gameSchema, err = compileSchema(filepath.Join(j.SchemaDir, "game.schema.json")); err != nil {
return nil, err
}
testSchema, err := compileSchema(filepath.Join(*schemaDir, "test.schema.json"))
if err != nil {
return err
if d.testSchema, err = compileSchema(filepath.Join(j.SchemaDir, "test.schema.json")); err != nil {
return nil, err
}
answerSchema, err := draftAnswerSchema(*schemaDir)
if err != nil {
return err
if d.answerSchema, err = draftAnswerSchema(j.SchemaDir); err != nil {
return nil, err
}
if !*noPull {
fmt.Printf("[draft] pulling %s\n", *image)
if out, err := runCmd("docker", "pull", "-q", *image); err != nil {
return fmt.Errorf("docker pull: %v: %s", err, out)
if !j.NoPull {
fmt.Printf("[draft] pulling %s\n", j.Image)
if out, err := runCmd("docker", "pull", "-q", j.Image); err != nil {
return nil, fmt.Errorf("docker pull: %v: %s", err, out)
}
}
facts, err := inspectImage(*image, nil)
if err != nil {
return err
if d.facts, err = inspectImage(j.Image, nil); err != nil {
return nil, err
}
var report *bootReport
if *reportPath != "" {
raw, err := os.ReadFile(*reportPath)
if err != nil {
return err
if why := imageUnusable(j.Report); why != "" {
return nil, errors.New(why)
}
report = &bootReport{}
if err := json.Unmarshal(raw, report); err != nil {
return fmt.Errorf("%s: %w", *reportPath, err)
}
if why := imageUnusable(report); why != "" {
return errors.New(why)
}
}
messages := []chatMessage{
d.messages = []chatMessage{
{Role: "system", Content: draftSystemPrompt(string(rules))},
{Role: "user", Content: draftRequest(*gameName, *id, *image, *steamApp, *notes, facts, report, exampleTemplates(*gamesDir, 2))},
{Role: "user", Content: draftRequest(j.Name, id, j.Image, j.SteamApp, j.Notes, d.facts, j.Report)},
}
return d, nil
}
for round := 0; round <= *attempts; round++ {
fmt.Printf("[draft] round %d: asking %s\n", round+1, m.name())
func gameID(name string) string {
return strings.Trim(regexp.MustCompile(`[^a-z0-9]+`).ReplaceAllString(strings.ToLower(name), "-"), "-")
}
// refine asks for a (corrected) template until it passes validation and the image checks, at most
// attempts+1 answers. feedback, when given, is what went wrong since the last answer (e.g. its boot).
// It returns the problems still left, nil when the draft passes.
func (d *drafter) refine(attempts int, feedback []string) ([]string, error) {
if len(feedback) > 0 {
d.ask(feedback)
}
var problems []string
for round := 0; round <= attempts; round++ {
d.rounds++
fmt.Printf("[draft] round %d: asking %s\n", d.rounds, d.m.name())
start := time.Now()
answer, err := m.chat(messages, answerSchema)
answer, err := d.m.chat(d.messages, d.answerSchema)
if err != nil {
return err
return nil, err
}
fmt.Printf("[draft] answer in %.0fs\n", time.Since(start).Seconds())
messages = append(messages, chatMessage{Role: "assistant", Content: answer})
d.messages = append(d.messages, chatMessage{Role: "assistant", Content: answer})
problems := writeDraft(dir, *id, answer, report)
if len(problems) == 0 {
problems = validateGame(dir, *id, gameSchema, testSchema)
}
if len(problems) == 0 {
if g, err := loadMap(filepath.Join(dir, "game.yaml")); err == nil {
imageProblems, _ := checkTemplate(g, facts)
problems = append(problems, imageProblems...)
t, _ := loadMap(filepath.Join(dir, "test.yaml"))
problems = append(problems, draftCompleteness(g, t)...)
problems = append(problems, unreadSettings(g, facts)...)
problems = append(problems, savesOutsideVolumes(g, report)...)
}
}
if len(problems) == 0 {
fmt.Printf("[draft] %s passes validation and the image checks\n", dir)
fmt.Println("[draft] next: review it, then boot-test it (first-boot / CI) before opening a PR")
return nil
if problems = d.check(answer); len(problems) == 0 {
fmt.Printf("[draft] %s passes validation and the image checks\n", d.dir)
return nil, nil
}
fmt.Printf("[draft] %d problem(s):\n", len(problems))
for _, p := range problems {
fmt.Println(" - " + p)
}
messages = append(messages, chatMessage{Role: "user", Content: "Your template has these problems. Fix all of them and " +
"answer with the complete corrected JSON object (both \"game\" and \"test\"):\n- " + strings.Join(problems, "\n- ")})
if round < attempts {
d.ask(problems)
}
return fmt.Errorf("draft still has problems after %d fix rounds; it is in %s for a person to finish", *attempts, dir)
}
return problems, nil
}
func (d *drafter) ask(problems []string) {
d.messages = append(d.messages, chatMessage{Role: "user", Content: "Your template has these problems. Fix all of them and " +
"answer with the complete corrected JSON object (both \"game\" and \"test\"):\n- " + strings.Join(problems, "\n- ")})
}
func (d *drafter) check(answer string) []string {
problems := writeDraft(d.dir, d.id, answer, d.job.Report)
if len(problems) == 0 {
problems = validateGame(d.dir, d.id, d.gameSchema, d.testSchema)
}
if len(problems) == 0 {
if g, err := loadMap(filepath.Join(d.dir, "game.yaml")); err == nil {
imageProblems, _ := checkTemplate(g, d.facts)
problems = append(problems, imageProblems...)
t, _ := loadMap(filepath.Join(d.dir, "test.yaml"))
problems = append(problems, draftCompleteness(g, t)...)
problems = append(problems, unreadSettings(g, d.facts)...)
problems = append(problems, savesOutsideVolumes(g, d.job.Report)...)
problems = append(problems, splitSettings(g)...)
problems = append(problems, portSettings(g)...)
problems = append(problems, undocumentedOptions(g, d.job.Docs, d.facts)...)
}
}
return problems
}
func readBootReport(p string) (*bootReport, error) {
raw, err := os.ReadFile(p)
if err != nil {
return nil, err
}
r := &bootReport{}
if err := json.Unmarshal(raw, r); err != nil {
return nil, fmt.Errorf("%s: %w", p, err)
}
return r, nil
}
func draftSystemPrompt(rules string) string {
@@ -203,6 +273,11 @@ func draftCompleteness(game, test map[string]any) []string {
} else if v, _ := n.Int64(); v < 1024 {
problems = append(problems, "requiredMemoryMb is below 1024")
}
// Secrets only exist as CI settings a person adds; a draft with one can't be boot-tested anywhere
if secrets, _ := test["secrets"].(map[string]any); len(secrets) > 0 {
problems = append(problems, "test.yaml has 'secrets' ("+strings.Join(sortedKeys(secrets), ", ")+"), which no boot test of a "+
"new game has: remove them. Password settings get a throwaway value in boot tests automatically")
}
if _, ok := game["readyPattern"]; !ok {
if _, ok := test["settleSec"]; !ok {
problems = append(problems, "no readyPattern and no test.settleSec: boot tests need one of them")
@@ -265,26 +340,11 @@ func closestVar(name string, vars []string) string {
var outsideNoise = []string{"/Steam/logs/", "/.steam/", "/steamcmd/", "/.cache/", "/tmp/", "/.local/share/umu"}
// savesOutsideVolumes turns first boot's "written outside the volumes" into a hard check: anything the
// game itself wrote there (Valheim's worlds and admin lists in ~/.config/unity3d/...) is lost every time the
// game itself wrote there (worlds or player lists under the user's home folder) is lost every time the
// node recreates the container. A template passes once its startup command or settings point the game at
// /server or /data; a boot test then shows whether that really worked.
func savesOutsideVolumes(game map[string]any, report *bootReport) []string {
if report == nil {
return nil
}
dirs := map[string]bool{}
for _, c := range report.ContainerChanges {
_, p, _ := strings.Cut(c, " ")
noise := false
for _, n := range outsideNoise {
if strings.Contains(p, n) {
noise = true
}
}
if !noise && p != "" {
dirs[path.Dir(p)] = true
}
}
dirs := outsideDirs(report)
if len(dirs) == 0 {
return nil
}
@@ -302,7 +362,28 @@ func savesOutsideVolumes(game map[string]any, report *bootReport) []string {
return []string{fmt.Sprintf("first boot: the server wrote its own files outside the server volume, in %s. Those are lost "+
"whenever the node recreates the container (every start). Make the game keep them under /server, e.g. with "+
"its save-directory launch option or by pointing that folder at /server, and include it in backupPaths",
strings.Join(sortedKeysBool(dirs), ", "))}
strings.Join(dirs, ", "))}
}
// outsideDirs lists the folders outside the volumes where the game itself wrote files during a boot.
func outsideDirs(report *bootReport) []string {
if report == nil {
return nil
}
dirs := map[string]bool{}
for _, c := range report.ContainerChanges {
_, p, _ := strings.Cut(c, " ")
noise := false
for _, n := range outsideNoise {
if strings.Contains(p, n) {
noise = true
}
}
if !noise && p != "" {
dirs[path.Dir(p)] = true
}
}
return sortedKeysBool(dirs)
}
// imageUnusable recognises a first boot where the image died within seconds because it needs root (or
@@ -325,7 +406,7 @@ func imageUnusable(r *bootReport) string {
*r.ExitCode, r.RanSeconds, r.User, denied[0])
}
func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts, report *bootReport, examples string) string {
func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts, report *bootReport) string {
var b strings.Builder
fmt.Fprintf(&b, "Write the template for **%s**.\n- id: %s\n- defaultDockerImage: %s\n", gameName, id, image)
if steamApp != "" {
@@ -369,44 +450,67 @@ func draftRequest(gameName, id, image, steamApp, notes string, facts *imageFacts
} else {
b.WriteString("\n(No first-boot report: keep readyPattern out unless the facts show the ready line.)\n")
}
b.WriteString("\n## Working templates from this repo, for format and style (not content)\n")
b.WriteString(examples)
b.WriteString("\n## Format example\nAn invented server, to show the format and house style only. Its command, options, " +
"ports and app id belong to no real game: never copy them; take every value from the facts above.\n```yaml\n")
b.WriteString(exampleTemplate)
b.WriteString("```\n")
return b.String()
}
// exampleTemplates picks the shortest templates in the repo as examples: real, validated, and small
// enough to leave the model room for the facts.
func exampleTemplates(gamesDir string, n int) string {
entries, _ := os.ReadDir(gamesDir)
type ex struct {
id string
body string
}
var all []ex
for _, e := range entries {
raw, err := os.ReadFile(filepath.Join(gamesDir, e.Name(), "game.yaml"))
if err == nil {
all = append(all, ex{e.Name(), string(raw)})
}
}
sort.Slice(all, func(i, j int) bool { return len(all[i].body) < len(all[j].body) })
var b strings.Builder
for i := 0; i < n && i < len(all); i++ {
fmt.Fprintf(&b, "\n### games/%s/game.yaml\n```yaml\n%s```\n", all[i].id, all[i].body)
}
return b.String()
}
// exampleTemplate shows the model the format only. It's invented on purpose: the builder must stay
// game-agnostic, so the model never sees another game's template (whose options it would copy). Its
// command, options, port and app id belong to no real server; validated in draft_test.go.
const exampleTemplate = `id: example-server
name: Example Server
defaultDockerImage: cm2network/steamcmd
defaultStartupCommand: args=(-name "${SERVER_NAME}" -world "${WORLD_NAME:-world}" -port 30000 -datadir /server/saves); [ -n "${JOIN_PASSWORD}" ] && args+=(-password "${JOIN_PASSWORD}"); ./example_server "${args[@]}"
requiredMemoryMb: 4096
requiredDiskMb: 8192
defaultPorts:
- 30000/udp
- 30001/udp
installMethod: STEAMCMD
installConfig:
appId: "1000000"
defaultConfig:
JoinPassword: ""
ServerName: GBRHosting - {SERVER_NAME}
WorldName: world
configSchema:
type: object
properties:
JoinPassword:
title: Join Password
description: Password players enter to join. Leave empty for a public server.
type: string
default: ""
category: Security & Access
x-savedValue:
key: GAME_JOIN_PASSWORD
label: Game join password
secret: true
ServerName:
title: Server Name
description: The name shown in the server browser.
type: string
default: GBRHosting - {SERVER_NAME}
category: System & General
showOnDeploy: true
WorldName:
title: World Name
description: The world to load. A new name starts a new world.
type: string
default: world
category: World & Environment
backupPaths:
- saves
readyPattern: 'Server is ready for connections'
`
// writeDraft turns the model's JSON answer into games/<id>/game.yaml and test.yaml, using the importer's
// writer so drafts look like every other template.
func writeDraft(dir, id, answer string, report *bootReport) []string {
answer = strings.TrimSpace(answer)
if i := strings.Index(answer, "{"); i > 0 {
answer = answer[i:] // some models still wrap JSON in prose or code fences
}
if i := strings.LastIndex(answer, "}"); i >= 0 {
answer = answer[:i+1]
}
answer = trimToJSON(answer)
var parsed struct {
Game map[string]any `json:"game"`
Test map[string]any `json:"test"`
+60 -16
View File
@@ -2,14 +2,16 @@ package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestDraftCompletenessNamesMissingSchemaEntries(t *testing.T) {
// The Valheim draft that went round in circles: settings in defaultConfig, no configSchema
// A draft that went round in circles: settings in defaultConfig, no configSchema
game := map[string]any{
"defaultPorts": []any{"2456/udp"},
"defaultPorts": []any{"30000/udp"},
"requiredMemoryMb": json.Number("4096"),
"defaultConfig": map[string]any{"ServerName": "x", "ServerPassword": "", "MaxPlayers": json.Number("8")},
}
@@ -44,28 +46,28 @@ func TestUnreadSettings(t *testing.T) {
}
func TestSavesOutsideVolumes(t *testing.T) {
// Valheim's real first boot: its own files in ~/.config, plus Steam client logs (noise)
valheim := &bootReport{ContainerChanges: []string{
"A /home/steam/.config/unity3d/IronGate/Valheim/adminlist.txt",
"A /home/steam/.config/unity3d/IronGate/Valheim/bannedlist.txt",
"A /home/steam/Steam/logs/connection_log_2456.txt",
// A real first boot: the game's own files in ~/.config, plus Steam client logs (noise)
home := &bootReport{ContainerChanges: []string{
"A /home/steam/.config/example-game/adminlist.txt",
"A /home/steam/.config/example-game/bannedlist.txt",
"A /home/steam/Steam/logs/connection_log_30000.txt",
}}
game := map[string]any{"defaultStartupCommand": "./valheim_server.x86_64 -nographics -batchmode -world X"}
problems := savesOutsideVolumes(game, valheim)
if len(problems) != 1 || !strings.Contains(problems[0], "/home/steam/.config/unity3d/IronGate/Valheim") || strings.Contains(problems[0], "Steam/logs") {
game := map[string]any{"defaultStartupCommand": "./example_server -nographics -batchmode -world X"}
problems := savesOutsideVolumes(game, home)
if len(problems) != 1 || !strings.Contains(problems[0], "/home/steam/.config/example-game") || strings.Contains(problems[0], "Steam/logs") {
t.Fatalf("got %v", problems)
}
// Fixed with a save-directory option under /server
game["defaultStartupCommand"] = "./valheim_server.x86_64 -nographics -batchmode -world X -savedir /server/saves"
if p := savesOutsideVolumes(game, valheim); p != nil {
game["defaultStartupCommand"] = "./example_server -nographics -batchmode -world X -datadir /server/saves"
if p := savesOutsideVolumes(game, home); p != nil {
t.Errorf("redirected template still flagged: %v", p)
}
// ARK's real first boot: only Proton's cache, which is not game data
ark := &bootReport{ContainerChanges: []string{"A /home/gameserver/.cache/umu-protonfixes/protonfixes_test.log"}}
if p := savesOutsideVolumes(map[string]any{}, ark); p != nil {
t.Errorf("ARK proton cache flagged: %v", p)
// A Proton server's first boot: only Proton's cache, which is not game data
proton := &bootReport{ContainerChanges: []string{"A /home/gameserver/.cache/umu-protonfixes/protonfixes_test.log"}}
if p := savesOutsideVolumes(map[string]any{}, proton); p != nil {
t.Errorf("proton cache flagged: %v", p)
}
}
@@ -84,3 +86,45 @@ func TestImageUnusable(t *testing.T) {
t.Errorf("long run flagged: %q", why)
}
}
func TestDraftCompletenessRejectsSecrets(t *testing.T) {
// A draft that couldn't be boot-tested: the password mapped to a CI secret nobody created
test := map[string]any{"settleSec": json.Number("120"), "secrets": map[string]any{"Password": "EXAMPLE_TEST_PASSWORD"}}
problems := strings.Join(draftCompleteness(map[string]any{}, test), "\n")
if !strings.Contains(problems, "test.yaml has 'secrets' (Password)") {
t.Errorf("got:\n%s", problems)
}
}
// The model copies the example's format, so it must pass everything a draft has to pass
func TestExampleTemplateFollowsTheRules(t *testing.T) {
dir := filepath.Join(t.TempDir(), "example-server")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "game.yaml"), []byte(exampleTemplate), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "test.yaml"), []byte("timeoutSec: 2400\n"), 0o644); err != nil {
t.Fatal(err)
}
gs, err := compileSchema(filepath.Join("..", "..", "schema", "game.schema.json"))
if err != nil {
t.Fatal(err)
}
ts, err := compileSchema(filepath.Join("..", "..", "schema", "test.schema.json"))
if err != nil {
t.Fatal(err)
}
problems := validateGame(dir, "example-server", gs, ts)
g, err := loadMap(filepath.Join(dir, "game.yaml"))
if err != nil {
t.Fatal(err)
}
problems = append(problems, draftCompleteness(g, map[string]any{})...)
problems = append(problems, splitSettings(g)...)
problems = append(problems, portSettings(g)...)
for _, p := range problems {
t.Error(p)
}
}
+325
View File
@@ -0,0 +1,325 @@
package main
import (
"encoding/json"
"fmt"
"path"
"regexp"
"sort"
"strconv"
"strings"
)
// Checks a booting draft can still fail: things that only go wrong once an owner changes a setting, so
// neither the validator nor a boot test with the defaults sees them.
// splitSettings finds text settings that reach the server's command line unquoted, so a value with spaces
// ("GBRHosting - My Server") arrives as several arguments and the game only sees the first word. Covers
// direct use (-name ${SERVER_NAME}) and the common ARGS="-name ${SERVER_NAME}"; ./server $ARGS, where the
// quotes in the assignment don't survive the unquoted $ARGS.
func splitSettings(game map[string]any) []string {
startup, _ := game["defaultStartupCommand"].(string)
if startup == "" {
return nil
}
textVars := map[string]string{} // env name -> setting key
for key, p := range schemaProperties(game) {
if prop, _ := p.(map[string]any); prop["type"] == "string" {
textVars[envName(key)] = key
}
}
if len(textVars) == 0 {
return nil
}
s := scanShell(unwrapShell(startup))
split := map[string]string{} // setting key -> how it gets split
var reach func(holder string, seen map[string]bool) []string
reach = func(holder string, seen map[string]bool) []string {
if seen[holder] {
return nil
}
seen[holder] = true
var out []string
for _, v := range s.held[holder] {
if _, ok := textVars[v]; ok {
out = append(out, v)
}
out = append(out, reach(v, seen)...)
}
return out
}
for _, e := range s.uses {
if e.quoted {
continue
}
if key, ok := textVars[e.name]; ok {
split[key] = "${" + e.name + "} is not in double quotes"
}
if len(s.held[e.name]) > 0 {
for _, v := range reach(e.name, map[string]bool{}) {
if _, done := split[textVars[v]]; !done {
split[textVars[v]] = fmt.Sprintf("${%s} is inside $%s, which is used without double quotes", v, e.name)
}
}
}
}
var problems []string
for _, key := range sortedKeysString(split) {
problems = append(problems, fmt.Sprintf("setting %s is split into words on the command line (%s): a value with spaces like "+
"\"GBRHosting - My Server\" reaches the game as several arguments. Quote it where it is used (-name \"${%s}\"), or build "+
"optional arguments in a bash array: args=(-name \"$%s\"); [ -n \"$X\" ] && args+=(-x \"$X\"); ./server \"${args[@]}\"",
key, split[key], envName(key), envName(key)))
}
return problems
}
var shellWrap = regexp.MustCompile(`^\s*(?:ba)?sh\s+-c\s+'(.*)'\s*$`)
// unwrapShell looks inside a startup command that is one bash -c '...', which would otherwise be
// all single-quoted text.
func unwrapShell(cmd string) string {
if m := shellWrap.FindStringSubmatch(cmd); m != nil && !strings.Contains(m[1], "'") {
return m[1]
}
return cmd
}
type shellUse struct {
name string
quoted bool
}
type shellScan struct {
uses []shellUse // expansions where the value is used: words, arguments
held map[string][]string // variable -> variables its assigned value contains
}
var (
assignWord = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_]*)\+?=`)
varName = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*`)
nestedVar = regexp.MustCompile(`\$\{?([A-Za-z_][A-Za-z0-9_]*)`)
)
// scanShell is just enough of a shell parser for splitSettings: quotes, $VAR / ${VAR...} expansions, and
// assignments (NAME=..., NAME+=..., NAME=(...) arrays).
func scanShell(cmd string) *shellScan {
s := &shellScan{held: map[string][]string{}}
var single, double, wordStart = false, false, true
assign, inArray := "", false
record := func(name string, quoted bool) {
switch {
case assign != "" && !inArray:
// NAME="...$X..." : no splitting yet, but $NAME carries X along
s.held[assign] = append(s.held[assign], name)
case inArray && quoted:
s.held[assign] = append(s.held[assign], name) // safe unless the array is expanded unquoted
default:
s.uses = append(s.uses, shellUse{name, quoted})
}
}
for i := 0; i < len(cmd); i++ {
c := cmd[i]
if wordStart && !single && !double {
if m := assignWord.FindString(cmd[i:]); m != "" {
assign = strings.TrimRight(strings.TrimSuffix(m, "="), "+")
i += len(m)
if i < len(cmd) && cmd[i] == '(' {
inArray = true
} else {
i--
}
wordStart = false
continue
}
}
wordStart = false
switch {
case c == '\\' && !single:
i++
case c == '\'' && !double:
single = !single
case c == '"' && !single:
double = !double
case c == '$' && !single && i+1 < len(cmd):
if cmd[i+1] == '{' {
end := matchingBrace(cmd, i+1)
body := cmd[i+2 : end]
name := varName.FindString(strings.TrimLeft(body, "#!"))
if name != "" {
record(name, double)
}
// Defaults can hold more expansions: ${A:-${B}}
for _, m := range nestedVar.FindAllStringSubmatch(body[len(name):], -1) {
record(m[1], double)
}
i = end
} else if name := varName.FindString(cmd[i+1:]); name != "" {
record(name, double)
i += len(name)
}
case single || double:
case inArray && c == ')':
inArray, assign = false, ""
case inArray:
case c == ' ' || c == '\t' || c == '\n' || c == ';' || c == '&' || c == '|' || c == '(' || c == ')':
assign, wordStart = "", true
}
}
return s
}
func matchingBrace(s string, open int) int {
depth := 0
for i := open; i < len(s); i++ {
switch s[i] {
case '{':
depth++
case '}':
depth--
if depth == 0 {
return i
}
}
}
return len(s) - 1
}
var (
portSetting = regexp.MustCompile(`(?i)(port$|(^|_)port(_|$))`)
notAPort = regexp.MustCompile(`(?i)(sup|re|ex|im|trans)port$`)
// Reached over the Docker network, never published (AUTHORING.md)
internalPort = regexp.MustCompile(`(?i)rcon|rest|api|admin|telnet|web|http|metrics`)
)
// portSettings checks settings that set a port players connect to against defaultPorts. The node always
// publishes defaultPorts' container ports (moving only the host side when servers share a node), so a
// game listening anywhere else can't be reached: a default that isn't published is broken from the
// start, and an editable one breaks the moment an owner changes it.
func portSettings(game map[string]any) []string {
published := map[int]bool{}
ports, _ := game["defaultPorts"].([]any)
for _, p := range ports {
n, _ := strconv.Atoi(strings.Split(containerPort(fmt.Sprint(p)), "/")[0])
published[n] = true
}
defaults, _ := game["defaultConfig"].(map[string]any)
props := schemaProperties(game)
var problems []string
keys := make([]string, 0, len(defaults))
for k := range defaults {
keys = append(keys, k)
}
sort.Strings(keys)
for _, key := range keys {
if !portSetting.MatchString(key) || notAPort.MatchString(key) || internalPort.MatchString(key) {
continue
}
port, ok := portNumber(defaults[key])
if !ok {
continue // {PUBLIC_PORT_n} and other placeholders are filled per server
}
_, editable := props[key]
switch {
case !published[port]:
problems = append(problems, fmt.Sprintf("setting %s defaults to port %d, which defaultPorts does not publish: players can't reach it. "+
"Use one of the published ports, or add this port to defaultPorts if players connect to it", key, port))
case editable:
problems = append(problems, fmt.Sprintf("setting %s lets owners change the port, but the node always publishes container port %d "+
"(defaultPorts), so any other value makes the server unreachable. Don't make it a setting: remove %s from "+
"defaultConfig and configSchema and write %d directly in the startup command or config file. If the game must "+
"announce the port players use, give it {PUBLIC_PORT_%d} instead", key, port, key, port, port))
}
}
return problems
}
func portNumber(v any) (int, bool) {
var s string
switch n := v.(type) {
case json.Number:
s = n.String()
case string:
s = n
case int, int64, float64:
s = fmt.Sprint(n)
default:
return 0, false
}
p, err := strconv.Atoi(strings.TrimSpace(s))
return p, err == nil && p > 0 && p < 65536
}
var (
optionToken = regexp.MustCompile(`(?:^|[\s("'])(-{1,2}[A-Za-z][A-Za-z0-9_.]*(?:-[A-Za-z0-9_.]+)*)`)
segmentSep = regexp.MustCompile(`;|&&|\|\||\||\n`)
testOps = map[string]bool{"eq": true, "ne": true, "lt": true, "gt": true, "le": true, "ge": true, "nt": true, "ot": true, "ef": true}
// Options of these are the shell's and its tools', not the game's
shellTools = map[string]bool{"mkdir": true, "cp": true, "mv": true, "ln": true, "rm": true, "chmod": true, "chown": true,
"wget": true, "curl": true, "tar": true, "unzip": true, "test": true, "[": true, "[[": true, "echo": true, "printf": true,
"sleep": true, "find": true, "sed": true, "grep": true, "cd": true, "export": true, "touch": true, "bash": true, "sh": true,
"cat": true, "head": true, "tail": true, "ls": true, "set": true, "ulimit": true, "env": true, "id": true}
)
// undocumentedOptions finds launch options in the startup command that none of the researched pages (nor
// the image's own scripts) mention. Servers silently ignore options they don't know, so a made-up one
// boots fine and passes every boot test while its setting does nothing.
func undocumentedOptions(game map[string]any, docs string, facts *imageFacts) []string {
if strings.TrimSpace(docs) == "" {
return nil
}
startup, _ := game["defaultStartupCommand"].(string)
corpus := docs
if facts != nil {
for _, s := range facts.Scripts {
corpus += "\n" + s
}
}
seen := map[string]bool{}
var missing []string
for _, opt := range launchOptions(unwrapShell(startup)) {
name := strings.TrimLeft(opt, "-")
key := strings.ToLower(name)
if seen[key] {
continue
}
seen[key] = true
documented := regexp.MustCompile(`(?i)(^|[^A-Za-z0-9_])-{1,2}` + regexp.QuoteMeta(name) + `($|[^A-Za-z0-9_])`)
if !documented.MatchString(corpus) {
missing = append(missing, opt)
}
}
if len(missing) == 0 {
return nil
}
return []string{fmt.Sprintf("the startup command passes %s, which none of the pages researched for this game mention. "+
"Servers silently ignore options they don't know, so it boots fine but does nothing: remove it (and any setting that "+
"only feeds it), or spell it exactly as the documentation does", strings.Join(missing, ", "))}
}
// launchOptions lists the -options passed to the game, skipping the shell's own (mkdir -p, [ -n ... ]).
func launchOptions(cmd string) []string {
var out []string
for _, seg := range segmentSep.Split(cmd, -1) {
fields := strings.Fields(strings.TrimLeft(strings.TrimSpace(seg), "({"))
i := 0
for i < len(fields) && (fields[i] == "exec" || fields[i] == "then" || fields[i] == "do" || fields[i] == "else" ||
(assignWord.MatchString(fields[i]) && !strings.Contains(fields[i], "=("))) {
i++
}
if i == len(fields) {
continue
}
if word := path.Base(fields[i]); shellTools[word] {
continue
}
for _, m := range optionToken.FindAllStringSubmatch(strings.Join(fields[i:], " "), -1) {
name := strings.TrimLeft(m[1], "-")
if len(name) < 2 || testOps[name] {
continue
}
out = append(out, m[1])
}
}
return out
}
+151
View File
@@ -0,0 +1,151 @@
package main
import (
"encoding/json"
"strings"
"testing"
)
func textSettings(keys ...string) map[string]any {
props := map[string]any{}
for _, k := range keys {
props[k] = map[string]any{"type": "string"}
}
return map[string]any{"properties": props}
}
func TestSplitSettingsCatchesAnArgsString(t *testing.T) {
// A real draft: quoted in the assignment, split by the unquoted $ARGS
game := map[string]any{
"defaultStartupCommand": `ARGS="-name ${SERVER_NAME:-Example Server} -port 30000 -world ${WORLD_NAME:-Dedicated}"; ` +
`[ -n "$PASSWORD" ] && ARGS="$ARGS -password $PASSWORD"; [ "$CROSSPLAY" = "true" ] && ARGS="$ARGS -crossplay"; ` +
`HOME=/server ./example_server $ARGS`,
"configSchema": textSettings("ServerName", "WorldName", "Password"),
}
problems := splitSettings(game)
got := strings.Join(problems, "\n")
if len(problems) != 3 {
t.Fatalf("want ServerName, WorldName and Password, got:\n%s", got)
}
for _, key := range []string{"setting Password", "setting ServerName", "setting WorldName", "inside $ARGS"} {
if !strings.Contains(got, key) {
t.Errorf("missing %q in:\n%s", key, got)
}
}
}
func TestSplitSettingsAcceptsQuotedAndArrays(t *testing.T) {
for _, startup := range []string{
`./example_server -name "${SERVER_NAME}" -world "$WORLD_NAME" -port 30000`,
`args=(-name "$SERVER_NAME" -world "${WORLD_NAME:-Dedicated}"); [ -n "$PASSWORD" ] && args+=(-password "$PASSWORD"); ./example_server "${args[@]}"`,
`bash -c 'exec ./server -name "$SERVER_NAME"'`,
} {
game := map[string]any{"defaultStartupCommand": startup, "configSchema": textSettings("ServerName", "WorldName", "Password")}
if p := splitSettings(game); len(p) != 0 {
t.Errorf("%s\n flagged: %v", startup, p)
}
}
}
func TestSplitSettingsDirectAndArrayMisuse(t *testing.T) {
cases := map[string]string{
`./server -name ${SERVER_NAME}`: "is not in double quotes",
`bash -c 'exec ./server -name $SERVER_NAME'`: "is not in double quotes",
`args=(-name "$SERVER_NAME"); ./server ${args[@]}`: "inside $args",
`args=(-name $SERVER_NAME); ./server "${args[@]}"`: "is not in double quotes",
`export OPTS="-name ${SERVER_NAME}" && ./run $OPTS`: "inside $OPTS",
}
for startup, want := range cases {
game := map[string]any{"defaultStartupCommand": startup, "configSchema": textSettings("ServerName")}
got := strings.Join(splitSettings(game), "\n")
if !strings.Contains(got, want) {
t.Errorf("%s\n want %q, got %q", startup, want, got)
}
}
// Numbers and booleans can't contain spaces
game := map[string]any{"defaultStartupCommand": `./server -port ${GAME_PORT}`, "configSchema": map[string]any{
"properties": map[string]any{"GamePort": map[string]any{"type": "integer"}}}}
if p := splitSettings(game); len(p) != 0 {
t.Errorf("integer setting flagged: %v", p)
}
}
func TestPortSettings(t *testing.T) {
game := map[string]any{
"defaultPorts": []any{"30000/udp", "30001/udp"},
"defaultConfig": map[string]any{
"GamePort": json.Number("30000"),
"QueryPort": json.Number("27015"),
"RCONPort": json.Number("25575"),
"PublicPort": "{PUBLIC_PORT_30000}",
"Transport": "udp",
"Export": json.Number("9"),
},
"configSchema": map[string]any{"properties": map[string]any{
"GamePort": map[string]any{"type": "integer"}, "QueryPort": map[string]any{"type": "integer"},
}},
}
got := strings.Join(portSettings(game), "\n")
if !strings.Contains(got, "setting GamePort lets owners change the port") || !strings.Contains(got, "write 30000 directly") {
t.Errorf("editable published port not caught:\n%s", got)
}
if !strings.Contains(got, "setting QueryPort defaults to port 27015, which defaultPorts does not publish") {
t.Errorf("unpublished port not caught:\n%s", got)
}
for _, key := range []string{"RCONPort", "PublicPort", "Transport", "Export"} {
if strings.Contains(got, "setting "+key+" ") {
t.Errorf("%s should be left alone:\n%s", key, got)
}
}
// A published port written by the template itself, not offered to owners, is fine
delete(game["configSchema"].(map[string]any)["properties"].(map[string]any), "GamePort")
delete(game["defaultConfig"].(map[string]any), "QueryPort")
if p := portSettings(game); len(p) != 0 {
t.Errorf("got %v", p)
}
}
func TestUndocumentedOptionsCatchesMaxPlayers(t *testing.T) {
// A real draft's startup: every option is documented except -maxplayers
game := map[string]any{"defaultStartupCommand": `args=(-name "${SERVER_NAME:-GBRHosting - Example}" -port 30000 -world "${WORLD_NAME:-Dedicated}" -datadir /server); ` +
`[ -n "${SERVER_PASSWORD}" ] && args+=(-password "${SERVER_PASSWORD}"); [ "${CROSSPLAY}" = "true" ] && args+=(-crossplay); ` +
`[ -n "${MAX_PLAYERS}" ] && args+=(-maxplayers "${MAX_PLAYERS}"); ./example_server "${args[@]}"`}
docs := "Launch with `-name`, `-port`, `-world`, `-password`, `-datadir <path>` and `-crossplay` to allow consoles."
got := undocumentedOptions(game, docs, nil)
if len(got) != 1 || !strings.Contains(got[0], "passes -maxplayers,") {
t.Fatalf("want only -maxplayers flagged, got %v", got)
}
// Once removed, nothing is left to flag
game["defaultStartupCommand"] = strings.Replace(game["defaultStartupCommand"].(string), `[ -n "${MAX_PLAYERS}" ] && args+=(-maxplayers "${MAX_PLAYERS}"); `, "", 1)
if got := undocumentedOptions(game, docs, nil); len(got) != 0 {
t.Errorf("got %v", got)
}
}
func TestUndocumentedOptionsIgnoresShellAndUsesImageScripts(t *testing.T) {
game := map[string]any{"defaultStartupCommand": `mkdir -p /server/.config; cp -f /a /b; [ -n "$X" ] && [ "$Y" -eq 1 ] && ` +
`exec ./server --listen-port=7777 -ServerPlatform=ALL -logs`}
facts := &imageFacts{Scripts: map[string]string{"/start.sh": `exec "$@" -logs`}}
got := strings.Join(undocumentedOptions(game, "Use --listen-port and -ServerPlatform=ALL for crossplay.", facts), "\n")
if got != "" {
t.Errorf("shell options, documented ones and ones the image's script uses must pass, got %q", got)
}
// Without sources the check is off (draft run by hand, or no reader configured)
game["defaultStartupCommand"] = "./server -madeup"
if p := undocumentedOptions(game, "", nil); p != nil {
t.Errorf("got %v", p)
}
if p := undocumentedOptions(game, "unrelated page", nil); len(p) != 1 {
t.Errorf("want -madeup flagged, got %v", p)
}
}
func TestUndocumentedOptionsWholeWords(t *testing.T) {
// "-port" in the docs doesn't document "-portal", and "-port" isn't documented by "-portal"
game := map[string]any{"defaultStartupCommand": "./server -port 1 -portal x"}
got := strings.Join(undocumentedOptions(game, "options: -port", nil), "")
if !strings.Contains(got, "passes -portal,") || strings.Contains(got, "-port,") {
t.Errorf("got %q", got)
}
}
+2 -2
View File
@@ -466,8 +466,8 @@ func (b *booter) copyOut(vol, mount string, files []string, report *bootReport)
return cmd.Wait()
}
// fileLines reads the copied log files, so ready and error lines that never reach the console (ARK
// writes to ShooterGame.log) are found too.
// fileLines reads the copied log files, so ready and error lines that never reach the console (servers
// that only log to a file) are found too.
func (b *booter) fileLines(report *bootReport) []logLine {
var out []logLine
for _, f := range report.FilesCreated {
+14 -14
View File
@@ -6,21 +6,21 @@ import (
)
func TestParseConfigFormats(t *testing.T) {
// Palworld's Unreal-style list
pal := parseConfig("/server/Pal/Saved/Config/LinuxServer/PalWorldSettings.ini", "[/Script/Pal.PalGameWorldSettings]\n"+
`OptionSettings=(Difficulty=None,ExpRate=1.000000,ServerName="Default Palworld Server",CrossplayPlatforms=(Steam,Xbox,PS5,Mac))`)
want := map[string]string{"Difficulty": "None", "ExpRate": "1.000000", "ServerName": "Default Palworld Server", "CrossplayPlatforms": "(Steam,Xbox,PS5,Mac)"}
if len(pal) != len(want) {
t.Fatalf("palworld: got %d settings: %+v", len(pal), pal)
// An Unreal-style option list
list := parseConfig("/server/Game/Saved/Config/LinuxServer/GameSettings.ini", "[/Script/Game.WorldSettings]\n"+
`OptionSettings=(Difficulty=None,ExpRate=1.000000,ServerName="Default Server",CrossplayPlatforms=(Steam,Xbox,PS5,Mac))`)
want := map[string]string{"Difficulty": "None", "ExpRate": "1.000000", "ServerName": "Default Server", "CrossplayPlatforms": "(Steam,Xbox,PS5,Mac)"}
if len(list) != len(want) {
t.Fatalf("option list: got %d settings: %+v", len(list), list)
}
for _, s := range pal {
for _, s := range list {
if s.Section != "OptionSettings" || want[s.Key] != s.Value {
t.Errorf("palworld setting %+v", s)
t.Errorf("option list setting %+v", s)
}
}
// Minecraft's server.properties
mc := parseConfig("/data/server.properties", "#Minecraft server properties\nmax-players=20\nmotd=A Minecraft Server\nlevel-seed=\n")
// A server.properties file
mc := parseConfig("/data/server.properties", "#Server properties\nmax-players=20\nmotd=A Server\nlevel-seed=\n")
if len(mc) != 3 || mc[0].Key != "max-players" || mc[0].Value != "20" || mc[2].Value != "" {
t.Errorf("properties: %+v", mc)
}
@@ -66,7 +66,7 @@ func TestAnalyzeLines(t *testing.T) {
{At: 2, Source: "console", Text: "Failed to load optional module"},
{At: 9, Source: "console", Text: "[09:00:01 INFO]: Done (8.1s)! For help, type \"help\""},
{At: 9, Source: "console", Text: "[09:00:01 INFO]: Thread RCON Listener started"},
// After the stop at 200s: the shutdown, as Minecraft's runner prints it
// After the stop at 200s: the shutdown, as a wrapper script prints it
{At: 201, Source: "console", Text: "[06:01:39] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started"},
{At: 202, Source: "console", Text: "2026-10-25T06:01:39.976Z\tINFO\tmc-server-runner\tDone"},
}, []logLine{{At: -1, Source: "/server/x.log", Text: "Listening on 0.0.0.0:7777"}}, 200)
@@ -91,11 +91,11 @@ func TestAnalyzeLines(t *testing.T) {
func TestConfigCandidates(t *testing.T) {
for rel, want := range map[string]bool{
"server.properties": true,
"Pal/Saved/Config/LinuxServer/x.ini": true,
"ShooterGame/Saved/Logs/ShooterGame.log": true,
"Game/Saved/Config/LinuxServer/x.ini": true,
"Game/Saved/Logs/Server.log": true,
"steamapps/appmanifest_123.acf": false,
"Engine/Config/BaseEngine.ini": false,
"plugins/Geyser/config.yml": false,
"plugins/SomePlugin/config.yml": false,
"world/level.dat": false,
} {
if got := isConfigCandidate(rel, 1000); got != want {
+4 -4
View File
@@ -278,8 +278,8 @@ func parseProbe(f *imageFacts, out string) {
}
// suggestContainerUser works out whether the image only works as a specific user. Images either declare
// it (USER) or are built around an account whose home folder nobody else may enter (ARK's gameserver,
// uid 25000, with /home/gameserver at 700).
// it (USER) or are built around an account whose home folder nobody else may enter (e.g. a home folder
// at mode 700 owned by a uid other than 1000).
func suggestContainerUser(f *imageFacts) {
if u := strings.TrimSpace(f.User); u != "" && u != "root" && u != "0" {
resolved := resolveUser(f, u)
@@ -364,7 +364,7 @@ func checkTemplate(game map[string]any, f *imageFacts) (problems, notes []string
}
if cmd, _ := game["defaultStartupCommand"].(string); cmd != "" {
for _, m := range assignedVars.FindAllStringSubmatch(cmd, -1) {
provided[m[1]] = true // set by the command itself, e.g. export ASA_START_PARAMS=...
provided[m[1]] = true // set by the command itself, e.g. export START_PARAMS=...
}
withDefault := map[string]bool{}
for _, m := range defaultedRef.FindAllStringSubmatch(cmd, -1) {
@@ -410,7 +410,7 @@ func containerPort(p string) string {
return p
}
// startupScriptPaths picks absolute paths out of a template's startup command (e.g. ARK's
// startupScriptPaths picks absolute paths out of a template's startup command (e.g. an image's
// /usr/bin/start_server) so the probe reads those scripts too.
func startupScriptPaths(game map[string]any) []string {
cmd, _ := game["defaultStartupCommand"].(string)
+8 -8
View File
@@ -21,7 +21,7 @@ func TestEnvNameMatchesNodeAgent(t *testing.T) {
}
}
// Probe output as ARK's image produces it (trimmed)
// Probe output of an image built around its own user (trimmed)
const arkProbe = probeMarker + ` passwd
root:x:0:0:root:/root:/bin/bash
gameserver:x:25000:25000::/home/gameserver:/bin/bash
@@ -34,7 +34,7 @@ bash|/usr/bin/bash
` + probeMarker + ` script /usr/bin/start_server
#!/bin/bash
cd /home/gameserver/steamcmd
echo "$ASA_START_PARAMS ${ENABLE_DEBUG} $HOME"
echo "$START_PARAMS ${ENABLE_DEBUG} $HOME"
`
func arkFacts() *imageFacts {
@@ -55,7 +55,7 @@ func TestProbeParsing(t *testing.T) {
if !strings.Contains(f.Scripts["/usr/bin/start_server"], "steamcmd") {
t.Fatalf("script not captured: %q", f.Scripts)
}
if want := []string{"ASA_START_PARAMS", "ENABLE_DEBUG"}; !reflect.DeepEqual(f.EnvRead, want) {
if want := []string{"ENABLE_DEBUG", "START_PARAMS"}; !reflect.DeepEqual(f.EnvRead, want) {
t.Errorf("envRead = %v, want %v (HOME is common and left out)", f.EnvRead, want)
}
}
@@ -63,7 +63,7 @@ func TestProbeParsing(t *testing.T) {
func TestSuggestContainerUser(t *testing.T) {
// Locked home owned by another uid: use that account, with its own group
if f := arkFacts(); f.SuggestedContainerUser != "25000:25000" {
t.Errorf("ARK: suggested %q, want 25000:25000", f.SuggestedContainerUser)
t.Errorf("suggested %q, want 25000:25000", f.SuggestedContainerUser)
}
// uid 1000 images need nothing
@@ -92,9 +92,9 @@ func TestCheckTemplate(t *testing.T) {
f := arkFacts()
game := map[string]any{
"defaultConfig": map[string]any{"RCONPort": 27020, "MaxPlayers": 70},
// Mirrors ARK's old command: ASA_START_PARAMS is assigned, SERVER_MAP has a default, RCON_PORT is the
// A command where START_PARAMS is assigned, SERVER_MAP has a default, RCON_PORT is the
// misnamed one, MAX_PLAYERS is set by a setting, MISSING has no default at all
"defaultStartupCommand": `export ASA_START_PARAMS="${ASA_START_PARAMS:-${SERVER_MAP:-TheIsland}?RCONPort=${RCON_PORT:-27020} -Max=${MAX_PLAYERS:-70} ${MISSING}}"`,
"defaultStartupCommand": `export START_PARAMS="${START_PARAMS:-${SERVER_MAP:-DefaultMap}?RCONPort=${RCON_PORT:-27020} -Max=${MAX_PLAYERS:-70} ${MISSING}}"`,
}
problems, _ := checkTemplate(game, f)
joined := strings.Join(problems, "\n")
@@ -103,7 +103,7 @@ func TestCheckTemplate(t *testing.T) {
t.Errorf("missing problem %q in:\n%s", want, joined)
}
}
for _, unwanted := range []string{"ASA_START_PARAMS", "SERVER_MAP", "MAX_PLAYERS}"} {
for _, unwanted := range []string{"START_PARAMS", "SERVER_MAP", "MAX_PLAYERS}"} {
if strings.Contains(joined, unwanted) {
t.Errorf("false problem about %s in:\n%s", unwanted, joined)
}
@@ -114,7 +114,7 @@ func TestCheckTemplate(t *testing.T) {
// With the fixes, only the deliberately missing variable is left; 25000:1000 counts (same uid)
game["containerUser"] = "25000:1000"
game["defaultStartupCommand"] = `export ASA_START_PARAMS="${ASA_START_PARAMS:-x}?RCONPort=${RCONPORT:-27020}"`
game["defaultStartupCommand"] = `export START_PARAMS="${START_PARAMS:-x}?RCONPort=${RCONPORT:-27020}"`
if problems, _ := checkTemplate(game, f); len(problems) != 0 {
t.Errorf("fixed template still has problems: %v", problems)
}
+10 -1
View File
@@ -7,6 +7,9 @@
// go run ./cmd/gbrgames inspect-image -game ID # read the image's facts and check the template against them
// go run ./cmd/gbrgames first-boot -game ID # start the server once, untouched; record its log and files
// go run ./cmd/gbrgames draft -name N -image I # have an AI model write a new game's template from those facts
// go run ./cmd/gbrgames research -name N # find how a game's server runs: variants, images, Steam app
// go run ./cmd/gbrgames build -name N -variant V # first boot, draft, and boot the draft until it works
// go run ./cmd/gbrgames worker # take research/build jobs from the panel (see BUILDER.md)
package main
import (
@@ -34,6 +37,12 @@ func main() {
err = runFirstBoot(os.Args[2:])
case "draft":
err = runDraft(os.Args[2:])
case "research":
err = runResearch(os.Args[2:])
case "build":
err = runBuild(os.Args[2:])
case "worker":
err = runWorker(os.Args[2:])
default:
usage()
}
@@ -44,6 +53,6 @@ func main() {
}
func usage() {
fmt.Fprintln(os.Stderr, "usage: gbrgames validate [-games DIR] | import [-games DIR] [-all] EXPORT.json | export [-repo R] [-commit SHA] [-out FILE] | boottest-payload [-out FILE] ID | inspect-image [-game ID] [-json] [-no-pull] [IMAGE] | first-boot [-game ID | -image IMAGE] [-env K=V] [-max D] [-quiet D] [-out DIR] | draft -name NAME -image IMAGE [-steam-app ID] [-first-boot REPORT] [-provider ollama|gemini] [-model M]")
fmt.Fprintln(os.Stderr, "usage: gbrgames validate [-games DIR] | import [-games DIR] [-all] EXPORT.json | export [-repo R] [-commit SHA] [-out FILE] | boottest-payload [-out FILE] ID | inspect-image [-game ID] [-json] [-no-pull] [IMAGE] | first-boot [-game ID | -image IMAGE] [-env K=V] [-max D] [-quiet D] [-out DIR] | draft -name NAME -image IMAGE [-steam-app ID] [-first-boot REPORT] [-provider ollama|gemini] [-model M] | research -name NAME [-notes N] [-out FILE] | build -name NAME -variant JSON|@FILE [-out FILE] | worker [-once]")
os.Exit(2)
}
+403
View File
@@ -0,0 +1,403 @@
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"net/url"
"os"
"regexp"
"sort"
"strings"
"time"
)
// research finds out how a game's dedicated server can be run before anything is drafted: it searches the
// web (a self-hosted SearXNG), reads the most useful pages (a reader that turns pages into text), and has
// the model list the server's variants (editions, crossplay builds, docker image or SteamCMD app). Every
// image and Steam app it names is then checked for real, so a variant shown to an admin exists.
type variant struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
Recommended bool `json:"recommended"`
Images []string `json:"images"`
SteamAppID string `json:"steamAppId"`
StartupHint string `json:"startupHint"`
Ports []string `json:"ports"`
Notes string `json:"notes"`
Sources []string `json:"sources"`
// Filled by the checks, not the model
Verified []string `json:"verified,omitempty"`
Problems []string `json:"problems,omitempty"`
}
type researchResult struct {
GameName string `json:"gameName"`
Supported bool `json:"supported"`
Reason string `json:"reason,omitempty"`
Message string `json:"message"`
Variants []variant `json:"variants"`
Sources []string `json:"sources"`
}
var researchSchema = map[string]any{
"type": "object",
"required": []string{"supported", "reason", "message", "variants"},
"properties": map[string]any{
"supported": map[string]any{"type": "boolean"},
"reason": map[string]any{"enum": []string{"ok", "unreleased", "single_player", "no_dedicated_server", "sunset", "unknown"}},
"message": map[string]any{"type": "string"},
"variants": map[string]any{
"type": "array",
"items": map[string]any{
"type": "object",
"required": []string{"id", "title", "description", "recommended", "images", "steamAppId", "startupHint", "ports", "notes", "sources"},
"properties": map[string]any{
"id": map[string]any{"type": "string"},
"title": map[string]any{"type": "string"},
"description": map[string]any{"type": "string"},
"recommended": map[string]any{"type": "boolean"},
"images": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"steamAppId": map[string]any{"type": "string"},
"startupHint": map[string]any{"type": "string"},
"ports": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"notes": map[string]any{"type": "string"},
"sources": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
},
},
},
},
}
func runResearch(args []string) error {
fs := flag.NewFlagSet("research", flag.ExitOnError)
gameName := fs.String("name", "", "the game's name, as players know it")
notes := fs.String("notes", "", "what the admin asked for (crossplay, a modded edition...)")
provider := fs.String("provider", "ollama", "ollama or gemini")
modelName := fs.String("model", "", "model name (default as for draft)")
pages := fs.Int("pages", 6, "how many search results to read")
out := fs.String("out", "", "write the result JSON here (default stdout)")
fs.Parse(args)
if *gameName == "" {
return errors.New("research needs -name")
}
m, err := newModel(*provider, *modelName)
if err != nil {
return err
}
res, err := research(m, *gameName, *notes, *pages)
if err != nil {
return err
}
raw, _ := json.MarshalIndent(res, "", " ")
if *out == "" {
fmt.Println(string(raw))
return nil
}
return os.WriteFile(*out, raw, 0o644)
}
func research(m model, gameName, notes string, pages int) (*researchResult, error) {
searx, reader := os.Getenv("SEARXNG_URL"), os.Getenv("READER_URL")
if searx == "" || reader == "" {
return nil, errors.New("set SEARXNG_URL (e.g. http://host:8080) and READER_URL (e.g. http://host:3000)")
}
queries := []string{
gameName + " dedicated server docker image",
gameName + " dedicated server steamcmd app id linux",
gameName + " dedicated server setup ports launch options",
gameName + " crossplay console dedicated server",
}
var hits []searchHit
for _, q := range queries {
fmt.Printf("[research] searching: %s\n", q)
h, err := searxSearch(searx, q)
if err != nil {
fmt.Printf("[research] search failed: %v\n", err)
continue
}
hits = append(hits, h...)
}
if len(hits) == 0 {
return nil, errors.New("no search results (is SearXNG reachable?)")
}
picked := pickPages(hits, pages)
var b strings.Builder
fmt.Fprintf(&b, "Game: **%s**\n", gameName)
if notes != "" {
fmt.Fprintf(&b, "The admin asked for: %s\n", notes)
}
b.WriteString("\n## Search results\n")
for _, h := range dedupeHits(hits) {
fmt.Fprintf(&b, "- %s — %s: %.200s\n", h.URL, h.Title, h.Content)
}
var sources []string
for _, h := range picked {
fmt.Printf("[research] reading %s\n", h.URL)
text, err := readPage(reader, h.URL)
if err != nil {
fmt.Printf("[research] could not read it: %v\n", err)
continue
}
sources = append(sources, h.URL)
fmt.Fprintf(&b, "\n## Page: %s\n%s\n", h.URL, text)
}
messages := []chatMessage{
{Role: "system", Content: researchPrompt},
{Role: "user", Content: b.String()},
}
fmt.Printf("[research] asking %s\n", m.name())
answer, err := m.chat(messages, researchSchema)
if err != nil {
return nil, err
}
res := &researchResult{}
if err := json.Unmarshal([]byte(trimToJSON(answer)), res); err != nil {
return nil, fmt.Errorf("the model's answer is not valid JSON: %w", err)
}
res.GameName = gameName
res.Sources = sources
verifyVariants(res)
return res, nil
}
const researchPrompt = "You research how to host a game's dedicated server for GBR Hosting, a game server host. Servers run " +
"in Docker on Linux, as a non-root user, with a volume at /server. From the search results and pages given, list " +
"the ways this game's dedicated server can be run (variants): separate editions of the game with their own server, " +
"crossplay builds, or notably different server images. For each variant give:\n" +
"- images: Docker Hub images that run this server, best first (prefer images that run as a non-root user, and the " +
"plain SteamCMD route cm2network/steamcmd when the game ships a Linux dedicated server on Steam). Only images that " +
"appear in the pages.\n" +
"- steamAppId: the Steam app id of the DEDICATED SERVER (not the game itself), if installed via SteamCMD; else \"\".\n" +
"- startupHint: the command that starts the server from its install folder /server, with the launch options " +
"for name, port, world and save location exactly as the pages document them; \"\" if unknown.\n" +
"- ports: every port players connect to, as \"port/proto\".\n" +
"- notes: anything a template author must know (Windows-only server needing Proton, crossplay flags, mods).\n" +
"- sources: the page URLs that support it.\n" +
"Mark one variant recommended. Use only what the pages say or what you know for certain; leave fields empty " +
"instead of guessing. If no dedicated server can be hosted (single-player only, unreleased, official servers " +
"only, shut down), set supported false, the matching reason, no variants, and explain in message. Otherwise " +
"reason \"ok\" and a one-sentence message."
type searchHit struct {
URL string `json:"url"`
Title string `json:"title"`
Content string `json:"content"`
}
func searxSearch(base, query string) ([]searchHit, error) {
u := strings.TrimRight(base, "/") + "/search?format=json&q=" + url.QueryEscape(query)
raw, err := httpGet(u, 30*time.Second)
if err != nil {
return nil, err
}
var resp struct {
Results []searchHit `json:"results"`
}
if err := json.Unmarshal(raw, &resp); err != nil {
return nil, fmt.Errorf("unreadable search answer: %.200s", raw)
}
if len(resp.Results) > 8 {
resp.Results = resp.Results[:8]
}
return resp.Results, nil
}
// Pages that hold the facts a template needs, read first
var usefulPage = regexp.MustCompile(`(?i)hub\.docker\.com|github\.com|steamdb\.info|developer\.valvesoftware|wiki|fandom|reddit\.com/r/\w*(server|admin)`)
// pickPages reads one page per site, docs and image pages first, so a handful of reads covers the most ground.
func pickPages(hits []searchHit, n int) []searchHit {
hits = dedupeHits(hits)
sort.SliceStable(hits, func(i, j int) bool {
return usefulPage.MatchString(hits[i].URL) && !usefulPage.MatchString(hits[j].URL)
})
seen := map[string]bool{}
var out []searchHit
for _, h := range hits {
u, err := url.Parse(h.URL)
if err != nil || seen[u.Host] {
continue
}
seen[u.Host] = true
out = append(out, h)
if len(out) == n {
break
}
}
return out
}
func dedupeHits(hits []searchHit) []searchHit {
seen := map[string]bool{}
var out []searchHit
for _, h := range hits {
if h.URL == "" || seen[h.URL] {
continue
}
seen[h.URL] = true
out = append(out, h)
}
return out
}
func readPage(reader, pageURL string) (string, error) {
return readPageLimit(reader, pageURL, 9000)
}
// readPageLimit reads a page as text through the reader, cut to limit bytes.
func readPageLimit(reader, pageURL string, limit int) (string, error) {
raw, err := httpGet(strings.TrimRight(reader, "/")+"/"+pageURL, 60*time.Second)
if err != nil {
return "", err
}
text := string(raw)
if len(text) > limit {
text = text[:limit] + "\n...(truncated)"
}
return text, nil
}
func httpGet(u string, timeout time.Duration) ([]byte, error) {
client := &http.Client{Timeout: timeout}
resp, err := client.Get(u)
if err != nil {
return nil, err
}
defer resp.Body.Close()
raw, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, err
}
if resp.StatusCode >= 300 {
return nil, fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
return raw, nil
}
// trimToJSON cuts prose or code fences some models put around a JSON answer.
func trimToJSON(answer string) string {
answer = strings.TrimSpace(answer)
if i := strings.Index(answer, "{"); i > 0 {
answer = answer[i:]
}
if i := strings.LastIndex(answer, "}"); i >= 0 {
answer = answer[:i+1]
}
return answer
}
// verifyVariants checks every image and Steam app the model named: images must exist in their registry,
// Steam apps must exist and be readable anonymously (what SteamCMD installs need). What fails is dropped
// and noted; a variant left with nothing runnable can't be recommended.
func verifyVariants(res *researchResult) {
var apps []string
for _, v := range res.Variants {
if v.SteamAppID != "" {
apps = append(apps, v.SteamAppID)
}
}
names := steamAppNames(apps)
anyRunnable := false
for i := range res.Variants {
v := &res.Variants[i]
var images []string
for _, img := range v.Images {
img = strings.TrimSpace(img)
if img == "" {
continue
}
if out, err := runCmd("docker", "manifest", "inspect", img); err != nil {
v.Problems = append(v.Problems, fmt.Sprintf("image %s not found: %.120s", img, out))
continue
}
v.Verified = append(v.Verified, "image "+img+" exists")
images = append(images, img)
}
v.Images = images
if v.SteamAppID != "" {
if name, ok := names[v.SteamAppID]; ok {
v.Verified = append(v.Verified, fmt.Sprintf("Steam app %s is %q", v.SteamAppID, name))
if !hasImage(v.Images, "cm2network/steamcmd") {
v.Images = append(v.Images, "cm2network/steamcmd")
}
} else {
v.Problems = append(v.Problems, "Steam app "+v.SteamAppID+" not found for anonymous SteamCMD")
v.SteamAppID = ""
}
}
if len(v.Images) == 0 {
v.Problems = append(v.Problems, "no image or Steam app that could be checked; it can't be built")
v.Recommended = false
} else {
anyRunnable = true
}
}
if !anyRunnable && res.Supported {
res.Supported = false
res.Reason = "unknown"
res.Message = "No variant has a Docker image or Steam app that could be confirmed. " + res.Message
}
// Exactly one recommendation, among the runnable ones
rec := -1
for i, v := range res.Variants {
if v.Recommended && rec < 0 && len(v.Images) > 0 {
rec = i
}
res.Variants[i].Recommended = false
}
if rec < 0 {
for i, v := range res.Variants {
if len(v.Images) > 0 {
rec = i
break
}
}
}
if rec >= 0 {
res.Variants[rec].Recommended = true
}
}
func hasImage(images []string, name string) bool {
for _, i := range images {
if i == name || strings.HasPrefix(i, name+":") {
return true
}
}
return false
}
var appName = regexp.MustCompile(`"common"\s*\{[^}]*?"name"\s+"([^"]+)"`)
// steamAppNames asks SteamCMD (anonymously, in its official image) for each app's info: an app that
// anonymous SteamCMD can't see can't be installed by the node-agent either.
func steamAppNames(apps []string) map[string]string {
names := map[string]string{}
if len(apps) == 0 {
return names
}
for _, app := range apps {
if !regexp.MustCompile(`^\d+$`).MatchString(app) {
continue
}
out, err := runCmd("docker", "run", "--rm", "--network", bootTestNetwork, "cm2network/steamcmd", "bash", "-c",
"/home/steam/steamcmd/steamcmd.sh +login anonymous +app_info_update 1 +app_info_print "+app+" +quit")
if err != nil {
fmt.Printf("[research] steamcmd for app %s: %v\n", app, err)
continue
}
if m := appName.FindStringSubmatch(out); m != nil {
names[app] = m[1]
}
}
return names
}
+41
View File
@@ -0,0 +1,41 @@
package main
import "testing"
func TestPickPagesPrefersDocsAndOnePerSite(t *testing.T) {
hits := []searchHit{
{URL: "https://blog.example.com/example-server"},
{URL: "https://github.com/a/example-docker"},
{URL: "https://github.com/b/other"},
{URL: "https://hub.docker.com/r/x/example"},
{URL: "https://blog.example.com/example-server"},
}
got := pickPages(hits, 3)
if len(got) != 3 {
t.Fatalf("got %d pages", len(got))
}
if got[0].URL != "https://github.com/a/example-docker" || got[1].URL != "https://hub.docker.com/r/x/example" || got[2].URL != "https://blog.example.com/example-server" {
t.Errorf("order: %v", got)
}
}
func TestTrimToJSON(t *testing.T) {
if got := trimToJSON("Here you go:\n```json\n{\"a\": 1}\n```"); got != `{"a": 1}` {
t.Errorf("got %q", got)
}
}
func TestAppNameFromAppInfo(t *testing.T) {
out := `"1000000"
{
"common"
{
"name" "Example Dedicated Server"
"type" "Tool"
}
}`
m := appName.FindStringSubmatch(out)
if m == nil || m[1] != "Example Dedicated Server" {
t.Errorf("got %v", m)
}
}
+2 -1
View File
@@ -168,7 +168,8 @@ func validateGame(dir, folder string, gameSchema, testSchema *jsonschema.Schema)
add("test.yaml config.%s is not a key of this game", key)
}
if sensitiveKey.MatchString(key) {
add("test.yaml config.%s looks like a credential; map it under 'secrets' to a Gitea secret instead", key)
add("test.yaml config.%s looks like a credential; remove it (boot tests fill password settings with a throwaway "+
"value), or, only if the server can't start without a real one, map it under 'secrets' to a CI secret", key)
}
}
for _, key := range sortedKeys(secrets) {
+491
View File
@@ -0,0 +1,491 @@
package main
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"sync"
"time"
)
// worker runs the AI builder. It asks the panel for queued jobs (it only connects out,
// like node-agents do), runs research or build in a fresh clone of this repo, streams the output to the
// panel as the job's log, and finishes a build with a pull request opened as the bot account. CI then
// validates and boot-tests that pull request like any other.
//
// Panel API (headers X-Builder-Token and X-Builder-Worker):
//
// POST /api/v1/builder/jobs/claim {worker} -> 200 builderJob | 204 nothing queued
// POST /api/v1/builder/jobs/{id}/progress {stage, lines} -> 200 {cancelled}
// POST /api/v1/builder/jobs/{id}/result {research | build+prUrl | error}
type builderJob struct {
ID string `json:"id"`
Kind string `json:"kind"` // "research" or "build"
GameName string `json:"gameName"`
Notes string `json:"notes"`
Variant json.RawMessage `json:"variant"`
}
type jobResult struct {
Research *researchResult `json:"research,omitempty"`
Build *buildResult `json:"build,omitempty"`
PRURL string `json:"prUrl,omitempty"`
Error string `json:"error,omitempty"`
}
type workerConfig struct {
panel, token string
gitea, repo, botToken string
workDir, name string
poll time.Duration
provider, model string
botLogin, botEmail string // the token's account, used as the commit author
}
// checkBot reads which account GITEA_TOKEN belongs to and makes sure it can push to the games repo, so a
// wrong token or missing permission shows when the worker starts, not at the end of a 30-minute build.
func (c *workerConfig) checkBot() error {
var user struct {
Login string `json:"login"`
Email string `json:"email"`
}
if err := c.giteaGet("/user", &user); err != nil {
if !strings.HasPrefix(err.Error(), "HTTP 403") {
return fmt.Errorf("GITEA_TOKEN: %w", err)
}
// A repository-only token (all the worker needs) may not read its own account: name the author from
// BOT_NAME instead. Whether it can push is still checked below.
user.Login = envOr("BOT_NAME", "gbr-bot")
}
var repo struct {
Permissions struct {
Push bool `json:"push"`
} `json:"permissions"`
}
if err := c.giteaGet("/repos/"+c.repo, &repo); err != nil {
return fmt.Errorf("%s: %w", c.repo, err)
}
if !repo.Permissions.Push {
return fmt.Errorf("the account %s can't push to %s: give it Write access there", user.Login, c.repo)
}
c.botLogin, c.botEmail = user.Login, user.Email
if c.botEmail == "" {
c.botEmail = user.Login + "@users.noreply"
}
return nil
}
func (c *workerConfig) giteaGet(p string, out any) error {
req, err := http.NewRequest(http.MethodGet, c.gitea+"/api/v1"+p, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "token "+c.botToken)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 300 {
raw, _ := io.ReadAll(resp.Body)
return fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
return json.NewDecoder(resp.Body).Decode(out)
}
func runWorker(args []string) error {
fs := flag.NewFlagSet("worker", flag.ExitOnError)
once := fs.Bool("once", false, "handle at most one job, then exit")
fs.Parse(args)
host, _ := os.Hostname()
c := workerConfig{
panel: strings.TrimRight(os.Getenv("PANEL_URL"), "/"), token: os.Getenv("BUILDER_TOKEN"),
gitea: strings.TrimRight(envOr("GITEA_URL", "https://www.gitea.gbreed.com"), "/"), repo: envOr("GAMES_REPO", "gbreed-public/gbr-games"),
botToken: os.Getenv("GITEA_TOKEN"), workDir: envOr("WORK_DIR", "/work"), name: envOr("WORKER_NAME", host),
provider: envOr("MODEL_PROVIDER", "ollama"), model: os.Getenv("MODEL"),
}
var err error
if c.poll, err = time.ParseDuration(envOr("POLL_INTERVAL", "20s")); err != nil {
return err
}
for _, v := range []struct{ name, value string }{{"PANEL_URL", c.panel}, {"BUILDER_TOKEN", c.token}, {"GITEA_TOKEN", c.botToken}} {
if v.value == "" {
return fmt.Errorf("set %s", v.name)
}
}
if err := c.checkBot(); err != nil {
return err
}
fmt.Printf("[worker] %s polling %s every %s; pull requests as %s\n", c.name, c.panel, c.poll, c.botLogin)
for {
job, err := c.claim()
if err != nil {
fmt.Printf("[worker] claim: %v\n", err)
}
if job != nil {
fmt.Printf("[worker] job %s: %s %q\n", job.ID, job.Kind, job.GameName)
res := c.handle(job)
if err := c.post("/jobs/"+job.ID+"/result", res, nil); err != nil {
fmt.Printf("[worker] posting the result of %s: %v\n", job.ID, err)
}
if *once {
return nil
}
continue
}
if *once {
return nil
}
time.Sleep(c.poll)
}
}
func envOr(name, def string) string {
if v := os.Getenv(name); v != "" {
return v
}
return def
}
func (c *workerConfig) claim() (*builderJob, error) {
var job builderJob
status, err := c.request("/jobs/claim", map[string]string{"worker": c.name}, &job)
if err != nil || status == http.StatusNoContent {
return nil, err
}
return &job, nil
}
func (c *workerConfig) post(p string, body, out any) error {
_, err := c.request(p, body, out)
return err
}
func (c *workerConfig) request(p string, body, out any) (int, error) {
payload, err := json.Marshal(body)
if err != nil {
return 0, err
}
req, err := http.NewRequest(http.MethodPost, c.panel+"/api/v1/builder"+p, bytes.NewReader(payload))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-Builder-Token", c.token)
req.Header.Set("X-Builder-Worker", c.name)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return 0, err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode >= 300 {
return resp.StatusCode, fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
if out != nil && len(bytes.TrimSpace(raw)) > 0 {
if err := json.Unmarshal(raw, out); err != nil {
return resp.StatusCode, err
}
}
return resp.StatusCode, nil
}
// handle runs one job in its own clone and returns what to report; it never panics the worker.
func (c *workerConfig) handle(job *builderJob) *jobResult {
dir := filepath.Join(c.workDir, "job-"+job.ID)
os.RemoveAll(dir)
defer os.RemoveAll(dir)
if out, err := runCmd("git", "clone", "-q", "--depth", "1", c.gitea+"/"+c.repo+".git", dir); err != nil {
return &jobResult{Error: "git clone: " + out}
}
self, err := os.Executable()
if err != nil {
return &jobResult{Error: err.Error()}
}
resultFile := filepath.Join(dir, "job-result.json")
modelArgs := []string{"-provider", c.provider}
if c.model != "" {
modelArgs = append(modelArgs, "-model", c.model)
}
switch job.Kind {
case "research":
args := append([]string{"research", "-name", job.GameName, "-notes", job.Notes, "-out", resultFile}, modelArgs...)
if err := c.runStreamed(job.ID, dir, self, args, "RESEARCH"); err != nil {
return &jobResult{Error: err.Error()}
}
res := &researchResult{}
if err := readJSON(resultFile, res); err != nil {
return &jobResult{Error: err.Error()}
}
return &jobResult{Research: res}
case "build":
variantFile := filepath.Join(dir, "job-variant.json")
if err := os.WriteFile(variantFile, job.Variant, 0o644); err != nil {
return &jobResult{Error: err.Error()}
}
args := append([]string{"build", "-name", job.GameName, "-notes", job.Notes, "-variant", "@" + variantFile, "-out", resultFile}, modelArgs...)
runErr := c.runStreamed(job.ID, dir, self, args, "FIRST_BOOT")
res := &buildResult{}
if err := readJSON(resultFile, res); err != nil {
msg := "the build produced no result"
if runErr != nil {
msg = runErr.Error()
}
return &jobResult{Error: msg}
}
// A draft that validates but didn't boot still goes up as a draft PR: a person can finish it, and
// CI shows the same boot failure
if _, err := os.Stat(filepath.Join(dir, res.Dir, "game.yaml")); err != nil {
return &jobResult{Build: res, Error: "no template was written: " + strings.Join(res.Problems, "; ")}
}
c.progress(job.ID, "OPENING_PR", []string{"[worker] opening the pull request"})
pr, err := c.openPR(dir, job, res)
if err != nil {
return &jobResult{Build: res, Error: "opening the pull request: " + err.Error()}
}
return &jobResult{Build: res, PRURL: pr}
}
return &jobResult{Error: "unknown job kind " + job.Kind}
}
// Output prefixes that mark a new stage of a build, shown in the panel's progress bar
var stagePrefixes = []struct{ prefix, stage string }{
{"[build] first boot", "FIRST_BOOT"},
{"[first-boot]", ""},
{"[draft]", "DRAFTING"},
{"[build] boot ", "BOOT_TEST"},
}
// runStreamed runs a gbrgames command and sends its output to the panel every few seconds (which also
// tells the panel the worker is alive). The panel can answer "cancelled" to stop the job.
func (c *workerConfig) runStreamed(jobID, dir, self string, args []string, stage string) error {
cmd := exec.Command(self, args...)
cmd.Dir = dir
pr, pw := io.Pipe()
cmd.Stdout, cmd.Stderr = pw, pw
if err := cmd.Start(); err != nil {
return err
}
var mu sync.Mutex
var pending []string
var lastErr string
done := make(chan struct{})
go func() {
sc := bufio.NewScanner(pr)
sc.Buffer(make([]byte, 1024*1024), 1024*1024)
for sc.Scan() {
line := strings.TrimRight(ansi.ReplaceAllString(sc.Text(), ""), "\r ")
fmt.Println(line)
mu.Lock()
pending = append(pending, line)
for _, s := range stagePrefixes {
if s.stage != "" && strings.HasPrefix(line, s.prefix) {
stage = s.stage
}
}
if strings.HasPrefix(line, "error: ") {
lastErr = strings.TrimPrefix(line, "error: ")
}
mu.Unlock()
}
close(done)
}()
wait := make(chan error, 1)
go func() {
err := cmd.Wait()
pw.Close()
<-done
wait <- err
}()
tick := time.NewTicker(5 * time.Second)
defer tick.Stop()
flush := func() bool {
mu.Lock()
lines, st := pending, stage
pending = nil
mu.Unlock()
return c.progress(jobID, st, lines)
}
for {
select {
case err := <-wait:
flush()
if err != nil {
if lastErr != "" {
return errors.New(lastErr)
}
return err
}
return nil
case <-tick.C:
if cancelled := flush(); cancelled {
cmd.Process.Kill()
<-wait
return errors.New("cancelled from the panel")
}
}
}
}
// progress sends log lines (possibly none, as a heartbeat) and reports whether the job was cancelled.
func (c *workerConfig) progress(jobID, stage string, lines []string) bool {
var resp struct {
Cancelled bool `json:"cancelled"`
}
if lines == nil {
lines = []string{}
}
if err := c.post("/jobs/"+jobID+"/progress", map[string]any{"stage": stage, "lines": lines}, &resp); err != nil {
fmt.Printf("[worker] progress: %v\n", err)
}
return resp.Cancelled
}
// openPR commits the new game folder to the game's draft branch as the bot and opens a pull request to
// main. A game has one draft at a time: an earlier draft's PR is closed and its branch deleted first.
func (c *workerConfig) openPR(dir string, job *builderJob, res *buildResult) (string, error) {
branch := draftBranch(res.GameID)
replaced, err := c.replaceOldDrafts(res.GameID)
if err != nil {
return "", fmt.Errorf("replacing the earlier draft: %w", err)
}
summary := res.Summary
for _, n := range replaced {
summary += fmt.Sprintf("\nReplaces #%d (closed, branch deleted).\n", n)
}
auth := "http.extraHeader=Authorization: token " + c.botToken
steps := []struct {
name string
args []string
}{
{"checkout", []string{"checkout", "-q", "-b", branch}},
{"add", []string{"add", filepath.ToSlash(res.Dir)}},
{"commit", []string{"-c", "user.name=" + c.botLogin, "-c", "user.email=" + c.botEmail, "commit", "-q", "-m",
fmt.Sprintf("feat(games): draft %s\n\nDrafted and boot-tested by the GBR builder (job %s).", job.GameName, job.ID)}},
// Forced: the branch is the bot's own, and whatever was on it was the draft being replaced
{"push", []string{"-c", auth, "push", "-q", "--force", "origin", branch}},
}
for _, s := range steps {
cmd := exec.Command("git", s.args...)
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil {
// The push command holds the token: name the step only, and mask it in git's output
return "", fmt.Errorf("git %s: %s", s.name, strings.ReplaceAll(string(out), c.botToken, "***"))
}
}
title := "feat(games): add " + job.GameName
if !res.Passed {
title = "WIP: " + title + " (did not boot yet)"
}
var pr struct {
HTMLURL string `json:"html_url"`
}
body := map[string]any{"title": title, "head": branch, "base": "main", "body": summary}
if err := c.giteaCall(http.MethodPost, "/repos/"+c.repo+"/pulls", body, &pr); err != nil {
return "", err
}
return pr.HTMLURL, nil
}
// replaceOldDrafts closes the game's open draft PRs and deletes their branches, so a rebuild leaves one
// draft to review instead of a pile. Only draft/ branches for this game are touched, never anyone's other
// work. Returns the numbers of the PRs it closed.
func (c *workerConfig) replaceOldDrafts(gameID string) ([]int, error) {
var pulls []struct {
Number int `json:"number"`
Head struct {
Ref string `json:"ref"`
} `json:"head"`
}
if err := c.giteaCall(http.MethodGet, "/repos/"+c.repo+"/pulls?state=open&limit=50", nil, &pulls); err != nil {
return nil, err
}
branches := map[string]bool{draftBranch(gameID): true}
var closed []int
for _, p := range pulls {
if !isDraftBranchOf(p.Head.Ref, gameID) {
continue
}
if err := c.giteaCall(http.MethodPatch, fmt.Sprintf("/repos/%s/pulls/%d", c.repo, p.Number), map[string]string{"state": "closed"}, nil); err != nil {
return closed, err
}
fmt.Printf("[worker] closed #%d, the earlier draft of %s\n", p.Number, gameID)
closed = append(closed, p.Number)
branches[p.Head.Ref] = true
}
for b := range branches {
// Gone already is fine: not every game has had a draft
if err := c.giteaCall(http.MethodDelete, "/repos/"+c.repo+"/branches/"+url.PathEscape(b), nil, nil); err != nil && !strings.HasPrefix(err.Error(), "HTTP 404") {
return closed, fmt.Errorf("deleting branch %s: %w", b, err)
}
}
return closed, nil
}
func draftBranch(gameID string) string {
return "draft/" + gameID
}
// isDraftBranchOf matches draft/<id>, and draft/<id>-<job> as the worker named them before.
func isDraftBranchOf(ref, gameID string) bool {
d := draftBranch(gameID)
if ref == d {
return true
}
rest, ok := strings.CutPrefix(ref, d+"-")
return ok && regexp.MustCompile(`^[0-9a-f]{8}$`).MatchString(rest)
}
// giteaCall sends one API call as the bot; out may be nil.
func (c *workerConfig) giteaCall(method, p string, body, out any) error {
var rd io.Reader
if body != nil {
payload, err := json.Marshal(body)
if err != nil {
return err
}
rd = bytes.NewReader(payload)
}
req, err := http.NewRequest(method, c.gitea+"/api/v1"+p, rd)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "token "+c.botToken)
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode >= 300 {
return fmt.Errorf("HTTP %d: %.200s", resp.StatusCode, raw)
}
if out != nil && len(bytes.TrimSpace(raw)) > 0 {
return json.Unmarshal(raw, out)
}
return nil
}
func readJSON(p string, out any) error {
raw, err := os.ReadFile(p)
if err != nil {
return err
}
return json.Unmarshal(raw, out)
}
+115
View File
@@ -0,0 +1,115 @@
package main
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestCheckBotNeedsPushAccess(t *testing.T) {
push := false
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "token secret" {
w.WriteHeader(http.StatusUnauthorized)
return
}
switch r.URL.Path {
case "/api/v1/user":
w.Write([]byte(`{"login":"git_ai_bot","email":"bot@example.com"}`))
case "/api/v1/repos/org/games":
if push {
w.Write([]byte(`{"permissions":{"push":true}}`))
} else {
w.Write([]byte(`{"permissions":{"push":false}}`))
}
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer srv.Close()
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
if err := c.checkBot(); err == nil || !strings.Contains(err.Error(), "git_ai_bot can't push to org/games") {
t.Errorf("read-only bot: got %v", err)
}
push = true
if err := c.checkBot(); err != nil {
t.Fatal(err)
}
if c.botLogin != "git_ai_bot" || c.botEmail != "bot@example.com" {
t.Errorf("identity = %s <%s>", c.botLogin, c.botEmail)
}
c.botToken = "wrong"
if err := c.checkBot(); err == nil || !strings.Contains(err.Error(), "GITEA_TOKEN") {
t.Errorf("bad token: got %v", err)
}
}
func TestCheckBotWithRepositoryOnlyToken(t *testing.T) {
// Gitea answers /user with 403 when the token only has repository scope
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/v1/user":
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(`{"message":"token does not have at least one of required scope(s), required=[read:user]"}`))
case "/api/v1/repos/org/games":
w.Write([]byte(`{"permissions":{"push":true}}`))
}
}))
defer srv.Close()
t.Setenv("BOT_NAME", "gbr-bot")
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
if err := c.checkBot(); err != nil {
t.Fatal(err)
}
if c.botLogin != "gbr-bot" || c.botEmail != "gbr-bot@users.noreply" {
t.Errorf("identity = %s <%s>", c.botLogin, c.botEmail)
}
}
func TestReplaceOldDraftsClosesOnlyThisGamesDrafts(t *testing.T) {
var calls []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls = append(calls, r.Method+" "+r.URL.Path)
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/repos/org/games/pulls":
w.Write([]byte(`[
{"number": 13, "head": {"ref": "draft/example-4b174b2d"}},
{"number": 20, "head": {"ref": "draft/example"}},
{"number": 21, "head": {"ref": "draft/example-other"}},
{"number": 22, "head": {"ref": "draft/other-4b174b2d"}},
{"number": 23, "head": {"ref": "fix/example-ports"}}
]`))
case r.Method == http.MethodDelete && r.URL.Path == "/api/v1/repos/org/games/branches/draft/example-4b174b2d":
w.WriteHeader(http.StatusNoContent)
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNotFound) // already gone is fine
default:
w.Write([]byte(`{}`))
}
}))
defer srv.Close()
c := &workerConfig{gitea: srv.URL, repo: "org/games", botToken: "secret"}
closed, err := c.replaceOldDrafts("example")
if err != nil {
t.Fatal(err)
}
if len(closed) != 2 || closed[0] != 13 || closed[1] != 20 {
t.Errorf("closed %v, want [13 20]: not another game's draft, not draft/example-other, not someone's fix branch", closed)
}
got := strings.Join(calls, "\n")
for _, want := range []string{"PATCH /api/v1/repos/org/games/pulls/13", "PATCH /api/v1/repos/org/games/pulls/20",
"DELETE /api/v1/repos/org/games/branches/draft/example-4b174b2d", "DELETE /api/v1/repos/org/games/branches/draft/example"} {
if !strings.Contains(got, want) {
t.Errorf("missing %q in:\n%s", want, got)
}
}
for _, unwanted := range []string{"pulls/21", "pulls/22", "pulls/23"} {
if strings.Contains(got, unwanted) {
t.Errorf("touched %s:\n%s", unwanted, got)
}
}
}
@@ -3,7 +3,7 @@ id: ark-survival-ascended-crossplay
name: 'ARK: Survival Ascended (Crossplay Dedicated Server)'
defaultDockerImage: mschnitzer/asa-linux-server:latest
containerUser: "25000:25000"
defaultStartupCommand: bash -c 'mkdir -p /server /data/Steam /data/steamcmd /data/cluster-shared && rm -rf /home/gameserver/server-files /home/gameserver/Steam /home/gameserver/steamcmd /home/gameserver/cluster-shared && ln -sfn /server /home/gameserver/server-files && ln -sfn /data/Steam /home/gameserver/Steam && ln -sfn /data/steamcmd /home/gameserver/steamcmd && ln -sfn /data/cluster-shared /home/gameserver/cluster-shared && (cd /home/gameserver/steamcmd && { [ -x steamcmd.sh ] || wget -qO- https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz | tar xz; } && for i in 1 2 3; do ./steamcmd.sh +login anonymous +quit; ./steamcmd.sh +force_install_dir /home/gameserver/server-files +login anonymous +app_update 2430930 validate +quit; [ -f /home/gameserver/server-files/ShooterGame/Binaries/Win64/ArkAscendedServer.exe ] && break; echo "ARK install attempt $i failed, retrying in 15s"; sleep 15; done) && export ASA_START_PARAMS="${ASA_START_PARAMS:-${SERVER_MAP:-TheIsland_WP}?listen?Port=${PORT:-7777}?RCONPort=${RCONPORT:-27020}?RCONEnabled=True -WinLiveMaxPlayers=${MAX_PLAYERS:-70} -ServerPlatform=ALL}" && exec /usr/bin/start_server'
defaultStartupCommand: bash -c 'mkdir -p /server /data/Steam /data/steamcmd /data/cluster-shared && rm -rf /home/gameserver/server-files /home/gameserver/Steam /home/gameserver/steamcmd /home/gameserver/cluster-shared && ln -sfn /server /home/gameserver/server-files && ln -sfn /data/Steam /home/gameserver/Steam && ln -sfn /data/steamcmd /home/gameserver/steamcmd && ln -sfn /data/cluster-shared /home/gameserver/cluster-shared && (cd /home/gameserver/steamcmd && { [ -x steamcmd.sh ] || wget -qO- https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz | tar xz; } && for i in 1 2 3; do ./steamcmd.sh +login anonymous +quit; ./steamcmd.sh +force_install_dir /home/gameserver/server-files +login anonymous +app_update 2430930 validate +quit; [ -f /home/gameserver/server-files/ShooterGame/Binaries/Win64/ArkAscendedServer.exe ] && break; echo "ARK install attempt $i failed, retrying in 15s"; sleep 15; done) && export ASA_START_PARAMS="${ASA_START_PARAMS:-${SERVER_MAP:-TheIsland_WP}?listen?Port=7777?RCONPort=${RCONPORT:-27020}?RCONEnabled=True -WinLiveMaxPlayers=${MAX_PLAYERS:-70} -ServerPlatform=ALL}" && exec /usr/bin/start_server'
requiredMemoryMb: 16384
requiredDiskMb: 32768
defaultPorts:
@@ -25,7 +25,6 @@ defaultConfig:
MaxPlayers: 70
NightTimeSpeedScale: 1
OverrideOfficialDifficulty: 5
Port: 7777
PreventDiseases: false
PreventDownloadDinos: false
PreventDownloadItems: false
@@ -137,12 +136,6 @@ configSchema:
description: Sets the maximum wild dino level ceiling (e.g. 5.0 allows wild creatures up to level 150).
title: Override Official Difficulty
type: number
Port:
category: System & General
default: 7777
description: Primary UDP port used by game clients to connect to the dedicated server.
title: Game Port
type: integer
PreventDiseases:
category: Gameplay & Rules
default: false
+1 -8
View File
@@ -40,7 +40,6 @@ defaultConfig:
PVP: true
RCON_PASSWORD: ""
RCON_PORT: 25575
SERVER_PORT: 25565
TYPE: PAPER
USE_AIKAR_FLAGS: true
VERSION: LATEST
@@ -181,12 +180,6 @@ configSchema:
description: TCP port for remote console. Only reachable from the host node (not published publicly); GBR uses it for actions and player moderation.
title: RCON Port
type: integer
SERVER_PORT:
category: System & General
default: 25565
description: The TCP network port on which Java Edition Minecraft players connect.
title: Java Server Port
type: integer
TYPE:
category: System & General
default: PAPER
@@ -362,7 +355,7 @@ agnosticCommandConfig:
joinButtonLabel: How to Join
joinModalTitle: How to Join This Server
pingPort: 25565
pingPortKey: SERVER_PORT
pingPortKey: null
playerJoinRegex: :\s+([a-zA-Z0-9_.*]+) joined the game
playerLeaveRegex: :\s+([a-zA-Z0-9_.*]+) left the game
playersCommand: null
+1 -1
View File
@@ -17,7 +17,7 @@
"defaultDockerImage": { "description": "Image the game server runs in.", "type": "string", "minLength": 1 },
"containerUser": {
"description": "uid:gid the server runs as, for images built around their own user (ARK's needs 25000:25000). Defaults to 1000:1000; root is never allowed.",
"description": "uid:gid the server runs as, for images built around their own user, as inspect-image reports it. Defaults to 1000:1000; root is never allowed.",
"type": ["string", "null"],
"pattern": "^[1-9][0-9]{0,8}:[1-9][0-9]{0,8}$"
},