14. MultiplayerExperimental

The Multiplayer map of the sample project shows it with two windows: a listen server and a client, each sculpting and painting the same ground. The sample’s README has three ways to run it (in the editor with 2 players and Play As Listen Server, Scripts/play_multiplayer.ps1, or two machines).

Design

• The server is the only machine that changes the ground. It keeps one ordered log of edits, as records, and sends every record to every client.
• A client applies the server’s records in order. When its player sculpts, the client sends the strokes to the server as requests. The server checks each request, applies it, and sends the result back to everyone as records. The player sees their own stroke once its records come back: the round trip, plus at most 67 ms of batching.
• Each player has a link: an actor (AOrreryTerrainNetLink) that the server spawns when the player logs in, owned by the player’s PlayerController and relevant only to that player. It carries the terrain’s RPCs on its own channel, so terrain traffic never fills your PlayerController’s channel. It replicates no property.
• Every 2 seconds while the ground changes (every 30 seconds when it does not), the server sends a checkpoint: a hash of the ground. A client that disagrees asks for the parts that differ and repairs them, without joining again. Each disagreement logs a warning in LogOrreryTerrainNet.

Set it up

1. Use a listen server (in the editor: Play As Listen Server; in a game: start the host with ?listen). A dedicated server is not a supported setup yet (see Limits below). The terrain actor and its source must be in the map on every machine: the natural ground never travels, only the edits.
2. That is all. UOrreryTerrainNetSubsystem starts on the server at BeginPlay and opens a link for each player at login. Your GameMode needs no change.
3. Raise the engine’s net rates if your players will join a world with much edited ground (see Bandwidth).

The settings are in Project Settings > Plugins > Orrery Terrain > Network:

Setting	Default	Effect
Built In Replication	on	Off: the plugin sends nothing, and you move the records with your own transport (see Your own transport).
Max Steps Per Second	60	Brush steps a second the server takes from one player. The rest are refused (Rate).
Max Edit Distance	0 m (no limit)	How far from the player’s reported position a brush may land (Reach).
Flush Hz	15	Record batches a second the server sends one player.
Client Flush Hz	30	Request batches a second a client sends.
Window Bytes	192 KB	Bytes in flight to one player before the server waits.
Checkpoint Seconds / Idle Checkpoint Seconds	2 s / 30 s	How often the ground is checked while it changes, and while it does not.
Max Lag Megabytes	16 MB	How far a player may fall behind before it joins again from a fresh copy.
Max Held Seconds / Max Held Records	2 s / 256	How long a request waits for the ground under it to load on the server before it is refused (NotLoaded).
Server Residency Radius	1000 m	How far around each player’s reported view and pawn the server keeps full-resolution ground. The client cannot change it. See Reading the ground on a server.
Max Edit Ground Tiles	200	Most base tiles one request, or one open stroke, may load beyond what players keep resident. A larger one is refused as NotLoaded, and the reason gives the count. A small plate far from every player needs about 64. 0 means no limit.
Max Links	0 (no limit)	Players the terrain serves. One more is refused (Full).
Max Stamp Tiles Per Second	2048	Edit tiles of height stamps a second the server takes from one player (each stamp counts the tiles it touches). The rest are refused (Rate). 0 is no limit.
Edit Policy Class	none	Your rules for who may edit where (see Edit rules).
Max Upload Masks Per Author / Max Upload Kilobytes Per Author	4 / 4096 KB	Brush masks one player may upload in a session, and their samples (see Brush masks from players). 0 turns uploads off.
Max Upload Mask Kilobytes	2048 KB	The largest mask a player may upload, in samples: a 1024 x 1024 16-bit height mask. 0 turns uploads off.
Mask Upload Kilobytes Per Second	256 KB/s	How fast a client sends an upload, beside its other requests.
Join Cache / Join Cache Megabytes	on / 2048 MB	A client keeps a copy of the edited ground for when it comes back, and how much disk those copies may take (see Coming back).

Player actions

On a client, the plugin’s tools send requests by themselves:

• The brush tool and the sculpt component: strokes, paint, undo and redo.
• Texture brushes, splats and height stamps (Stamp Terrain (Async), Stamp At, Splat At, and a splat drag): the same tools, sent as requests. The server checks a stamp against the stamp limits (Limits). A mask the server holds reaches every client before the first step that uses it.
• Terrain Spline Feature: Rebuild Spline Feature (Async) asks the server to replace the feature the component is linked to (Live splines in a multiplayer game).
• Commit Plate (Async) and Commit Corridor (Async): the node completes when the server has answered and the client shows the feature. The synchronous CommitPlate and CommitCorridor refuse on a client: a replica applies only the server’s records.

Client restrictions

A client holds a replica: it applies the server’s records and nothing else. Local edits are refused, and the plugin’s tools turn them into requests for you.

• Begin Stroke returns 0 and the synchronous CommitPlate, CommitCorridor and UndoConstruction refuse (the edit access is Replica). Use the brush tool, the sculpt component or the async commit nodes, or send a request (below).
• A client does not choose what the natural ground is. It must load the same source asset as the server, or the join is refused and the client cannot join.
• A client does not load a save into the shared terrain. The server owns saving and loading: load on the server before players join, because a load changes the ground for everyone and clients rejoin from the new state.
• A client does not reach beyond the server’s rules: the rate, reach, sculpt limits and your edit policy decide each request, and the answer (accepted records or a refusal) comes back to the player who asked.
• A client changes only the features it made. Replacing or removing another player’s feature, or the server’s (the host’s, the editor’s or a loaded save’s), is refused as Policy, and so is moving a feature out of or into ground your policy’s Can Edit refuses; override Can Change Feature on your edit policy to widen the rule, for an admin. A client never commits a feature under an owner domain Orrery reserves (TSED, RAIL), whatever your policy says.

From your own C++ code, UOrreryTerrainNetSubsystem takes RequestSculpt, RequestUndo, RequestFeature and RequestRemoveFeature. Each returns a request id, or 0 when the client is not live yet.

Undo with several players. Each player undoes their own strokes. An undo is refused (Conflict) when another stroke has written over the same ground since: an undo never erases another player’s work.

Answers from the server

Bind On Request Result on the net subsystem (GetWorld()->GetSubsystem<UOrreryTerrainNetSubsystem>(), or Get Orrery Terrain Net Subsystem in Blueprint). Accepted brush steps are not answered: their records come back. Refusals, undos, redos and feature requests are. A result carries the request id, its kind, the refusal (None when accepted), the record that carries the change, and for a feature its id and earthworks.

Refusal	Meaning
Rate, Reach, Limits	Too many steps a second, too far from the player, or outside the sculpt limits.
Policy	Your edit policy said no.
NotLoaded	The ground under the request did not load on the server in time, or it needs more than Max Edit Ground Tiles.
Conflict, NothingToUndo	Undo or redo: another stroke wrote over the ground since, or there is nothing to take back.
Feature	The feature stack refused the plate or corridor; Reason says why.
Rejoined	The player was joining again when the server got the request. Nothing changed; send it again.
UnknownMask	The brush names a brush mask the server does not hold, and the server takes no uploads.
UploadsDisabled, UploadQuota, UploadTooLarge, UploadCorrupt, UploadTimedOut	A brush mask upload, and the brush steps that waited for it: your policy took no upload, the player is past the quota, the mask is over the size cap, the bytes were not the mask, or the upload stalled (its next use offers it again).
ReadOnly, NoTerrain, Malformed, OutOfOrder, ReadFailed	The server’s terrain refuses edits now, has no terrain, or could not read the request or an undo image.

Edit rules

Make a Blueprint or C++ class from UOrreryTerrainEditPolicy and set it as the Edit Policy Class. Override Can Edit (it gets the player’s controller, the kind of edit and the box it touches) for zoning or money: the server asks it for every request that passed the rate, reach and limit checks, and for a feature’s replace with both the old box and the new. Override Can Change Feature (it gets the feature’s id, its owner, the player who made it and both boxes) to decide who may replace or remove an existing feature. With no policy set, a client may edit ground anywhere but may replace or remove only its own features, and may not use a reserved owner domain (TSED, RAIL); no policy lifts that last rule.

Brush masks from players

A brush mask the server registered (a brush asset it loaded) reaches every player before the first stroke that uses it, and every player who joins later. A mask only a client holds, one a player made or imported, is refused (UnknownMask) unless you let players upload masks.

Uploads are off by default. To turn them on, override Can Add Brush Mask on your edit policy: it gets the player’s controller and the mask’s id, side, bits and size, with what that player has uploaded so far, and returns true to take it. The default policy, and no policy, take none. Think before you do: a mask can draw shapes or words into a world everyone shares, an accepted mask stays on the server for the session and goes to every player and every later joiner, and each machine holds it in memory (up to 2 MB of samples for the largest).

• When a player’s stroke names a mask the server is not known to hold, the client first offers it. The server checks the size cap (Max Upload Mask Kilobytes, and half of Max Lag Megabytes), the player’s quota, then asks your policy; nothing more is sent if any says no.
• The mask then goes up in 16 KB parts, at most Mask Upload Kilobytes Per Second. The stroke’s steps wait on the server meanwhile, in order, up to 8 x Max Held Records of them. The server checks the mask’s hash once every byte arrived, then sends it to every player before the steps apply.
• An upload never sends another player to a rejoin: the server waits until the mask fits under half of every player’s lag limit, for at most 30 seconds (UploadTimedOut).
• The quota counts per player connection: joining again keeps it, and an upload broken by a rejoin resumes where it stopped. A player who disconnects and comes back starts a new count, so count per account in Can Add Brush Mask if that matters.
• On Request Result answers an upload with the kind AddBrushMask and the mask’s id (MaskHash): first with the bytes the server holds (UploadedBytes), then accepted, with the record that carries the mask, or refused.

Joining

When a player connects, the client and the server first compare their builds and their natural ground. A mismatch is refused with a reason the client reads with Get Join Refusal:

Refusal	Meaning
VersionMismatch	The two builds code or apply edits differently: another plugin version.
NaturalMismatch	The two sides load other natural ground: another source asset or another import.
GridMismatch	The two terrains have another grid.
Full	The server takes no more players (Max Links).
JoinCorrupt	The copy of the ground failed its check twice.

Then the server sends the edited ground: the features, the paint, and the sculpted tiles nearest to the player first. Editing goes on meanwhile. Is Client Live turns true once the ground near the player has arrived; the rest keeps streaming in the background. A player can join while another holds a stroke open.

A Terrain Spline Feature component on a late joiner, or one that streams in after a replace, answers Is Stale and Get State as the server does: when the join has loaded, or the component registers, it compares its own spline with the stored feature and nothing is built or sent. Replicate your splines’ points yourself if every machine should hold the same spline.

Join from a file

A joiner receives the whole edited ground, which can be many megabytes. So that neither side has to hold that save in memory, both sides keep it in a file:

• The host writes the save for each join into Saved/OrreryTerrain/Join/<pid>-<world>-<n>.tsjoin and streams it from there. The file is deleted with the join, and a server sweeps files left by dead processes when it starts. A joiner costs the host about the join’s plan, 256 KiB of read-ahead and one frame of data in memory (312 KiB measured with an 8.83 MiB save).
• The client appends what it receives to Saved/OrreryTerrain/Join/client-<pid>-<n>.tsjoin and pages the edited ground from that file once it is live. After the load it holds about 3 KiB in memory (2.24 MiB save).
• If a file cannot be made, written or read back, the join is refused. There is no fall-back to memory. The host logs why (SaveFailed); a client whose copy fails its check asks once more, then reports JoinCorrupt.
• Make sure the project’s Saved/ folder is writable on both machines (a packaged game on a read-only install location needs a writable Saved).
• Orrery.Terrain.Net.JoinFile 0 keeps the saves in memory instead of files. Use it only to compare or to work around a disk problem.

Get Net Stats gives each side’s counters for a debug panel: links, joins in progress, records, bytes, checkpoints, repairs and rejoins, and Join Cache Bytes, what the last join took from a kept copy.

Coming back

A client keeps a copy of the edited ground on its disk. When the player comes back to ground it has seen, the server sends only what that copy does not hold: nothing of the ground when nothing changed, and otherwise the save’s index, its paint and stack, and the sculpted tiles that changed. Costs are in Bandwidth.

• A kept copy is never trusted. Every byte taken from it is checked against the hashes the server sends, and a tile that fails is sent again. A client joined from its copy reaches the same state hash and the same checkpoints as one joined by the full stream.
• If a copy cannot be used, the player joins by the full stream, and the client’s log says why: the file is truncated or changed, it is another version, a tile fails its hash, or the server did not get an answer about it. This does not count as a failed join.
• What makes a copy usable is the natural ground, the grid and the plugin’s save and join versions, not the server. A copy from one server can serve another on the same ground; only the tiles that match are used.
• Where: Saved/OrreryTerrain/JoinCache/<ground>/<hash>.tsjoin. A copy is kept once a join has passed its check, and again when the player leaves if the ground moved on since. A leave while the engine is exiting keeps nothing new.
• Disk: the network setting Join Cache Megabytes (2048) caps the folder across all servers and maps, and a copy larger than that is not kept. The least recently used copies go first, and each ground keeps its two newest. A file that another client on the machine has open stays until a later write, so the folder can pass the cap by what is in use.
• Several clients on one machine (PIE, several game instances) share the folder safely: a copy is written under a temporary name, renamed once and never changed.
• Turning off the network setting Join Cache stops it: on a client it keeps and names nothing, on a server it ignores what clients name. Join Cache Megabytes 0 also turns it off on a client.

Bandwidth

Live editing is cheap. One player sculpting costs each other player about 0.16 KB/s of record batches, measured over a real connection. With four players sculpting at once, each player received 0.54 KB/s (0.14 KB/s per sculptor) and sent 0.21 KB/s of requests. The plugin’s tests fail above 2 KB/s per sculptor.

A late join costs what has been edited, not how long the game has run:

Edited ground	Per sculpted km²
Lightly sculpted	51-115 KB
Construction prep (pads, cuts, embankments)	about 0.43 MB
Heavily brushed	0.73-0.94 MB

Unreal’s default rate is 100,000 bytes per second per player, and your game uses part of it. A join of 15 MB (20 km² heavily brushed) takes about 3 minutes at that rate and 15 seconds at 1 MB/s. For terrain-heavy games, raise the rates in your project’s Config/DefaultEngine.ini:

[/Script/Engine.Player]
ConfiguredInternetSpeed=1000000
ConfiguredLanSpeed=1000000

[/Script/OnlineSubsystemUtils.IpNetDriver]
MaxClientRate=1000000
MaxInternetClientRate=1000000

The plugin never changes these for you. The sample project sets them.

A player who comes back pays far less (Coming back): measured on a 16 x 16 tile sculpted world over the plugin’s test transport, a full join of 3.39 MB took 4.3 s at 1 MB/s. Coming back took 153 bytes and 0.12 s when nothing had changed, 117 KB and 0.29 s after one small stroke, and 237 KB and 0.32 s after a tenth of the ground was resculpted. When anything changed, the save’s index, paint and edit stack always travel, about 110 KB in this measurement, and that part grows with the world: for a fully sculpted 64 km terrain, its index alone is an estimated 18 MB.

Replication Graph and Iris

Replication Graph. The link is relevant only to its owner (bOnlyRelevantToOwner). Unreal’s UBasicReplicationGraph already routes such actors to each connection’s always-relevant node, so it needs nothing. A graph of your own must do the same for AOrreryTerrainNetLink, or the link never reaches its player and the client never joins:

void UMyReplicationGraph::RouteAddNetworkActorToNodes(const FNewReplicatedActorInfo& ActorInfo, FGlobalActorReplicationInfo& GlobalInfo)
{
    if (ActorInfo.Actor->IsA<AOrreryTerrainNetLink>())
    {
        // Owned by the player's PlayerController: add it to that connection's always-relevant node.
        if (UReplicationGraphNode_AlwaysRelevant_ForConnection* Node = MyAlwaysRelevantNodeFor(ActorInfo.Actor->GetNetConnection()))
        {
            Node->NotifyAddNetworkActor(ActorInfo);
        }
        return;
    }
    Super::RouteAddNetworkActorToNodes(ActorInfo, GlobalInfo);
}

Remove it from the same node in RouteRemoveNetworkActorToNodes. MyAlwaysRelevantNodeFor stands for however your graph finds a connection’s node (UBasicReplicationGraph::GetAlwaysRelevantNodeForConnection is one way).

Iris. The link uses only RPCs, which behave the same on Unreal’s generic replication and on Iris, and it uses the registered subobject list Iris asks for. The plugin’s multi-process net test passes on both, at the same bytes per second. Iris also asks your project for net.SubObjects.DefaultUseSubObjectReplicationList=1 under [SystemSettings] (it ensures once at start without it); that setting is for your own actors, the terrain needs nothing.

Your own transport

Turn Built In Replication off when your game moves data its own way (a relay, a lockstep game, a custom server). The server’s terrain then hands you each record (OnTerrainRecordBP, or AddRecordListener in C++) and a client applies them with Apply Terrain Record, in order. The sculpt component can hand you a client’s strokes instead of applying them (bRouteCommandsExternally, On Command Requested). Joins, checkpoints and repair are then yours.

Reading the ground on a server

• Read ExactOnly for anything the server decides. Where it answers, the server and every client get the same answer.
• Do not read Coarse for that. On a dedicated server it leaves out plates and corridors, so it can differ from a client’s over an embankment (Determinism).
• A server keeps full-resolution ground around each player’s reported view and pawn, out to Server Residency Radius. Elsewhere a dedicated server has none, so ExactOnly answers no data there until it loads. To read anywhere else, register an invoker near the point or use a wait node. Set the radius to 0 to turn this off; a dedicated server then follows its player controllers at the terrain’s own residency radius.
• Ground near a plate or corridor bakes only with the ground around it resident, up to about 720 m away, which can be past the radius. An edit and a bake ask for that ground themselves, so the radius does not have to cover it.
  • A request waits up to Max Held Seconds while its ground loads, then applies. It is refused as NotLoaded when that is too slow or the ground is more than Max Edit Ground Tiles.
  • Ground within the radius that waits for its bake loads what it still needs, bakes on a worker, and lets the extra tiles go again. All such waits together hold at most the terrain’s Max Bake Request Tiles (200 by default, about 6.6 MB of heights), so many edits at once can delay a bake but cannot grow the server’s memory past that.
  • Until it bakes, ExactOnly answers Pending there. GetBakeWait says why: NotKept (no source wants this ground), Loading, OverBudget or LoadFailed. The terrain’s stats count BakeRequestBases, BakeRequestsWaiting and BakeRequestsFailed.

Limits

• Version 1.0 supports listen servers only. The plugin’s own setup for a dedicated server is built: it renders nothing, keeps no far data, and keeps the ground resident from the players’ links. It is still being tested on each supported engine, so a game that runs one now does it at its own risk.
• Multiplayer has been tested with Windows servers. The runtime modules build for Linux dedicated servers (Limits), but a session against a Linux server is still to come.
• Every machine applies every player’s strokes: CPU grows with the number of players sculpting at once. In a test with four players sculpting at once (headless, so without rendering), the server used about 3 % of one core more than when idle, and each client at most 2 %.
• A player who disconnects and comes back downloads the edited ground again. The download is the whole edited ground, not only what changed since they left. A saved copy for a quick return is planned for 1.1.
• Grass is grown on each machine from the replicated paint. It is not replicated itself.

Check it worked

1. Open the sample project’s Multiplayer map and run it as a listen server with two players (README, Multiplayer).
2. Sculpt in one window. The ridge appears in the other within a fraction of a second.
3. Press 2 in the client window: its own last stroke is undone. Press it after the server player has sculpted over the same ground: the panel shows Conflict.
4. Start a third player after some edits. It shows the edited ground once it is live.