Adds a per-level persistence system. by MistakeNot4892 · Pull Request #5176 · NebulaSS13/Nebula

MistakeNot4892 · 2025-10-20T14:41:00Z

Description of changes

Adds /atom/Serialize() to return an assoc list of relevant values suitable for saving out.
Adds /atom/Preload() and /atom/PreloadData() and hooks in SSatoms flush to deserialize atoms during flush.
Adds /datum/level_data/load_persistent_data() and /datum/level_data/save_persistent_data() as entrypoints for per-level persistence.
Adds a fire() override to SSpersistence to do a periodic level data save.
Converts the innards of the previous persistence system to use the new save/load handlers and procs.

General flow of level persistence:

Saving:
- SSpersistence periodically iterates the z-level list, finds levels that want to serde, and calls save_persistent_data()
- Levels return a list of instances to get_persistent_instances(), instances have Serialize() called and return a list of modified fields.
- Fields are serialized (to JSON with the default handler) and written to disk.
Loading:
- SSmapping initializes and calls preload_persistent_data() and load_persistent_data() on relevant /datum/level_data z-level objects.
- load_persistent_data() creates the base instances and (for atoms) sets __init_deserialisation_payload with the data loaded from tile.
- SSatoms flush calls Preload() on all deserialized atoms which pre-populates vars on the atom.
- Ssatoms proceeds to Initialize() atoms as normal.

TODO

Future work

~~Completely integrate /decl/persistence_handler into this system instead of the bespoke serde on SSpersistence.~~ ended up largely doing this in this PR.
Add a bespoke system for serializing area changes (blueprints etc) on a z-level.

Why and what will this PR improve

Adds a framework for handling persistence in the future.

Authorship

Myself.

Changelog

Nothing player-facing.

MistakeNot4892 · 2025-10-20T14:49:15Z

Depends on #5175

out-of-phaze · 2025-10-20T16:19:01Z

+	// Is this actually desirable? People moving around or modifying
+	// atoms across save could result in inconsistent data.
+	set waitfor = FALSE


i think you'd have to actually add if(_persistent_save_running) return hooks into move and other client-driven/verb things to make it block anyway, is the thing. i'm not sure waitfor = TRUE would be enough. so it'd only be a partial solution either way

We could potentially suspend all subsystems and kick people back to the lobby during saves, maybe? Similar to how DMMS suspends fluids and such until the run finishes.

MistakeNot4892 · 2025-10-28T03:51:42Z

I'm going to call ~~datum and~~ area serde out of scope for this PR, I don't have any clue how to handle them in a clean way.

The atom side of this appears to be working fine, I've tested with a debug item and it all seems pretty good.

EDIT: Went ahead and just let it instance datums with the serde data as a list arg, in case it's needed in the future.

MistakeNot4892 · 2025-10-28T09:02:18Z

Slipped and accidentally added turf serde, limited testing on Shaded Hills but needs proper testing before I sign off on it.

MistakeNot4892 · 2025-10-28T09:03:15Z

Depends on #5182

MistakeNot4892 · 2025-10-28T10:04:58Z

Testing notes:

~~shutter state needs to be serialized on walls~~
fluids need to be serialized on floors
flooring is busted

PsyCommando · 2025-11-05T21:27:08Z

+	//   persistent_data_location = "data/level_data"
+	VAR_PROTECTED/persistent_data_location
+	/// Decl handler, mostly forcing myself to keep this general so it can be optimized with a DB or something down the track.
+	VAR_PRIVATE/persistence_handler = /decl/serialization_handler/json


Similarly as my last comment, shouldn't the way the level is saved really be stored in the level_data? Because, it doesn't really make sense to save each levels with a different serializer. It should be something decided at the start of the serialization imo.

The serialization logic doesn't really care where the data comes from, once it's in DM it's all in the same format. I don't really think it matters much. The alternative would be forcing the entire game to use the same handler, which to me limits the ability for people to mix and match or run their own local changes alongside general changes from upstream.

What would the benefit be to forcing a single handler for all cases?

i could see using a different handler for, specifically, custom ships vs a main map

The serialization logic doesn't really care where the data comes from, once it's in DM it's all in the same format. I don't really think it matters much. The alternative would be forcing the entire game to use the same handler, which to me limits the ability for people to mix and match or run their own local changes alongside general changes from upstream.

Yeah, but why should levels each individually chose to save in a different format? I don't really see a point to that? Because, you really want the same kind of data to be in the same format for coherence's sake and maintainability.

Also, what if you host a server, and you used json to store the save, but now your setup changed and you wanna use a DB. You'd need to change every single levels to the db handler and recompile. While, if you'd use all the same handler for levels you could easily make it a config entry, or call the serialization proc with a different handler specified and not have to mess with the code or do mass replace and risk making mistakes.

Unless I really don't understand what you mean/intend to do here?

What would the benefit be to forcing a single handler for all cases?

It would be decided at the time the serialization proc is called ideally. And it would be less error prone, much more predictable, easier to maintain, simpler to back up, etc.. It would also allow for consolidation of saved level data later on.

i could see using a different handler for, specifically, custom ships vs a main map

Yeah, but, depending on why you'd want a different handler, you probably would need extra metadata or extra processing for saving something to be used separately from the main save. Especially with anything multi-z.

Like, for example, I'll assume what you're thinking of is to move a ship and crew to another server via export or something else.
You'd need to make substantial changes to do that, and probably need a separate mechanism called on demand specifically for saving and moving ships and sending the save.
It would be less of a bother to just specify what handler to use when launching serialization, and what you want to serialize rather than making sure all ships being transferred have the right handler set and figure out what should even be done in the case they have the wrong handler?

I am getting increasingly confused by some of these comments. Having the ability to specify handler per level really doesn't impose any overhead or constraints that you wouldn't have by having it as a param or such. If you decide to change your data format you have a lot more problems than changing a handful of compile time values (such as having to manually migrate all your data).

Making it a config value or such just means there's no capacity whatsoever to have variance. If I wanted to save custom ships to JSON so players could save and load their layout privately, that wouldn't be possible. If I wanted to have large level serde handled by a DB for performance but some other aspect handled by JSON for some other reason, that wouldn't be possible.

If you don't want to use multiple handlers, don't change the value ever and you're golden. I really cannot see a benefit to centralising and restricting things like you seem to be asking for here and earlier.

out-of-phaze · 2025-11-06T01:42:13Z

-	if(populate)
+	// If preloaded from serde, handle expected list structure.
+	// Returns if preload is successful to skip populate_reagents() call.
+	FINALIZE_REAGENTS_SERDE_AND_RETURN(reagents)


think occasionally update_alpha_mask on the fluid overlay can run before this does, which is a little concerning. led to runtimes locally

Should be fixable by istype(neighbor.reagents) in that proc

MistakeNot4892 · 2025-11-06T12:17:15Z

I'm not really sure what to do about randomly generated levels. I might pull the reordering logic out of this PR and just prevent placing more mobs if we deserialized. Otherwise we end up with mask turfs, etc. left on the map.

out-of-phaze · 2025-11-06T16:49:07Z

I'm not really sure what to do about randomly generated levels. I might pull the reordering logic out of this PR and just prevent placing more mobs if we deserialized. Otherwise we end up with mask turfs, etc. left on the map.

should we not just automatically mark those turfs as serialized?

out-of-phaze

needs rebase

MistakeNot4892 · 2025-12-20T11:13:49Z

Current outstanding issue with this one is that I cannot for the life of me seem to get map templates placed during init to flag the turfs as changed. Generating on Shaded Hills and restarting leaves template-shaped sections of mask turfs. Not sure what the issue is, it happened even after I made all ChangeTurf() calls and relevant sections of DMMS flag the turfs directly. I am assuming that something is newing /turf directly and not flagging.

out-of-phaze · 2025-12-20T18:54:26Z

Current outstanding issue with this one is that I cannot for the life of me seem to get map templates placed during init to flag the turfs as changed. Generating on Shaded Hills and restarting leaves template-shaped sections of mask turfs. Not sure what the issue is, it happened even after I made all ChangeTurf() calls and relevant sections of DMMS flag the turfs directly. I am assuming that something is newing /turf directly and not flagging.

does /area/fantasy/outside/point_of_interest have AREA_FLAG_ALLOW_LEVEL_PERSISTENCE? it doesn't seem so, that'd mean at least a few non-passthrough templates (chemistry shack etc) would fail to save

MistakeNot4892 · 2025-12-21T00:25:22Z

does /area/fantasy/outside/point_of_interest have AREA_FLAG_ALLOW_LEVEL_PERSISTENCE? it doesn't seem so, that'd mean at least a few non-passthrough templates (chemistry shack etc) would fail to save

Genuinely no idea why I didn't think of this. Thank you :(

MistakeNot4892 · 2025-12-29T09:41:52Z

This is now working as far as I can tell. Main things I am worried about in live testing is accidentally cooking legacy persistence data, but it should at least write out a backup before it does any kind of migration of that data.

I am not committing any changes that actually use this system in full yet, I'm going to save that for after we've done playtesting (probably via Pyrelight or maybe Scav)

MistakeNot4892 · 2025-12-29T22:05:13Z

Realised belatedly that I need to get area serde in before this will work as intended, due to map templates setting area, but I will do that in a second PR, I don't want to fiddle with this one further currently.

out-of-phaze

i frankly really don't like that we still have the legacy persistence system, it really doesn't work for anything but papers and dirt (the generic 'filth' stuff was extremely annoying). but i guess some maps want a middle-ground re: persistence.

once this is in i'll try to make it faster

out-of-phaze · 2026-01-09T21:52:15Z

 	/// Used to avoid unnecessary refstring creation in Destroy().
 	var/tmp/has_state_machine = FALSE
+	/// Var for holding a unique-to-this-run identifier for a serialized datum.
+	VAR_PRIVATE/__run_uid


this could use /tmp/ possibly

Yes please. I really wish /tmp was used more. It helps make transient vars a whole lot more obvious.

out-of-phaze

was approved prior to conflict resolution, has conflicts again (my bad), if you fix conflicts i'll merge

MistakeNot4892 added has dependencies This PR should not be merged prior to any PRs linked in body or comments. work in progress This PR is under development and shouldn't be merged. labels Oct 20, 2025

MistakeNot4892 force-pushed the feature/limited_persistence branch 2 times, most recently from 1802990 to 1dd8215 Compare October 20, 2025 14:46

out-of-phaze requested changes Oct 20, 2025

View reviewed changes

MistakeNot4892 force-pushed the feature/limited_persistence branch from 1dd8215 to 4d406a4 Compare October 28, 2025 02:47

MistakeNot4892 added ready for review This PR is ready for review and merge. and removed has dependencies This PR should not be merged prior to any PRs linked in body or comments. work in progress This PR is under development and shouldn't be merged. labels Oct 28, 2025

MistakeNot4892 force-pushed the feature/limited_persistence branch from 4d406a4 to 96e2512 Compare October 28, 2025 03:50

MistakeNot4892 force-pushed the feature/limited_persistence branch 6 times, most recently from aa31be7 to 2862dd3 Compare October 28, 2025 08:55

MistakeNot4892 added has dependencies This PR should not be merged prior to any PRs linked in body or comments. work in progress This PR is under development and shouldn't be merged. and removed ready for review This PR is ready for review and merge. labels Oct 28, 2025

MistakeNot4892 force-pushed the feature/limited_persistence branch 6 times, most recently from f491ffe to 3a905e3 Compare October 28, 2025 23:40

PsyCommando reviewed Nov 5, 2025

View reviewed changes

Comment thread code/__defines/serde.dm

out-of-phaze requested changes Nov 6, 2025

View reviewed changes

Comment thread code/game/turfs/turf_serde.dm Outdated

out-of-phaze requested changes Nov 6, 2025

View reviewed changes

MistakeNot4892 force-pushed the feature/limited_persistence branch 12 times, most recently from 7d6d697 to 56e0c1b Compare November 6, 2025 12:16

MistakeNot4892 force-pushed the feature/limited_persistence branch from 56e0c1b to 47f36b1 Compare November 28, 2025 01:14

out-of-phaze requested changes Dec 20, 2025

View reviewed changes

Comment thread code/modules/reagents/Chemistry-Holder.dm

Comment thread code/modules/reagents/Chemistry-Holder.dm Outdated

Comment thread code/__defines/serde.dm Outdated

out-of-phaze previously approved these changes Jan 9, 2026

View reviewed changes

out-of-phaze reviewed Jan 9, 2026

View reviewed changes

out-of-phaze previously approved these changes Jan 28, 2026

View reviewed changes

Adds a per-level persistence system.

d7df53d

out-of-phaze approved these changes Jan 29, 2026

View reviewed changes

Conversation

MistakeNot4892 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

TODO

Future work

Why and what will this PR improve

Authorship

Changelog

Uh oh!

MistakeNot4892 commented Oct 20, 2025

Uh oh!

Uh oh!

out-of-phaze Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MistakeNot4892 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MistakeNot4892 commented Oct 28, 2025

Uh oh!

MistakeNot4892 commented Oct 28, 2025

Uh oh!

MistakeNot4892 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PsyCommando Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PsyCommando Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MistakeNot4892 commented Nov 6, 2025

Uh oh!

out-of-phaze commented Nov 6, 2025

Uh oh!

out-of-phaze left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MistakeNot4892 commented Dec 20, 2025

Uh oh!

out-of-phaze commented Dec 20, 2025

Uh oh!

MistakeNot4892 commented Dec 21, 2025

Uh oh!

MistakeNot4892 commented Dec 29, 2025

Uh oh!

MistakeNot4892 commented Dec 29, 2025

Uh oh!

out-of-phaze left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MistakeNot4892 commented Oct 20, 2025 •

edited

Loading

out-of-phaze Oct 20, 2025 •

edited

Loading

MistakeNot4892 commented Oct 28, 2025 •

edited

Loading

MistakeNot4892 commented Oct 28, 2025 •

edited

Loading

PsyCommando Nov 6, 2025 •

edited

Loading

PsyCommando Nov 6, 2025 •

edited

Loading