Linus Torvalds: “I'm happily hacking on a new save format using ‘libgit2’” (opens in new tab)

You probably know this, but remember that storing user data as code is a place where you (general "you") have to think very carefully about security.

Is there any way that arbitrary code in the file could compromise the user's system? If so, does the user know to treat these data files as executables? Is there any way someone untrusted could ever edit the file without the user's knowledge? Even in combination with other programs the user might be running? Are you sure about all of that?

Maybe Lua in particular is sandboxed so that's not a problem (beats me), but in general this is an area where safe high-level languages can all of a sudden turn dangerous. Personally I would rarely find it worth it.

Preach it. My favorite persistence code: stuff that has nothing to do with SQL/NoSQL.

I leaned heavily on Python's pickle module for serializing a few thousand entities to disk a few years ago. By streaming them to the application at startup time, it remained plenty fast for all datasets it'd encounter. I intended to replace it with SQLite one day, but I never had to. I could just keep them all in memory.

I'd probably choose something a bit safer now, but it was hard to beat the simplicity.

Why does using a Lua-basef format stop the files being corrupted?

I did this with C# in my last game. All the map/object editors output C# code on save, which was then included in the compiled code on the next build. The beauty is that your "data" files get automatically updated when you refactor your regular code! On top of that loading is faster, because you don't need to worry about fetching a file and parsing it, the whole thing is just compiled code embedded in your executable.

That Lua was originally designed as a configuration language becomes really clear when you start doing things like this. Having my code and configuration being separate but equal was really a paradigm shift for me.

Also, the Tiled Map Editor exports directly to Lua.

IIRC that's how Office and Photoshop file format started. I think it's a nightmare for compatibility in the end.

So, it's similar to JSON (JavaScript), but valid Lua syntax.

  local t = {}
  t = {["foo"] = "bar", [123] = 456}
  t.foo2 = "bar2"

While one could say that this is about savefiles for games, I would say it's implications could more be about savefiles for software projects. If you are building the game in LUA, of course LUA is going to be the preferred way to save your game in since you are already using LUA objects and interpreting files in that language will be easy to integrate.

If you ever used maven xml configs, java object marshalling or c# xml you would understand the pains of using xml as a file format for software projects and data representation. You have to find a solution that is language agnostic, neither LUA or JSON is.

I did something similar, but used JSON instead (pretty trivial to (de)/serialize LUA tables to JSON. This made it easy to send data to the server, and inspect with standard tools as well.

This sounds a lot like NSCoding for Objective-C (Cocoa). Though you'd still have to define the types/classes and name for each property you want to save. But you could technically save it in a big blob, and then read it into memory as you resume.

Could persist to disk as a binary, sql or a plist (xml).

I guess the only downside is, that if you got a lot of composite classes all with their own properties and associations (say a graph), there's a lot of manual work to be done.

I've had to write output save file formats for various projects on several occasions, and it never occurred to me to take this approach.

Thanks for sharing this, it's one of those ideas that (to me) seems so brilliant in its simplicity that I probably would've never thought of it.

Any hiccups in the day-to-day work using this approach? I'm just trying to get a better idea of the workflow since I'm very seriously considering applying it to my next project.

That's how people are going to cheat at your game.

XML is unnecessarily verbose, for the supposed sake of human readability. But used as a serialization format, it isn't really readable or editable by humans (except in the sense that a Turing machine is programmable): remember that the ML in XML stands for "markup language", and SGML, its predecessor, was designed as a way of marking up normal text, not littering data with angular brackets and identifiers. (XML/SGML arguably isn't that hot as a markup language, either.)

If you really need a hierarchical serialization format that is "verified for validity and syntax", the problem is that XML has prevented the adoption of something better (because it was "good enough").

If you don't need that, then XML is overkill and bloat and makes your format less readable than it could be. And you rarely need it, because either your data is computer-generated and -read, so there's little point in putting in extra schema checks, or schema verification is woefully insufficient (because it can't verify the contents of fields, relations between fields, or a ton of other stuff that can accidentally go wrong).

If used sensibly XML isn't too bad. But there's a whole lot of cruft in the standard that seems to do nothing except make it harder to use. Part of this is a problem with popular libraries rather than inherent to the format, but we judge a thing by its ecosystem rather than in isolation. So: namespaces are a pain, making it much harder than it should be to just make my xpath work. DTDs are annoying, especially when a production system breaks because a remote server that was hosting a DTD goes down so now your parser refuses to load a file. User-defined entities seem pointless, and though most parsers can handle the billion laughs these days it wasn't always so. The handling of text nodes is confusing; whitespace is irrelevant except when it isn't. Specifying the encoding inside the document itself seems wrong, and supporting multiple encodings at all causes trouble (e.g. sometimes it's simply impossible to include one document in another inline).

Is XML schema really so much better than e.g. JSON schema?

To me it feels like there's an impedance mismatch between the kind of structures XML lends itself to and the kind of structures programs are good at dealing with. So for program-to-program communications with a certain level of validation I find Protocol Buffers is a much better fit. Conversely in cases where human readability is really important, XML isn't good enough compared to JSON.

The issue is probably that 99.999% of all XML use cases don't use (or need) the verification aspect. For all of those, XML is overkill. Besides, surely it would be possible to design a verification layer on top of JSON, for instance - the fact that one does not currently exist does not mean that XML (and abuse of XML!) should not be criticized.

I personally miss having schemas and XSLT in JSON.

> doing everything in XML is a stupid idea (e.g. XSLT and Ant)

XSLT actually made a lot of sense. If everyone writes code to transform format1 to format2 then what you end up with a lot of slightly different transformations. Its main downfall, just like XML itself, was that it was annoying and time consuming to write.

How would you replace all this if you moved away from XML?

http://git.hohndel.org/?p=subsurface.git;a=tree;f=xslt;hb=HE...

The problem with XSD and DTD is they only offer primitive ways to validate data, and it takes significant effort to validate some data (eg,[ https://stackoverflow.com/questions/3382944 ]). As a result, there have been a bunch of other XML schema validators created to counter these problems, but we should really ask why we need to keep inventing new languages when the existing ones turn out to be insufficient.

If we start out instead with something that's turing complete and simple to begin with (perhaps S-expressions?), we can (often trivially) write our own validators/type-checkers, or any other processing tool to verify the document structure, with few or no constraints, and without requiring the effort and expertise to parse complex syntax.

Unfortunately, XML is way too open-ended for my tastes. You end up getting entire rows of DB content (with full text paragraphs and everything) entirely in one tag, with attributes and values. There are so many options that you typically get a lot of idiot programmers who don't understand the purpose of all the shit in XML, so they fuck up their implementation.

Simply put, XML does not correctly model the data by which we intend to interchange. It was a noble effort, but it didn't come from a place of innovation. It came from corporate needs for standardization.

S-expressions are also a reasonable choice, with some well-placed carriage returns and a serialization implementer that names things well

My biggest hatred of xml as a data structure, and believe me I've seen this in production systems more then once, is that it allows for the following.

  <customer>
	<account>
		<type>Personal</type>
		...
	</account>
	<account>
		<type>Business</type>
		...
	</account>
	<custid>496F3AB</custid>
  </customer>

This may seem innocuous, but XML allows mixing of arrays and objects too liberally, and makes automatic parsing overly complex. At first <customer> appears to be an array of account objects, but wait now that we reach the end we find that <customer> is an object with multiple keys and must create an unnamed array key to hold accounts.

XML is a document markup language, not a data format.

People dislike XML because it's way overkill for %99 of people's use cases but it still gets used anyway! Most people who use it should of been using something simpler like JSON to create their configuration file or return their list of strings in some HTTP API. You can have bloody security vulnerabilities with XML, like you had recently with facebook: https://www.facebook.com/BugBounty/posts/778897822124446

The likelihood of a JSON feature biting you in the ass like that is far lower. Don't use XML until you actually need something XML SPECIFICALLY provides.

Also JSON easily translates with easy to work with dictionaries and lists, XML parsers take more code to work with equivalent items.

He hates it, but he's using it. Take it for what it's worth.

> But if I want something that is able to express data structures customized by myself, usually with hierarchical data that can be verified for validity and syntax (XML Schemas or old-school DTD), what other options are there?

S-expressions work great. Syntax checking is far simpler, and validity checking is hence something you can roll yourself (and writing an S-expression schema checker ain't tough).

Similarly, what's with the binary format/protocol hate? That's what brought us our current extremely common mishandling of text encoding.

I haven't used it myself, but lisp seems well suited to the task. I've also heard good things about yaml, which is more well-supported by your language of choice.

I'd say use JSON, but JSON doesn't have adequate schemas yet (JSON-Schemas is crap).

Because XML solves a 'problem' in the worst possible way. It is not that easy to parse for machines and only the simplest XML files are readable by humans.

Besides, since 1960 or thereabouts we have S-Expressions. The world should just have used that without reinventing the wheel once again.

> You now end up with the following problems: locking, synchronising filesystem state with the program, inode usage, file handles to manage galore and concurrency. All sorts.

Which is why he's reusing git for resolving those pain points? Well presumably all except "synchronizing filesystem state with the program" -- where he's gone from using some kind of xml parser to marshal xml to objects/structs in ram to using a (simple(r)?) text parser to do the same.

I'm guessing he just writes/reads a full (part) of a log (a branch of the full tree, or whatever is used in the program. Maybe a list anchored at a date?) -- and lets git sort the history/backup thing.

So, yes, it's a different format, but I think the argument you're making is off -- seeing as he already has git for that? It's more like combining Maildir (or mboxes, only commited when valid) and git.

Maybe you want to wait till he release something. Cause you know, if he took months to get the big picture in mind, I doubt you grasp what he envision just by reading his comment.

I took this to mean that what he is replacing is a single XML file whose content is a tree of element nodes. Every time you have to make a change to that file (changing, removing or adding children nodes within the file) you would have to store a new copy of the file. The most efficient you can get is to store just the text diffs using git or something.

But what he replaces it with is a git object store. Each xml-node becomes a git object. They each point to a parent (just as git commits point to a parent commit).

Now writing to this datastore means adding a new node to the git object database and changing the parent references.

Where git stores commits that are related sequentially in time, this stores nodes in a tree relationship that IS the document.

If he's not talking about this then I'd like to officially take credit for my weird idea right now.

The impression I got was that he was going to store his data in a git object database and that the files would be virtual in there. It would be like the .git directory without the working files on disk. It's all just conjecture until his code his out.

Regardless, I would think that some applications are simple enough (store few enough separate objects in the file system) that the issues you cite are not likely to cause a problem.

Readers may be familiar with the TI-83 programmable graphing calculator's assembly language functionality (especially those who took high school math classes in the mid-to-late 1990's). The TI-83's only user-writable storage was 32K of RAM (there was a small lithium battery to keep it powered when you changed the AAA's; also some of the RAM was used for system stuff so somewhat less than 32K was actually available for user purposes).

You could write hex values in the program text editor, then you could tell the calculator to execute the hex codes as machine code. I understand the previous models, TI-82 and TI-85, were hacked / backdoored to run user-supplied assembly language, so TI responded by including an official entry point and developer documentation for the TI-83.

People later wrote loaders which allowed programs to be stored as binary instead of text (using half the space). Some loaders also had the capability to run binary programs by swapping them into the target address rather than copying them (theoretically a third option would be possible, running programs in-place if they weren't written to depend on their load address, but this wasn't a direction the community went in. gcc users may be familiar with -fPIC which produces code which can run from any address, and this flag is necessary when compiling code for shared libraries.)

This allowed people to create massive 20K+ applications (an RPG called Joltima comes to mind), that used most of the available RAM.

The fact that this loading scheme made static variables permanent was also quite convenient. (And most variables were static; stack-based addressing would be tough because the Z80 only has two index registers, one of which is used -- or perhaps I should say "utterly wasted" -- by the TI-83 OS.)

The next generation, the TI-83+, included I think 256K of flash ROM, and a special silver edition was released which contained 2 MB.

Reminds me of the approach taken by Xmonad where the configuration is compiled into a new executable and then run.

Thank you for that anecdote, it made my day. Simply awesome.

That's just mean. Waterboarding isn't that bad...

With my occasional sysadmin hat on, until a few weeks ago I had the luck to never have had to deal with XML configuration files. Then came Solr and now I know what horror is. (To be clear, Solr itself is great, but those god damn config files...)

  "A change in perspective is worth 80 IQ points."
  
  -- Alan Kay

My biggest hurdle solving new problems is divining a unifying, simplifying metaphor. Once you have the right notion, that Eureka! moment, everything falls into place, like magic.

Like how Kepler was able to fully explain Bache's astronomical data once he realized the planets orbits the sun.

Personal example: I used to write print production software. Placing pages onto much larger sheets of paper that get folded and bound into a book. A task called image positioning aka imposition. It took me years to figure out how to model the problem. Key insight was simulating the work backwards, from binding back to the press. Then when I showed the new solution to my coworkers, the response was "Well, duh."

Yeah, I noted that too, also that it took him months to to get his good mental picture. It makes me feel not so bad about spending months trying to get clear on some of my stuff.

The question nobody is asking, but actually should is: I wonder what other good G+ content you are missing?

G+ is largely misunderstood. It is a lousy tool for interaction with people connected to you purely socially. It's a very good way to find and interact with people connected to you by interest.

For me it's not just G+, but also Facebook and Twitter. Only reason I ever visit those sites is indirectly through HN posts and similar.

I know this is completely off-topic, and I'll happily be downvoted for it, but why in the world does Google+ capture keyboard shortcuts that are already bound to other well known browser functions? (C-PgUp, C-PgDn, C-w, etc).

Linus : G+ :: notch : Java

I wonder if Linus ever reads Hacker News.....

Document markup is the one place XML is a no-brainer - more specifically, long-form, highly structured documents (i.e., essentially books).

Without it, publishing would be stuck in a morass of nebulous, ill-documented proprietary messes, and a great deal of current learning would be at risk of being lost to posterity. The fact that there are associated open standards such as XSLT with which to transform it is just the icing on the cake as far as publishing is concerned.

This is why there's so much distaste for XML - people try to use it for applications where it isn't ideal (and there are many more of those than there are applications where it is ideal) because they've swallowed someone else's hype, and as a consequence they have a bad time. If not for the unbelievable exaggeration a few years back (I heard people claim without irony that XML - a markup language for god's sake - would literally change the world), the divisiveness wouldn't exist, and it would be a technology used by experts quietly getting on with the jobs it's best for.

For those who missed it, here's what Linus wrote in the comments:

"+Aaron Traas no, XML isn't even good for document markup.

Use 'asciidoc' for document markup. Really. It's actually readable by humans, and easier to parse and way more flexible than XML.

XML is crap. Really. There are no excuses. XML is nasty to parse for humans, and it's a disaster to parse even for computers. There's just no reason for that horrible crap to exist.

As to JSON, it's certainly a better format than XML both for humans and computers, but it ends up sharing a lot of the same issues in the end: putting everything in one file is just not a good idea. There's a reason people end up using simple databases for a lot of things.

INI files are fine for simple config stuff. I still think that "git config" is a good implementation."

This link needs to be posted again and again and again.

I'm sure quite a lot of people will easily recognize it. :^)

Subject: Re: S-exp vs XML, HTML, LaTeX (was: Why lisp is growing)

https://groups.google.com/forum/message/raw?msg=comp.lang.li...

You should be speccing what you want to achieve: the goals, the why, the impact, the external limitations, measures of success and so forth. This also allows you to describe and plan testing up front. The "how" is best handled in an iterative manner.

A lot of people use AGILE to avoid planning at all, which is a particular destructive anti-pattern, and the exact opposite of what you need.

I agree it's confusing, I think the submitter just meant "system for files" or something.

Yep, I thought it sounded like Camlistore, but as a library.

I've got a few strategies that might help, depending on the circumstances.

A notebook: I'll write down some notes and just kind of free write whatever thoughts come to mind. If there's something that I think is important to come back to, I'll draw an empty box in the left margin (to be filled with a check mark later)

Readme: start writing the Readme for the project, even if you're not entirely sure of the details. Include code examples. If you don't like how the API is coming together, change it. It's way less work to modify the API now than it will be later.

Write a test: I don't always unit test, but when I do I test first :). This works well on projects that already have a decent test suite. It's kind of an executable version of the Readme.

Branch and Hack: branches are cheap. Make one and start playing. Don't like how it's turning out? Make a new branch and try again!

Ctrl-Z: maybe the answer won't come to you right away. Let it sit and run in the background for a while and come back to it. If I'm worried about forgetting details, I'll write it down in a notebook first.

This is the first profanity-free Linus rant that I've read in a long time.

Why reinvent binary serialization when you could use ASN.1, or any of the thousand binary serialization formats that pre-date protobufs?

Ironically that has already been reinvented in the form of Cap'n Proto: http://kentonv.github.io/capnproto/

(other than that I agree it's a good solution)

Did you read the article? It's not really about the filesystem. 1 part your fault for seemingly not reading the article you're commenting about, 1 part the submitter's fault for choosing such a misleading title.

I've also used libgit as a means to a similar end - providing versioned data across a local filesystem. Its an idea whose time has come ..

They can, if you recreate the primary feature of Git on top of them.

Have you tried turning it off and on again?

Is your title plugged in?