~2025-34236-49: /* Serialization formats */

2025-11-18T00:37:09Z

Serialization formats

← Previous revision		Revision as of 00:37, 18 November 2025
Line 41:		Line 41:
	==Serialization formats==		==Serialization formats==
	{{Main\|Comparison of data serialization formats}}		{{Main\|Comparison of data serialization formats}}
	The [[Xerox Network Systems]] Courier technology in the early 1980s influenced the first widely adopted standard. [[Sun Microsystems]] published the [[External Data Representation]] (XDR) in 1987.<ref>{{cite journal \|title= XDR: External Data Representation Standard \|author= Sun Microsystems \|journal= RFC 1014 \|year= 1987 \|publisher=Network Working Group \|url= http://tools.ietf.org/html/rfc1014 \|access-date= July 11, 2011 }}</ref> XDR is an [[open format]], and standardized as [https://tools.ietf.org/html/std67 STD 67] (RFC 4506) by [[Internet ~~Standard~~\|IETF]].		The [[Xerox Network Systems]] Courier technology in the early 1980s influenced the first widely adopted standard. [[Sun Microsystems]] published the [[External Data Representation]] (XDR) in 1987.<ref>{{cite journal \|title= XDR: External Data Representation Standard \|author= Sun Microsystems \|journal= RFC 1014 \|year= 1987 \|publisher=Network Working Group \|url= http://tools.ietf.org/html/rfc1014 \|access-date= July 11, 2011 }}</ref> XDR is an [[open format]], and standardized as [https://tools.ietf.org/html/std67 STD 67] (RFC 4506) by [[Internet Engineering Task Force\|IETF]].

	In the late 1990s, a push to provide an alternative to the standard serialization protocols started: [[XML]], an [[SGML]] subset, was used to produce a human-readable [[binary-to-text encoding\|text-based encoding]]. Such an encoding can be useful for persistent objects that may be read and understood by humans or communicated to other systems regardless of programming language. It has the disadvantage of losing the more compact, byte-stream-based encoding, but by this point larger storage and transmission capacities made file size less of a concern than in the early days of computing. In the 2000s, XML was often used for asynchronous transfer of structured data between client and server in [[Ajax (programming)\|Ajax]] web applications. XML is an open format and standardized as a W3C recommendation.		In the late 1990s, a push to provide an alternative to the standard serialization protocols started: [[XML]], an [[SGML]] subset, was used to produce a human-readable [[binary-to-text encoding\|text-based encoding]]. Such an encoding can be useful for persistent objects that may be read and understood by humans or communicated to other systems regardless of programming language. It has the disadvantage of losing the more compact, byte-stream-based encoding, but by this point larger storage and transmission capacities made file size less of a concern than in the early days of computing. In the 2000s, XML was often used for asynchronous transfer of structured data between client and server in [[Ajax (programming)\|Ajax]] web applications. XML is an open format and standardized as a W3C recommendation.
Line 58:		Line 58:
	===C and C++===		===C and C++===
	[[C (programming language)\|C]] and [[C++]] do not provide serialization as any sort of high-level construct, but both languages support writing any of the built-in [[C data types\|data types]], as well as [[plain old data]] [[struct (C programming language)\|structs]], as binary data. As such, it is usually trivial to write custom serialization functions. Moreover, compiler-based solutions, such as the ODB [[object–relational mapping\|ORM]] system for C++ and the [[gSOAP]] toolkit for C and C++, are capable of automatically producing serialization code with few or no modifications to class declarations. Other popular serialization frameworks are Boost.Serialization<ref>{{cite web\|url=http://www.boost.org/doc/libs/1_46_1/libs/serialization/doc/index.html\|title=Serialization\|website=www.boost.org}}</ref> from the [[Boost C++ Libraries\|Boost Framework]], the S11n framework,<ref>{{cite web\|url=http://s11n.net/\|title=s11n.net: object serialization/persistence in C++\|first=stephan\|last=beal\|website=s11n.net}}</ref> and Cereal.<ref>{{cite web\|url=https://uscilab.github.io/cereal/\|title=cereal Docs - Main\|website=uscilab.github.io}}</ref> [[Microsoft Foundation Class Library\|MFC framework]] (Microsoft) also provides serialization methodology as part of its Document-View architecture.		[[C (programming language)\|C]] and [[C++]] do not provide serialization as any sort of high-level construct, but both languages support writing any of the built-in [[C data types\|data types]], as well as [[plain old data]] [[struct (C programming language)\|structs]], as binary data. As such, it is usually trivial to write custom serialization functions. Moreover, compiler-based solutions, such as the ODB [[object–relational mapping\|ORM]] system for C++ and the [[gSOAP]] toolkit for C and C++, are capable of automatically producing serialization code with few or no modifications to class declarations. Other popular serialization frameworks are Boost.Serialization<ref>{{cite web\|url=http://www.boost.org/doc/libs/1_46_1/libs/serialization/doc/index.html\|title=Serialization\|website=www.boost.org}}</ref> from the [[Boost C++ Libraries\|Boost Framework]], the S11n framework,<ref>{{cite web\|url=http://s11n.net/\|title=s11n.net: object serialization/persistence in C++\|first=stephan\|last=beal\|website=s11n.net}}</ref> and Cereal.<ref>{{cite web\|url=https://uscilab.github.io/cereal/\|title=cereal Docs - Main\|website=uscilab.github.io}}</ref> [[Microsoft Foundation Class Library\|MFC framework]] (Microsoft) also provides serialization methodology as part of its Document-View architecture.

			With the introduction of [[reflective programming]] to [[C++26]], serialisation has been greatly simplified. Reflection allows for compile-time serialisation of, for instance, [[JSON]] into a <code>struct</code> with corresponding structure.<ref>{{Cite web\|title=Splicing a base class subobject\|url=https://isocpp.org/files/papers/P3293R3.html\|author=Peter Dimov, Dan Katz, Brry Rezvin, Daveed Vandevoorde\|date=19 June 2025\|website=isocpp.org\|publisher=WG 21}}</ref>

	===CFML===		===CFML===
Line 99:		Line 101:

	===Python===		===Python===
	The core general serialization mechanism is the <code>pickle</code> [[Python (programming language)#Libraries\|standard library]] module, alluding to the database systems term ''pickling''<ref>{{cite journal\|last1=Herlihy\|first1=Maurice\|last2=Liskov\|first2=Barbara\|author-link1=Maurice Herlihy\|author-link2=Barbara Liskov\|title=A Value Transmission Method for Abstract Data Types\|journal=[[ACM Transactions on Programming Languages and Systems]]\|date=October 1982\|volume=4\|issue=4\|pages=527–551\|doi=10.1145/69622.357182\|url=~~http~~://cs.brown.edu/~mph/HerlihyL82/p527-herlihy.pdf\|issn=0164-0925\|oclc=67989840\|citeseerx=10.1.1.87.5301\|s2cid=8126961}}</ref><ref>{{cite book\|last1=Birrell\|first1=Andrew\|last2=Jones\|first2=Mike\|last3=Wobber\|first3=Ted\|title=Proceedings of the eleventh ACM Symposium on Operating systems principles - SOSP '87 \|chapter=A simple and efficient implementation of a small database \|date=November 1987\|volume=11\|issue=5\|pages=149–154\|doi=10.1145/41457.37517\|isbn=089791242X \|issn=0163-5980\|oclc=476062921\|quote=Our implementation makes use of a mechanism called “pickles”, which will convert between any strongly typed data structure and a representation of that structure suitable for storing in permanent disk files. The operation Pickle.Write takes a pointer to a strongly typed data structure and delivers buffers of bits for writing to the disk. Conversely Pickle.Read reads buffers of bits from the disk and delivers a copy of the original data structure.() This conversion involves identifying the occurrences of addresses in the structure, and arranging that when the structure is read back from disk the addresses are replaced with addresses valid in the current execution environment. The pickle mechanism is entirely automatic: it is driven by the run-time typing structures that are present for our garbage collection mechanism. ... () Pickling is quite similar to the concept of marshalling in remote procedure calls. But in fact our pickling implementation works only by interpreting at run-time the structure of [[dynamically typed]] values, while our RPC implementation works only by generating code for the marshalling of statically typed values. Each facility would benefit from adding the mechanisms of the other, but that has not yet been done.\|citeseerx=10.1.1.100.1457\|s2cid=12908261}}</ref><ref>{{cite web\|last1=van Rossum\|first1=Guido\|author-link1=Guido van Rossum\|title=Flattening Python Objects\|url=http://legacy.python.org/workshops/1994-11/FlattenPython.html\|website=Python Programming Language – Legacy Website\|publisher=[[Python Software Foundation]]\|access-date=6 April 2017\|location=[[Delaware]], United States\|date=1 December 1994\|quote=Origin of the name 'flattening': Because I want to leave the original 'marshal' module alone, and Jim complained that 'serialization' also means something totally different that's actually relevant in the context of [[Concurrent computing\|concurrent]] access to persistent objects, I'll use the term 'flattening' from now on. ... (The Modula-3 system uses the term 'pickled' data for this concept. They have probably solved all problems already, and in a type-safe manner :-)}}</ref> to describe data serialization (''unpickling'' for ''deserializing''). Pickle uses a simple [[Stack (abstract data type)\|stack]]-based [[virtual machine]] that records the instructions used to reconstruct the object. It is a cross-version [https://docs.python.org/library/pickle.html#pickle-protocol customisable] but unsafe (not secure against erroneous or malicious data) serialization format. Malformed or maliciously constructed data, may cause the deserializer to import arbitrary modules and instantiate any object.<ref name=autogenerated1>{{cite web\|url=https://docs.python.org/2/library/pickle.html\|title=11.1. pickle — Python object serialization — Python 2.7.14rc1 documentation\|website=docs.python.org}}</ref><ref>{{cite web\|url=https://docs.python.org/release/3.0.1/library/pickle.html#pickle-restrict\|title=pickle — Python object serialization — Python v3.0.1 documentation\|website=docs.python.org}}</ref> The standard library also includes modules serializing to standard data formats: <code>[https://docs.python.org/library/json.html json]</code> (with built-in support for basic scalar and collection types and able to support arbitrary types via [https://docs.python.org/library/json.html#encoders-and-decoders encoding and decoding hooks]). <code>[https://docs.python.org/library/plistlib.html plistlib]</code> (with support for both binary and XML [[property list]] formats). <code>[https://docs.python.org/library/xdrlib.html xdrlib]</code> (with support for the External Data Representation (XDR) standard as described in RFC 1014). Finally, it is recommended that an object's <code>[https://docs.python.org/reference/datamodel.html#object.__repr__ __repr__]</code> be evaluable in the right environment, making it a rough match for Common Lisp's <code>[http://www.lispworks.com/documentation/HyperSpec/Body/f_pr_obj.htm print-object]</code>. Not all object types can be pickled automatically, especially ones that hold [[operating system]] resources like [[file handle]]s, but users can register custom "reduction" and construction functions to support the pickling and unpickling of arbitrary types. Pickle was originally implemented as the pure Python <code>pickle</code> module, but, in versions of Python prior to 3.0, the <code>cPickle</code> module (also a built-in) offers improved performance (up to 1000 times faster<ref name=autogenerated1 />). The <code>cPickle</code> was adapted from the [[Unladen Swallow]] project. In Python 3, users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version.<ref>{{cite web\|url=https://docs.python.org/release/3.1.5/whatsnew/3.0.html\|title=What's New In Python 3.0 — Python v3.1.5 documentation\|website=docs.python.org}}</ref>		The core general serialization mechanism is the <code>pickle</code> [[Python (programming language)#Libraries\|standard library]] module, alluding to the database systems term ''pickling''<ref>{{cite journal\|last1=Herlihy\|first1=Maurice\|last2=Liskov\|first2=Barbara\|author-link1=Maurice Herlihy\|author-link2=Barbara Liskov\|title=A Value Transmission Method for Abstract Data Types\|journal=[[ACM Transactions on Programming Languages and Systems]]\|date=October 1982\|volume=4\|issue=4\|pages=527–551\|doi=10.1145/69622.357182\|url=https://cs.brown.edu/~mph/HerlihyL82/p527-herlihy.pdf\|issn=0164-0925\|oclc=67989840\|citeseerx=10.1.1.87.5301\|s2cid=8126961}}</ref><ref>{{cite book\|last1=Birrell\|first1=Andrew\|last2=Jones\|first2=Mike\|last3=Wobber\|first3=Ted\|title=Proceedings of the eleventh ACM Symposium on Operating systems principles - SOSP '87 \|chapter=A simple and efficient implementation of a small database \|date=November 1987\|volume=11\|issue=5\|pages=149–154\|doi=10.1145/41457.37517\|isbn=089791242X \|issn=0163-5980\|oclc=476062921\|quote=Our implementation makes use of a mechanism called “pickles”, which will convert between any strongly typed data structure and a representation of that structure suitable for storing in permanent disk files. The operation Pickle.Write takes a pointer to a strongly typed data structure and delivers buffers of bits for writing to the disk. Conversely Pickle.Read reads buffers of bits from the disk and delivers a copy of the original data structure.() This conversion involves identifying the occurrences of addresses in the structure, and arranging that when the structure is read back from disk the addresses are replaced with addresses valid in the current execution environment. The pickle mechanism is entirely automatic: it is driven by the run-time typing structures that are present for our garbage collection mechanism. ... () Pickling is quite similar to the concept of marshalling in remote procedure calls. But in fact our pickling implementation works only by interpreting at run-time the structure of [[dynamically typed]] values, while our RPC implementation works only by generating code for the marshalling of statically typed values. Each facility would benefit from adding the mechanisms of the other, but that has not yet been done.\|citeseerx=10.1.1.100.1457\|s2cid=12908261}}</ref><ref>{{cite web\|last1=van Rossum\|first1=Guido\|author-link1=Guido van Rossum\|title=Flattening Python Objects\|url=http://legacy.python.org/workshops/1994-11/FlattenPython.html\|website=Python Programming Language – Legacy Website\|publisher=[[Python Software Foundation]]\|access-date=6 April 2017\|location=[[Delaware]], United States\|date=1 December 1994\|quote=Origin of the name 'flattening': Because I want to leave the original 'marshal' module alone, and Jim complained that 'serialization' also means something totally different that's actually relevant in the context of [[Concurrent computing\|concurrent]] access to persistent objects, I'll use the term 'flattening' from now on. ... (The Modula-3 system uses the term 'pickled' data for this concept. They have probably solved all problems already, and in a type-safe manner :-)}}</ref> to describe data serialization (''unpickling'' for ''deserializing''). Pickle uses a simple [[Stack (abstract data type)\|stack]]-based [[virtual machine]] that records the instructions used to reconstruct the object. It is a cross-version [https://docs.python.org/library/pickle.html#pickle-protocol customisable] but unsafe (not secure against erroneous or malicious data) serialization format. Malformed or maliciously constructed data, may cause the deserializer to import arbitrary modules and instantiate any object.<ref name=autogenerated1>{{cite web\|url=https://docs.python.org/2/library/pickle.html\|title=11.1. pickle — Python object serialization — Python 2.7.14rc1 documentation\|website=docs.python.org}}</ref><ref>{{cite web\|url=https://docs.python.org/release/3.0.1/library/pickle.html#pickle-restrict\|title=pickle — Python object serialization — Python v3.0.1 documentation\|website=docs.python.org}}</ref> The standard library also includes modules serializing to standard data formats: <code>[https://docs.python.org/library/json.html json]</code> (with built-in support for basic scalar and collection types and able to support arbitrary types via [https://docs.python.org/library/json.html#encoders-and-decoders encoding and decoding hooks]). <code>[https://docs.python.org/library/plistlib.html plistlib]</code> (with support for both binary and XML [[property list]] formats). <code>[https://docs.python.org/library/xdrlib.html xdrlib]</code> (with support for the External Data Representation (XDR) standard as described in RFC 1014). Finally, it is recommended that an object's <code>[https://docs.python.org/reference/datamodel.html#object.__repr__ __repr__]</code> be evaluable in the right environment, making it a rough match for Common Lisp's <code>[http://www.lispworks.com/documentation/HyperSpec/Body/f_pr_obj.htm print-object]</code>. Not all object types can be pickled automatically, especially ones that hold [[operating system]] resources like [[file handle]]s, but users can register custom "reduction" and construction functions to support the pickling and unpickling of arbitrary types. Pickle was originally implemented as the pure Python <code>pickle</code> module, but, in versions of Python prior to 3.0, the <code>cPickle</code> module (also a built-in) offers improved performance (up to 1000 times faster<ref name=autogenerated1 />). The <code>cPickle</code> was adapted from the [[Unladen Swallow]] project. In Python 3, users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version.<ref>{{cite web\|url=https://docs.python.org/release/3.1.5/whatsnew/3.0.html\|title=What's New In Python 3.0 — Python v3.1.5 documentation\|website=docs.python.org}}</ref>

	===R===		===R===
Line 115:		Line 117:

	===Smalltalk===		===Smalltalk===
	In general, non-recursive and non-sharing objects can be stored and retrieved in a human readable form using the <code>storeOn:</code>/<code>readFrom:</code> protocol. The <code>storeOn:</code> method generates the text of a Smalltalk expression which – when evaluated using <code>readFrom:</code> – recreates the original object. This scheme is special, in that it uses a procedural description of the object, not the data itself. It is therefore very flexible, allowing for classes to define more compact representations. However, in its original form, it does not handle cyclic data structures or preserve the identity of shared references (i.e. two references a single object will be restored as references to two equal, but not identical copies). For this, various portable and non-portable alternatives exist. Some of them are specific to a particular Smalltalk implementation or class library. There are several ways in [[Squeak\|Squeak Smalltalk]] to serialize and store objects. The easiest and most used are <code>storeOn:/readFrom:</code> and binary storage formats based on <code>SmartRefStream</code> serializers. In addition, bundled objects can be stored and retrieved using <code>ImageSegments</code>. Both provide a so-called "binary-object storage framework", which support serialization into and retrieval from a compact binary form. Both handle cyclic, recursive and shared structures, storage/retrieval of class and [[metaclass]] info and include mechanisms for "on the fly" object migration (i.e. to convert instances which were written by an older version of a class with a different object layout). The APIs are similar (storeBinary/readBinary), but the encoding details are different, making these two formats incompatible. However, the Smalltalk/X code is open source and free and can be loaded into other Smalltalks to allow for cross-dialect object interchange. Object serialization is not part of the ANSI Smalltalk specification. As a result, the code to serialize an object varies by Smalltalk implementation. The resulting binary data also varies. For instance, a serialized object created in Squeak Smalltalk cannot be restored in [[Ambrai Smalltalk]]. Consequently, various applications that do work on multiple Smalltalk implementations that rely on object serialization cannot share data between these different implementations. These applications include the MinneStore object database<ref>{{Cite web\|url=~~http~~://minnestore.sourceforge.net/\|archive-url=https://web.archive.org/web/20080511234145/http://minnestore.sourceforge.net/\|archive-date=11 May 2008\|website=SourceForge\|title=MinneStore version 2}}</ref> and some [[Remote procedure call\|RPC]] packages. A solution to this problem is SIXX,<ref>{{Cite web\|url=http://www.mars.dti.ne.jp/~umejava/smalltalk/sixx/index.html\|title=What's new\|access-date=25 July 2021\|website=SIXX - Smalltalk Instance eXchange in XML\|date=23 January 2010}}</ref> which is a package for multiple Smalltalks that uses an [[XML]]-based format for serialization.		In general, non-recursive and non-sharing objects can be stored and retrieved in a human readable form using the <code>storeOn:</code>/<code>readFrom:</code> protocol. The <code>storeOn:</code> method generates the text of a Smalltalk expression which – when evaluated using <code>readFrom:</code> – recreates the original object. This scheme is special, in that it uses a procedural description of the object, not the data itself. It is therefore very flexible, allowing for classes to define more compact representations. However, in its original form, it does not handle cyclic data structures or preserve the identity of shared references (i.e. two references a single object will be restored as references to two equal, but not identical copies). For this, various portable and non-portable alternatives exist. Some of them are specific to a particular Smalltalk implementation or class library. There are several ways in [[Squeak\|Squeak Smalltalk]] to serialize and store objects. The easiest and most used are <code>storeOn:/readFrom:</code> and binary storage formats based on <code>SmartRefStream</code> serializers. In addition, bundled objects can be stored and retrieved using <code>ImageSegments</code>. Both provide a so-called "binary-object storage framework", which support serialization into and retrieval from a compact binary form. Both handle cyclic, recursive and shared structures, storage/retrieval of class and [[metaclass]] info and include mechanisms for "on the fly" object migration (i.e. to convert instances which were written by an older version of a class with a different object layout). The APIs are similar (storeBinary/readBinary), but the encoding details are different, making these two formats incompatible. However, the Smalltalk/X code is open source and free and can be loaded into other Smalltalks to allow for cross-dialect object interchange. Object serialization is not part of the ANSI Smalltalk specification. As a result, the code to serialize an object varies by Smalltalk implementation. The resulting binary data also varies. For instance, a serialized object created in Squeak Smalltalk cannot be restored in [[Ambrai Smalltalk]]. Consequently, various applications that do work on multiple Smalltalk implementations that rely on object serialization cannot share data between these different implementations. These applications include the MinneStore object database<ref>{{Cite web\|url=https://minnestore.sourceforge.net/\|archive-url=https://web.archive.org/web/20080511234145/http://minnestore.sourceforge.net/\|archive-date=11 May 2008\|website=SourceForge\|title=MinneStore version 2}}</ref> and some [[Remote procedure call\|RPC]] packages. A solution to this problem is SIXX,<ref>{{Cite web\|url=http://www.mars.dti.ne.jp/~umejava/smalltalk/sixx/index.html\|title=What's new\|access-date=25 July 2021\|website=SIXX - Smalltalk Instance eXchange in XML\|date=23 January 2010}}</ref> which is a package for multiple Smalltalks that uses an [[XML]]-based format for serialization.

	===Swift===		===Swift===

Serialization - Revision history

imported>Cicada1010: Interceptor: Reverting edits by ~2026-31563-52

~2025-34236-49: /* Serialization formats */

2001:1C00:150F:4F00:215B:4B0F:B503:631E: Link to official docs (ruby-doc.org is not official)