Marshal格式

2002-04-04 草稿....

• 以4.8(对应于1.8)版的格式为蓝本

  # 截至2003-05-02为止的格式版本如下所示 p Marshal.Dump(Object.new).unpack("cc").join(".") => ruby 1.6.0 (2000-09-19) [i586-linux] "4.4" => ruby 1.6.1 (2000-09-27) [i586-linux] "4.4" => ruby 1.6.2 (2000-12-25) [i586-linux] "4.5" => ruby 1.6.3 (2001-03-19) [i586-linux] "4.5" => ruby 1.6.4 (2001-06-04) [i586-linux] "4.5" => ruby 1.6.5 (2001-09-19) [i586-linux] "4.6" => ruby 1.6.6 (2001-12-26) [i586-linux] "4.6" => ruby 1.6.7 (2002-03-01) [i586-linux] "4.6" => ruby 1.6.7 (2002-09-06) [i586-linux] "4.6" => ruby 1.7.3 (2002-09-06) [i586-linux] "4.7" => ruby 1.7.3 (2002-09-20) [i586-linux] "4.8" => ruby 1.8.0 (2003-08-03) [i586-linux] "4.8"

• 本文兼顾了以前的版本，同时也指出了兼容性问题
• 还提到了Ruby的Marshal中的BUG(?)

nil

true

false

分别是'0', 'T', 'F'

p Marshal.Dump(nil).unpack("x2 a*") # => ["0"]

此时，即使设置了实例变量也无法Dump。

class NilClass attr_accessor :foo end nil.foo = 1 p nil.foo # => 1 p Marshal.Dump(nil).unpack("x2 a*") # => ["0"]

Fixnum

在'i'之后是表示Fixnum的数据结构。

以数值n为例，在表示数值部分的形式中(不仅限于Fixnum，在其它地方也是如此)，保存着

形式 1:

n == 0: 0 0 < n < 123: n + 5 -124 < n < 0: n - 5

这样的数值(1 byte)。之所以加减5，是为了有别于下面的形式。

例:

p Marshal.Dump(-1).unpack("x2 a*") # => "i\372" p Marshal.Dump(0).unpack("x2 a*") # => "i\000" p Marshal.Dump(1).unpack("x2 a*") # => "i\006" p Marshal.Dump(2).unpack("x2 a*") # => "i\a" ("i\007")

若数值N超出形式1的范围时，则有下面的形式。

形式 2:

| len | n1 | n2 | n3 | n4 | <-1-> <- len -> byte bytes

len的值是-4 ～ -1, 1 ～ 4。这表示符号和后续的数据存在于n1 ～ n|len|。

# 举个更好的例子... def foo(len, n1, n2 = 0, n3 = 0, n4 = 0) case len when -3; n4 = 255 when -2; n3 = n4 = 255 when -1; n2 = n3 = n4 = 255 end n = (0xffffff00 | n1) & (0xffff00ff | n2 * 0x100) & (0xff00ffff | n3 * 0x10000) & (0x00ffffff | n4 * 0x1000000) # p "%x" % n n = -((n ^ 0xffff_ffff) + 1) if len < 0 n end p Marshal.Dump(-125).unpack("x2 acC*") # => ["i", -1, 131] p foo(-1, 131) p Marshal.Dump(-255).unpack("x2 acC*") # => ["i", -1, 1] p foo(-1, 1) p Marshal.Dump(-256).unpack("x2 acC*") # => ["i", -1, 0] p foo(-1, 0) p Marshal.Dump(-257).unpack("x2 acC*") # => ["i", -2, 255, 254] p foo(-2, 255, 254) p Marshal.Dump(124).unpack("x2 acC*") # => ["i", 1, 124] p foo(1, 124) p Marshal.Dump(256).unpack("x2 acC*") # => ["i", 2, 0, 1] p foo(2, 0, 1)

即使设定了实例变量，也无法Dump。

class Fixnum attr_accessor :foo end 99.foo = 1 p 99.foo # => 1 p 999.foo # => nil p Marshal.Dump(99).unpack("x2 ac") # => ["i", 104]

instance of the user class

'C': String, Regexp, Array, Hash 的子类的实例变量

| 'C' | 类名(Symbol)的 Dump | 父类的实例的 Dump |

例 1:

class Foo < String # (or Regexp, Array, Hash) end p Marshal.Dump(Foo.new("foo")).unpack("x2 a a c a3 aca*") # => ["C", ":", 8, "Foo", "\"", 8, "foo"] ^^^ (or '/', '[', '{')

例 2: 有实例变量(请参考instance variable)

class Foo < String # (or Regexp, Array, Hash) def initialize(obj) @foo = obj super(obj) end end p Marshal.Dump(Foo.new("foo")).unpack("x2 a a a c a3 aca3 caca4 aca*") # => ["I", "C", ":", 8, "Foo", "\"", 8, "foo", 6, ":", 9, "@foo", "\"", 8, "foo"]

除此以外，将变为'o'。这是因为内部结构有所差异所致(请参考Object)

例:

class Foo end p Marshal.Dump(Foo.new).unpack("x2 a a c a*") # => ["o", ":", 8, "Foo\000"]

'u'

若定义了_Dump、_load的话，就是'u'。因为无法Dump实例变量，所以必须使用_Dump/_load进行处理。

| 'u' | 类名(Symbol)的 Dump | _Dump 的结果的长度(Fixnum形式) | | _Dump 的返回值 |

例:

class Foo def self._load end def _Dump(obj) "hogehoge" end end p Marshal.Dump(Foo.new).unpack("x2 a aca3 c a*") # => ["u", ":", 8, "Foo", 13, "hogehoge"]

'U'ruby 1.8 特性

若定义了marshal_Dump、marshal_load的话，就是'U'。因为无法Dump实例变量，所以必须使用marshal_Dump/marshal_load来处理。

| 'U' | 类名(Symbol)的 Dump | marshal_Dump 方法的返回值的 Dump |

例:

class Foo def marshal_Dump "hogehoge" end def marshal_load(obj) end end p Marshal.Dump(Foo.new).unpack("x2 a aca3 a c a*") # => ["U", ":", 8, "Foo", "\"", 13, "hogehoge"]

Object

'o'

| 'o' | 类名(Symbol)的 Dump | 实例变量的数量(Fixnum形式) | | 实例变量名(Symbol) 的Dump(1) | 值(1) | : : | 实例变量名(Symbol) 的Dump(n) | 值(n) |

例 1:

p Marshal.Dump(Object.new).unpack("x2 a a c a*") # => ["o", ":", 11, "Object\000"]

例 2: 有实例变量

class Foo def initialize @foo = "foo" @bar = "bar" end end p Marshal.Dump(Foo.new).unpack("x2 a a c a3 c aca4 aca3 aca4 aca3") # => ["o", ":", 8, "Foo", 7, ":", 9, "@bar", "\"", 8, "bar", ":", 9, "@foo", "\"", 8, "foo"]

Float

'f'

| 'f' | 数串的长度(Fixnum形式) | "%.16g" 的字符串 |

例:

p Marshal.Dump(Math::PI).unpack("x2 a c a*") # => ["f", 22, "3.141592653589793"] p Marshal.Dump(0.0/0).unpack("x2 a c a*") # => ["f", 8, "nan"] p Marshal.Dump(1.0/0).unpack("x2 a c a*") # => ["f", 8, "inf"] p Marshal.Dump(-1.0/0).unpack("x2 a c a*") # => ["f", 9, "-inf"] p Marshal.Dump(-0.0).unpack("x2 a c a*") # => ["f", 9, "-0"]

Bignum

'l'

| 'l' | '+'/'-' | short的个数(Fixnum形式) | ... |

例:

p Marshal.Dump(2**32).unpack("x2 a a c a*") # => ["l", "+", 8, "\000\000\000\000\001\000"] # => ["l", "+", 8, "\000\000\001\000"] <- BUG: ruby version 1.6.3

String

'"'

| '"' | 长度(Fixnum形式) | 字符串 |

例:

p Marshal.Dump("hogehoge").unpack("x2 a c a*") # => ["\"", 13, "hogehoge"]

Regexp

'/'

| '/' | 长度(Fixnum形式) | source字符串 | 选项 |

选项是options的结果+汉字代码的flag值。

例:

p Marshal.Dump(/(hoge)*/).unpack("x2 a c a7 c") # => ["/", 12, "(hoge)*", 0] p Marshal.Dump(/hogehoge/m).unpack("x2 a c a8 c") # => ["/", 13, "hogehoge", 4] p Marshal.Dump(/hogehoge/e).unpack("x2 a c a8 c") # => ["/", 13, "hogehoge", 32]

Array

'['

| '[' | 元素数(Fixnum形式) | 元素的 Dump | ... |

例:

p Marshal.Dump(["hogehoge", /hogehoge/]).unpack("x2 a c aca8 aca*") # => ["[", 7, "\"", 13, "hogehoge", "/", 13, "hogehoge\000"]

Hash

'{'

| '{' | 元素数(Fixnum形式) | 键的 Dump | 值的 Dump | ... |

例:

p Marshal.Dump({"hogehoge", /hogehoge/}).unpack("x2 a c aca8 aca*") # => ["{", 6, "\"", 13, "hogehoge", "/", 13, "hogehoge\000"]

Hash with default value ( not Proc )

'}'

| '}' | 元素数(Fixnum形式) | 键的 Dump | 值的 Dump | ... | 默认值 |

例:

h = Hash.new(true) h["foo"] = "bar" p Marshal.Dump(h).unpack("x2 a c aca3 aca*") # => ["}", 6, "\"", 8, "foo", "\"", 8, "barT"]

若某Hash的默认对象是Proc的话，则无法Dump该Hash

h = Hash.new { } Marshal.Dump(h) => -:2:in `Dump': cannot Dump hash with default proc (TypeError)

Struct

'S': 结构体类的实例的Dump

| 'S' | 类名(Symbol) 的 Dump | 成员数量(Fixnum形式) | | 成员名(Symbol) 的 Dump | 值 | ... |

例:

Struct.new("XXX", :foo, :bar) p Marshal.Dump(Struct::XXX.new).unpack("x2 a ac a11 c aca3a aca3a") # => ["S", ":", 16, "Struct::XXX", 7, ":", 8, "foo", "0", ":", 8, "bar", "0"]

Class/Module (old format)

'M'

| 'M' | 长度(Fixnum形式) | 模块/类名 |

例: 因为已经无法dump这种形式，所以使用load进行说明。

class Mod end p Marshal.load([4,7, 'M', 3+5, 'Mod'].pack("ccaca*")) # => Mod

Class/Module

'c', 'm'

| 'c'/'m' | 类名的长度(Fixnum 形式) | 类名 |

例:

class Foo end p Marshal.Dump(Foo).unpack("x2 a c a*") # => ["c", 8, "Foo"]

例 2: 无法dump类/模块的实例变量

module Bar @bar = 1 end p Bar.instance_eval { @bar } Marshal.Dump(Bar, open("/tmp/foo", "w")) # => 1 module Bar end p bar = Marshal.load(open("/tmp/foo")) p bar.instance_eval { @bar } # => nil

例 3: 无法dump类变量

module Baz @@baz = 1 def self.baz @@baz end end p Baz.baz Marshal.Dump(Baz, open("/tmp/foo", "w")) # => 1 module Baz def self.baz @@baz end end p baz = Marshal.load(open("/tmp/foo")) baz.baz # => Baz -:3:in `baz': uninitialized class variable @@baz in Baz (NameError) from -:7

Symbol

':'

| ':' | 符号名的长度(Fixnum形式) | 符号名 |

例:

p Marshal.Dump(:foo).unpack("x2 a c a*") # => [":", 8, "foo"]

Symbol (link)

';'

| ';' | 表明Symbol实际状态的号码(Fixnum形式) |

在相应符号名已被dump/load时使用。该号码是内部管理的号码。(在dump/load时，会生成哈希表以便对Symbol进行管理。它表示记录位置)

例:

p Marshal.Dump([:foo, :foo]).unpack("x2 ac aca3 aC*") # => ["[", 7, ":", 8, "foo", ";", 0] p Marshal.Dump([:foo, :foo, :bar, :bar]). unpack("x2 ac aca3 aC aca3 aC*") # => ["[", 9, ":", 8, "foo", ";", 0, ":", 8, "bar", ";", 6]

instance variable

'I': Object, Class, Module 的实例以外的对象

| 'I' | 对象的 Dump | 实例变量的数量(Fixnum形式) | | 实例变量名(Symbol) 的Dump(1) | 值(1) | : : | 实例变量名(Symbol) 的Dump(n) | 值(n) |

因为Object的实例中包含实例变量，所以会采用其他的形式进行Dump(请参考Object)。该形式只针对Array 或 String 的实例。

例:

obj = String.new obj.instance_eval { @foo = "bar" } p Marshal.Dump(obj).unpack("x2 a ac c a c a4 aca*") # => ["I", "\"", 0, 6, ":", 9, "@foo", "\"", 8, "bar"]

类或模块(Class/Module的实例)不会dump实例变量的信息。(请参考Class/Module)

link

'@'

| '@' | 表明对象实际状态的号码(Fixnum形式 |

在相应对象已被dump/load时使用。该号码是内部管理的号码。(在dump/load时，会生成哈希表以便对对象进行管理。它表示记录位置)

例:

obj = Object.new p Marshal.Dump([obj, obj]).unpack("x2 ac aaca6c aca*") # => ["[", 7, "o", ":", 11, "Object", 0, "@", 6, ""] ary = [] ary.push ary p Marshal.Dump(ary).unpack("x2 acac") # => ["[", 6, "@", 0]

Marshal 的BUG

在ruby version 1.6中发现了下列BUG。括号()中列出的是正确的运作方式(1.7的运作方式)。

<= 1.6.7

• 类的clone中的实例是可以Dump的，但却不能加载(因为是无名类的对象，所以无法Dump)
• 当某对象通过include/extend无名Module而定义了特殊方法后，仍可以Dump/加载该对象(若某对象include了无名模块的话，则不能Dump该对象)

1.6.6, 1.6.7

• 拥有实例变量的Array和String是可以Dump的，但却不能加载(既能Dump，又能加载)

<= 1.6.5

• 类的clone中的实例是可以Dump的，但却不能正常加载。否则就会生成奇怪的对象(?)
• 特殊类被dump成为普通类了(特殊类是无法Dump的)
• 无名类是可以Dump的，但却不能加载(无名类是无法Dump的)

<= 1.6.4

• 模块可以Dump却不能加载(可以加载)
• 无名模块可以Dump却不能加载(无名模块是不能Dump的)

<= 1.6.3

• dump Float时，其保存精度偏低

<= 1.6.2

• dump时，无法保存正则表达式中/m, /x 选项的状态

1.6.2, 1.6.3

• 在1.6.2, 1.6.3中，Bignum可以Dump却不能加载。按理说其他版本也会有这个BUG，但因为没有测试脚本，所以无法证实。

<= 1.6.1

• dump时无法保存Range中的特定标识，该标识表明该范围中是否包含终点

下面就是测试脚本(请参考[RAA:RubyUnit])

# test for Marshal for ruby version 1.6 require 'rubyunit' $version_dependent_behavior = true # for test_userClass, test_userModule module UserModule def foo end end class UserClass def foo end end class TestMarshal < RUNIT::TestCase def assert_no_Dumpable(obj) ex = assert_exception(TypeError) { begin # Marshal.Dump will cause TypeError or ArgumentError Marshal.Dump obj rescue ArgumentError case $!.message when /can't Dump anonymous/, /cannot Dump hash with default proc/ raise TypeError else raise "unknown error" end end } end def assert_Dumpable_but_not_equal(obj) obj2 = Marshal.load(Marshal.Dump(obj)) assert(obj != obj2) assert_equals(obj.type, obj2.type) end def assert_Dumpable_and_equal(obj) obj2 = Marshal.load(Marshal.Dump(obj)) assert_equals(obj, obj2) assert_equals(obj.type, obj2.type) # check values of instance variable ivars = obj.instance_variables ivars2 = obj2.instance_variables assert_equals(ivars, ivars2) while ivars.size != 0 assert_equals(obj.instance_eval(ivars.shift), obj2.instance_eval(ivars2.shift)) end end def test_Object assert_Dumpable_but_not_equal Object.new end # object with singleton method def test_Object_with_singleton_method obj = Object.new # On ruby version 1.6.0 - 1.6.2, cause parse error (nested method) class <"1",2=>"2") # 1.7 feature. if $version_dependent_behavior and RUBY_VERSION >= '1.7.0' # On ruby version 1.7, hash with default Proc cannot be Dumped. # see [ruby-dev:15417] assert_no_Dumpable(Hash.new { }) end end def test_IO assert_no_Dumpable IO.new(0) end def test_File assert_no_Dumpable File.open("/") end def test_MatchData assert_no_Dumpable(/foo/ =~ "foo" && $~) end def test_Method assert_no_Dumpable Object.method(:method) end def test_UnboundMethod assert_no_Dumpable Object.instance_method(:id) end def test_Module # On ruby version 1.6.0 - 1.6.4, loaded module is not a module. if $version_dependent_behavior and RUBY_VERSION <= '1.6.4' Dump = Marshal.Dump Enumerable ex = assert_exception(TypeError) { Marshal.load Dump } assert_matches(ex.message, /is not a module/) else assert_Dumpable_and_equal Enumerable end end def test_userModule # On ruby version 1.6.0 - 1.6.4, loaded module is not a module. if $version_dependent_behavior and RUBY_VERSION <= '1.6.4' # same as test_Module else # Note: this module must be defineed for Marshal.load. assert_Dumpable_and_equal(UserModule) end end def test_anonymousModule # On ruby version 1.6.0 - 1.6.4, anonymous class is able to be Dumped, # but loaded object is not identical. if $version_dependent_behavior and RUBY_VERSION <= '1.6.4' Dump = Marshal.Dump(Module.new) ex = assert_exception(ArgumentError) { Marshal.load Dump } assert_matches(ex.message, /can\'t retrieve anonymous class/) else assert_no_Dumpable Module.new end end def test_Class assert_Dumpable_and_equal Class end def test_userClass # Note: this class must be defineed for Marshal.load. assert_Dumpable_and_equal(UserClass) end def test_anonymousClass # On ruby version 1.6.0 - 1.6.5, anonymous class able to be Dumped, # but can't load it. if $version_dependent_behavior and RUBY_VERSION <= '1.6.5' Dump = Marshal.Dump(Class.new) ex = assert_exception(ArgumentError) { Marshal.load(Dump) } assert_matches(ex.message, /can\'t retrieve anonymous class/) else assert_no_Dumpable Class.new end end def test_clonedClass # On ruby version 1.6.0 - 1.6.7, instance of cloned class is able to # Dumped, but loaded object is not identical. # see [ruby-dev:14961] if $version_dependent_behavior if RUBY_VERSION <= '1.6.5' obj = String.clone.new("foo") Dump = Marshal.Dump(obj) obj2 = Marshal.load Dump assert(obj == obj2) assert(obj.type != obj2.type) assert(obj.type.inspect == obj2.type.inspect) elsif RUBY_VERSION <= '1.6.7' Dump = Marshal.Dump(String.clone.new("foo")) assert_exception(ArgumentError) { Marshal.load Dump } else assert_no_Dumpable String.clone.new("foo") end else # anonymous class can't be Dumped assert_no_Dumpable String.clone.new("foo") end end def test_Numeric # assert_fail("") end def test_Integer # assert_fail("") end def test_Fixnum assert_Dumpable_and_equal 100 end def test_Bignum # derived from Rubicon assert_Dumpable_and_equal 123456789012345678901234567890 assert_Dumpable_and_equal -123**99 if $version_dependent_behavior and %w(1.6.2 1.6.3).member?(RUBY_VERSION) Dump = Marshal.Dump 2**32 ex = assert_exception(ArgumentError) { Marshal.load(Dump) } assert_matches(ex.message, /marshal data too short/) else assert_Dumpable_and_equal 2**32 end end def test_Float assert_Dumpable_and_equal 1.41421356 # On ruby version 1.6.4, Dumped format changed from "%.12g" to "%.16g" if $version_dependent_behavior and RUBY_VERSION <= '1.6.3' assert_Dumpable_but_not_equal Math::PI else assert_Dumpable_and_equal Math::PI end end def test_Proc assert_no_Dumpable proc { } end def test_Process__Status assert_Dumpable_and_equal system("true") && $? end def test_Range # Range#== is changed from 1.6.2 # On ruby version 1.6.0 - 1.6.1, Range.new(1,2) != Range.new(1,2) # assert_Dumpable_and_equal 1..2 # assert_Dumpable_and_equal 1...2 obj = Marshal.load(Marshal.Dump 1..2) assert_equals(1, obj.begin) assert_equals(2, obj.end) assert_equals(false, obj.exclude_end?) obj = Marshal.load(Marshal.Dump 1...2) assert_equals(1, obj.begin) assert_equals(2, obj.end) # On ruby version 1.6.0 - 1.6.1, the attribute exclude_end? is not saved. if $version_dependent_behavior and RUBY_VERSION <= '1.6.1' assert_equals(false, obj.exclude_end?) else assert_equals(true, obj.exclude_end?) end end def test_Regexp # this test is no consider the /foo/p assert_Dumpable_and_equal /foo/ assert_Dumpable_and_equal /foo/i assert_Dumpable_and_equal /foo/m assert_Dumpable_and_equal /foo/x assert_Dumpable_and_equal /foo/e assert_Dumpable_and_equal /foo/s assert_Dumpable_and_equal /foo/u # On ruby version 1.6.0 - 1.6.2, Regexp#== is ignore the option. for obj in [/foo/, /foo/i, /foo/m, /foo/x, /foo/e, /foo/s, /foo/u] obj2 = Marshal.load(Marshal.Dump obj) if $version_dependent_behavior and RUBY_VERSION <= '1.6.2' and %w(/foo/m /foo/x).member?(obj.inspect) # On ruby version 1.6.0 - 1.6.2, # //m options is not saved. assert_equals('/foo/', obj2.inspect) else assert_equals(obj.inspect, obj2.inspect) end end end def test_String assert_Dumpable_and_equal "foo" end def test_Struct assert_Dumpable_and_equal Struct.new("Foo", :foo, :bar) Object.const_set('Foo', Struct.new(:foo, :bar)) assert_Dumpable_and_equal Foo end def test_aStruct assert_Dumpable_and_equal Struct.new("Bar", :foo, :bar).new("foo", "bar") # see [ruby-dev:14961] end def test_Symbol assert_Dumpable_and_equal :foo end def test_Thread assert_no_Dumpable Thread.new { sleep } end def test_ThreadGroup assert_no_Dumpable ThreadGroup::Default end def test_Time assert_Dumpable_and_equal Time.now assert_Dumpable_and_equal Time.now.gmtime # time zone is not saved. assert_equals(false, Marshal.load(Marshal.Dump(Time.now)).utc?) assert_equals(false, Marshal.load(Marshal.Dump(Time.now.gmtime)).utc?) end def test_TrueClass assert_Dumpable_and_equal true end def test_NilClass assert_Dumpable_and_equal nil end end