Wisdom from the agesWhen you serialize information, build into the format some amount of redundancy. Not too much, but not too little. Here's an example.
In v1 of the networking protocol, the server sends to the client a single packet to describe several Foos at once.
Header (8 bytes)
Foo #0 (32 bytes)
Foo #1 (32 bytes)
Foo #2 (32 bytes)...for a total packet length of 104 bytes about 3 Foos. When there are 4 Foos, we'll send 136 bytes, when there are 5 Foos, we'll send 168 bytes, and so on. One header per packet, followed by the Foos.
When the v1 client receives the packet, let's assume he knows only the length of the payload. To deduce the number of Foos in the packet, the client will deduct 8 from the length for the header, and divide the remaining number of bytes by 32. Very nice.
Now, in v2 of the networking protocol, it turns out that the Foos gained another special feature, let's call it X. Feature X is optional, Foo works with or without it in v2. We don't want to cramp X into the existing 32 bytes; instead, we decide to send 40 bytes per Foo with X.
Header (8 bytes)
FooX #0 (40 bytes = 32 for the old Foo + 8 for X)
FooX #1 (40 bytes = 32 for the old Foo + 8 for X)
FooX #2 (40 bytes = 32 for the old Foo + 8 for X)The server happily sends out v2 to the newly updated clients that understand Foo with X.
Naively, you can't send these packets to a v1 client. If a v1 client gets a packet of length 88 with two FooX, he will deduct 8 for the header, and then decide that 80 isn't divisible by 32. Even if he doggedly interprets the bytes starting at offset 8 correctly as Foo #0, he will misinterpret the bytes at offset 8+32 as Foo #1 even though Foo #1 starts at offset 8+40.
Even worse: When you send a packet with 4 FooX to a v1 client, he'll be thoroughly confused because the packet length of 168 bytes makes complete sense to him. He'll divide 160 by 32, interprets it as 5 Foos, and won't even notice the problem from the packet length.
For the v1 clients, there are three possibilities in this new world where all the cool kids are doing X, but they're still happy with the X-less Foo:
- Don't allow v1 clients on the v2 server.
- Write extra decisions into the server to send v1 packets to v1 clients and v2 packets to v2 clients, and also take care to interpret packets coming from v1 clients one way, and v2 packets another way. Lots of extra code in the server. Who's up for long nights of romantic object-oriented refactoring?
- When you write the v1 server, anticipate the need for growing Foo in v2 six years down the road. Don't make the clients rely purely on the division by a previously agreed struct length. Already in v1, write the length per Foo into the header. This way, clients (both v1 and v2) can receive the v2 FooX packets correctly: Both deduct 8 for the header, then iterate the remaining bytes by the stated length of FooX, 40, and interpret the first 32 bytes correctly as Foo #0. The v2 clients will know, in addition, what to do with the following 8 bytes for X #0. The v1 clients will cleanly discard those 8 bytes before starting to interpret Foo #1.
Instead of writing the length of a transmitted Foo, you can also write the number of Foo, assuming the receiver knows the total packet length and will then divide. Either allows the clients to deduce the other from the total packet length. For the recipient, I feel it's easier when he's told the length per Foo.
I'm sure there are more related ideas in network serialization, but this one is particularly annoying and enlightening in hindsight. It's more natural to anticipate this when you've designed binary file formats in the past. I've always avoided inventing binary file formats and it shows in the networking packet design.
The art in protocol or format design, as in code design, is to anticipate the most likely change, and keep the design open to changes in that direction. It's okay if most other changes are hard and need more code.
-- Simon