Round database unit to the nearest attometer when reading from GDS2

[rocallahan]

Currently if you create a layout with a DBU of (e.g.) 16, write it to GDS2, and then read it again, you don't get exactly 16, but 16.000000000000004. It would be nice if reasonable database unit values roundtrip exactly.

[rocallahan]

I guess I need to update other values to be consistent. I'll look into that tomorrow.

[klayoutmatthias]

Good point, but I guess the problem is somewhere else.

Basically GDS is not able to exactly represent a DBU of 16 Mikrometers (OASIS and GDSTXT is). It has to pick a value close by and this is the hex string:

3d 10 c6 f7 a0 b5 ed 8e

Which decoded as base-16 float is:

16**-3*(0x10c6f7a0b5ed8d/0x100000000000000)
=
16.0000000000000026641015782313814952431130222976207733154296875

Due to the limitation of standard double float I guess this is evaluated as 16.000000000000004, so close by.

When I write such a GDS back, I get the same DBU hex string, so a round trip is safe. Your patch is just cosmetics as inside the GDS, the value would still not be 16 micron and any other tool reading the file sees this odd value.

This is the point where I'd say there is little one can do, but it ocured to me that by picking a hex string that is off by 1:

3d 10 c6 f7 a0 b5 ed 8d

renders

16**-3*(0x10c6f7a0b5ed8d/0x100000000000000)
=
15.99999999999999927596978921418013896982301957905292510986328125...

which is actually turned into 16.0.

So I wonder why the GDS writer does not pick the smaller and more precise number. There is no obvious bug to me in the conversion code. There is some subtle numerical issue however - the following patch produces a better fit for the DBU value:

diff --git a/src/plugins/streamers/gds2/db_plugin/dbGDS2Writer.cc b/src/plugins/streamers/gds2/db_plugin/dbGDS2Writer.cc
index b5bcd2626..b3911f9da 100644
--- a/src/plugins/streamers/gds2/db_plugin/dbGDS2Writer.cc
+++ b/src/plugins/streamers/gds2/db_plugin/dbGDS2Writer.cc
@@ -107,7 +107,7 @@ GDS2Writer::write_double (double d)
   tl_assert (e >= -64 && e < 64);
   b[0] |= ((e + 64) & 0x7f);
 
-  uint64_t m = uint64_t (d + 0.5);
+  uint64_t m = uint64_t (round (d));
   for (int i = 7; i > 0; --i) {
     b[i] = (m & 0xff);
     m >>= 8;

I tend towards going that way.

[rocallahan]

I tend towards going that way.

Yes, that will work for me. Thanks! I assume you don't need anything more from me.