Truncate BigInts when setting UintXArray or IntXArray instead of throwing

[seishun]

Currently, the following throws a TypeError:

const val = 123412341234n;
const buf = new Uint8Array(8);
buf[0] = val;

Simply casting to Number is not sufficient, since it can cause a loss of precision. The code for writing a 64-bit integer into a Node.js Buffer instance (which is also a Uint8Array instance) would look something like this:

  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);
  value = value >> 8n;
  buf[offset++] = Number(value & 0xffn);

This is not only verbose, but also weird, because both Number and BigInt are supersets of Uint8.

This is clearly not a case of avoiding silent truncations, since all TypedArrays (including the new BigInt64Array and BigUint64Array) still allow them when setting values of the correct type. For example, the following silently truncates according to the spec:

const val = 123412341234123412341234123412341234123412341234123412341234n;
const buf = new BigUint64Array(1);
buf[0] = val;

My informal proposal: when setting indexes of Int8Array, Uint8Array, Int16Array, Uint16Array, Int32Array or Uint32Array to BigInt values, it should do the same as with Number values - that is, set the index to the value modulo 2^8, 2^16 or 2^32, respectively.

Note: this is not the same as #106. That patch performed a conversion to Number, which can cause a loss of precision.

[littledan]

The decision to not do #106 wasn't centered on losing data after the decimal point. It was more about having a mental model which is based on keeping track of whether you are working with a Number or a BigInt. See the committee notes. What do you think of this argument?

[seishun]

losing data after the decimal point

That's not the issue. Consider the following code:

const val = 0x10000000000000000000000000001n;
const buf = new Uint8Array(8);
buf[0] = val;

With #106, buf[0] would be equal to 0 instead of the expected 1 due to the implicit conversion to Number.

It was more about having a mental model which is based on keeping track of whether you are working with a Number or a BigInt. See the committee notes. What do you think of this argument?

The only argument I found in the notes that seems to apply to my suggestion is the following:

With the ones we're introducing, if you write a value into a typed array and the value is either a Number or a BigInt, in the range of the typed array, and read it back, depending on which typed array you use, it may be the same type or it might be the other type.

I don't see what issues this can cause in practice. In fact, it's already the case with IntXArrays and UintXArrays, since they allow you to write a string, for example.

I do agree with the following argument:

For some you get the correct modulo mathematical value result, for others you get rounding then modulo, pandora's box

But it applies to #106 and is not an issue if modulo is applied to the numeric value directly, without converting to Number.

[littledan]

@waldemarhorwat said,

I'd either like to see no coercions, or if we want genericity we should add BigInt arrays of all widths. It would let folks standardize on using just those and not have to wonder whether they're getting a Number or a BigInt each time

What do you say to this concern?

[seishun]

I'd either like to see no coercions

Me too. But my proposal doesn't involve any coercions (any more than Numbers are now "coerced" into uint8 when writing into Uint8Array).

we should add BigInt arrays of all widths. It would let folks standardize on using just those and not have to wonder whether they're getting a Number or a BigInt each time

Adding a set of BigIntXArrays and BigUintXArrays just for the type you get when reading from it doesn't seem worthwhile to me, considering you can convert it yourself by calling BigInt(). Even if they are added, I would still argue for allowing writing both Numbers and BigInts to IntXArrays and UintXArrays. For instance, Node.js's Buffer is backed by Uint8Array and can't switch to BigUint8Array.

[littledan]

But my proposal doesn't involve any coercions (any more than Numbers are now "coerced" into uint8 when writing into Uint8Array).

We discussed this point at TC39, about there already being coercions and information loss and this not making it worse. However, the point here is about coercions between types. I don't think TC39 will permit coercions between types here, as it's a big mismatch with our general no-coercions design.

Adding a set of BigIntXArrays and BigUintXArrays just for the type you get when reading from it doesn't seem worthwhile to me, considering you can convert it yourself by calling BigInt()

Agreed. I don't want to add these. For this reason, if we frame things according to @waldemarhorwat's choice, I'd rather take the no coercions route. I believe we have discussed the idea of adding coercions but not array types and it was rejected already, without resting on the decision of whether to round or throw after the decimal point.

[seishun]

However, the point here is about coercions between types.

This proposal doesn't add any new coercions between types either. TypedArrays don't store values as Numbers. When you write a Number to a Uint8Array and then read it, the conversion goes Number -> byte -> Number. With this proposal, if you write a BigInt and then read it, it goes BigInt -> byte -> Number. Where's the new coercion here?

For this reason, if we frame things according to @waldemarhorwat's choice, I'd rather take the no coercions route.

I don't understand why not adding these leads to the no coercions route.

without resting on the decision of whether to round

@waldemarhorwat mentioned rounding as a problem, see the last quote in #137 (comment).

throw after the decimal point.

What does this mean? The decimal point is not relevant here, see my example in #137 (comment).

[littledan]

I'd really prefer not to have to present a change to the committee unless there is a very strong motivation for the change, given the back and forth so far. It seems like you can get by without this change with an explicit cast. When working with BigInts and Numbers, it's necessary to understand which type you are working with, given how arithmetic operations do not permit mixed types.

Given that, and the name Big in 64-bit arrays, will it ever be unintuitive to know whether you need to insert an explicit cast? A concrete example would help. The long series of 8-bit Sets at the beginning of this post doesn't seem so relevant to the question; that could be handled by making a BigInt64Array out of the underlying ArratBuffer.

[seishun]

I'd really prefer not to have to present a change to the committee unless there is a very strong motivation for the change, given the back and forth so far. It seems like you can get by without this change with an explicit cast.

Consider the following scenario:

1. A developer wants to set an index in a Uint8(16,32)Array to a BigInt value modulo 2^8(16,32).
2. Simple assignment throws TypeError, so they add an explicit cast to Number.
3. The resulting code seems to work correctly. They don't test values greater than 9007199254740991.
4. At some point the code runs with values greater than 9007199254740991, resulting in incorrect behavior. Many hours are spent tracking down the bug.

will it ever be unintuitive to know whether you need to insert an explicit cast?

That's the problem, an explicit cast is an "obvious" but incorrect solution. You have to perform the modulo operation yourself before casting, which is less than intuitive.

that could be handled by making a BigInt64Array out of the underlying ArratBuffer.

That's not an option for Node.js, or anyone who needs to write in non-platform byte order.

[littledan]

Consider the following scenario:

This scenario helps a lot. Thanks for explaining. Another way to look at this issue, rather than it being about TypedArrays, would be about whether we should throw exceptions when casting from BigInt to Number and losing information.

The cast from BigInt to Number will silently round if the BigInt is too big. Maybe this is the problematic decision. By contrast, the cast from Number to BigInt will throw an exception of the Number has something which cannot be accurately represented.

@jakobkummerow previously pointed this out. We decided on the rounding behavior for BigInt-to-Number in #14 based on @waldemarhorwat's guidance, and didn't really revisit it when we decided to throw some exceptions in #15 . Was this an oversight?

[seishun]

Another way to look at this issue, rather than it being about TypedArrays, would be about whether we should throw exceptions when casting from BigInt to Number and losing information.

I agree that casting from BigInt to Number is worth discussing, but it's a separate issue. If casting BigInt to Number throws when the BigInt is too big, then the step 4 in my scenario will be an error. That's better than incorrect behavior, but less than ideal.

If setting an index in a Uint8(16,32)Array to a BigInt value performs a modulo operation on the BigInt value, then the developers won't have to wonder how to correctly convert BigInt to Number without rounding or errors.

[littledan]

Developers who do things that are mixed between BigInts and Numbers will have to use explicit cast operations. This is core to the design of BigInt. If you're using TypedArrays, you're doing an even more advanced thing than basic arithmetic, so I expect the need for explicit casts to be even less of a problem. The Number constructor applied to a string does rounding as well.

[sjrd]

whether we should throw exceptions when casting from BigInt to Number and losing information.

If we do that, then it will complicate the implementation of a 64-bit integer -> double conversion as found in many source languages. For example, in Java (double) longVal is supposed to lose precision, and not throw. If the equivalent Number(bigIntVal) throws, it will be harder to implement it.

It's OK if the default Number(bigIntVal) throws, but then I would ask that there be another function BigInt.toNumberUnsafe(bigIntVal) or something similar, that would silently drop precision, as the current implementation.

[littledan]

OK, at this point, I'm strongly leaning to maintaining the current specification as is.

[seishun]

Developers who do things that are mixed between BigInts and Numbers will have to use explicit cast operations.

Writing a BigInt to a UintXArray or IntXArray is a "thing that is mixed between BigInts and Numbers" only because the current spec requires the value you write to be a Number. My proposal makes it a thing that is not mixed between BigInts and Numbers.

Also, the explicit cast is not the problem - having to do the modulo operation yourself to avoid rounding errors is the problem.

The Number constructor applied to a string does rounding as well.

I'm not proposing to change the behavior of the Number constructor, so I don't see how this is relevant.

It seems you misunderstand my proposal. Perhaps I should draft up a diff for the spec to make things clearer?