Log message:
(textproc/R-data.table) Updated 1.16.4 to 1.18.4, Fix build against 4.6.0

## data.table news and updates

**If you are viewing this file on CRAN, please check [latest news on
  GitHub](https://github.com/Rdatatable/data.table/blob/master/NEWS.md)
  where the formatting is also better.**

## data.table
   [v1.18.4](https://github.com/Rdatatable/data.table/milestone/45?closed=1)
   4 May 2026

### BUG FIXES

1. Non-equi joins combining an equality condition with two inequality
conditions on the same column (e.g., `on = .(id == id, val >= lo, val
<= hi)`) no longer error,
[#7641](https://github.com/Rdatatable/data.table/issues/7641). The
internal `chmatchdup` remapping of duplicate `rightcols` was
overwriting the original column indices, causing downstream code to
reference non-existent columns. Thanks @tarun-t for the report and
fix, and @aitap for the diagnosis.

2. By-reference sub-assignments of strings to factor columns now
_actually_ match the levels in UTF-8 when required and now don't
result in invalid factors being created,
[#7648](https://github.com/Rdatatable/data.table/issues/7648),
amending a previous incomplete fix to
[#6886](https://github.com/Rdatatable/data.table/issues/6886) in
v1.17.2. Thanks @BASS-JN for the report and @aitap for the fix.

3. `fread()` no longer misreads dates with negative years,
[#7704](https://github.com/Rdatatable/data.table/issues/7704). Thanks
to @kevinushey for the report and @aitap for the fix.

4. `frollapply()` no longer produces output longer than the input when
the window length is also longer than the input
[#7646](https://github.com/Rdatatable/data.table/issues/7646). Thanks
to @hadley-johnson for reporting and @jangorecki for the fix.

-------------------------------------------------------------------------------------
## data.table [v1.18.2.1]
         (https://github.com/Rdatatable/data.table/milestone/44?closed=1) 22 Jan 2026

### BUG FIXES

1. When fixing duplicate factor levels, `setattr()` no longer crashes
upon encountering missing factor values,
[#7595](https://github.com/Rdatatable/data.table/issues/7595). Thanks
to @sindribaldur for the report and @aitap for the fix.

2. `foverlaps()` no longer crashes due to out-of-bounds access to list
and integer vectors when `y` has no rows or the non-range part of the
join fails,
[#7597](https://github.com/Rdatatable/data.table/issues/7597). Thanks
to @nextpagesoft for the report and @aitap for the fix.

3. The dynamic library now exports only `R_init_data_table`,
preventing symbol name conflicts like `hash_create` with PostgreSQL,
[#7605](https://github.com/Rdatatable/data.table/issues/7605). Thanks
to @ced75 for the report and @aitap for the fix

### Notes

1. Removed use of non-API `ATTRIB`, `SET_ATTRIB`, and `findVar`
[#6180](https://github.com/Rdatatable/data.table/issues/6180). Thanks
@aitap for the continued assiduous work here, and @MichaelChirico for
the easy fix to replace `findVar` with `R_getVar`.

2. Fixed compilation failure like "error: unknown type name
'siginfo_t'" in v1.18.0 in some strict environments, e.g., FreeBSD,
where the header file declaring the POSIX function `waitid` does not
transitively include the header file defining the `siginfo_t` type,
[#7516](https://github.com/rdatatable/data.table/issues/7516). Thanks
to @jszhao for the report and @aitap for the fix.

3. `sum(<int64 column>)` by group is correct with missing entries and
GForce activated
([#7571](https://github.com/Rdatatable/data.table/issues/7571)). Thanks
to @rweberc for the report and @manmita for the fix. The issue was
caused by a faulty early `break` that spilled between groups, and
resulted in silently incorrect results!

4. `set()` now automatically pre-allocates new column slots if needed,
similar to what `:=` already does,
[#1831](https://github.com/Rdatatable/data.table/issues/1831)
[#4100](https://github.com/Rdatatable/data.table/issues/4100). Thanks
to @zachokeeffe and @tyner for the report and @ben-schwen for the fix.

-------------------------------------------------------------------------------------
## data.table [v1.18.0]
    (https://github.com/Rdatatable/data.table/milestone/37?closed=1)  23 \ 
December 2025

### BREAKING CHANGE

1. `dcast()` now errors when `fun.aggregate` returns length != 1
(consistent with documentation), regardless of `fill`,
[#6629](https://github.com/Rdatatable/data.table/issues/6629). Previously,
when `fill` was not `NULL`, `dcast` warned and returned an undefined
result. This change has been planned since 1.16.0 (25 Aug 2024).

2. `melt()` returns an integer column for `variable` when
`measure.vars` is a list of length=1, consistent with the documented
behavior,
[#5209](https://github.com/Rdatatable/data.table/issues/5209). Thanks
to @tdhock for reporting. Any users who were relying on this behavior
can change `measure.vars=list("col_name")` (output `variable` was
column name, now is column index/integer) to `measure.vars="col_name"`
(`variable` still is column name). This change has been planned since
1.16.0 (25 Aug 2024).

3. Rolling functions `frollmean` and `frollsum` distinguish
`Inf`/`-Inf` from `NA` to match the same rules as base R when
`algo="fast"` (previously they were considered the same). If your
input into those functions has `Inf` or `-Inf` then you will be
affected by this change. As a result, the argument that controls the
handling of `NA`s has been renamed from `hasNA` to `has.nf` (_has
non-finite_). `hasNA` continues to work with a warning, for now.

    ```r
    ## before
    frollsum(c(1,2,3,Inf,5,6), 2)
    #[1] NA  3  5 NA NA 11

    ## now
    frollsum(c(1,2,3,Inf,5,6), 2)
    #[1]  NA   3   5 Inf Inf  11
    ```

4. `frollapply` result is not coerced to numeric anymore. Users' code
could possibly break if it depends on forced coercion of input/output
to numeric type.

    ```r
    ## before
    frollapply(c(F,T,F,F,F,T), 2, any)
    #[1] NA  1  1  0  0  1

    ## now
    frollapply(c(F,T,F,F,F,T), 2, any)
    #[1]    NA  TRUE  TRUE FALSE FALSE  TRUE
    ```

    Additionally argument names in `frollapply` has been renamed from
    `x` to `X` and `n` to `N` to avoid conflicts with common argument
    names that may be passed to `...`, aligning to base R API of
    `lapply`. `x` and `n` continue to work with a warning, for now.

5. Negative and missing values of `n` argument of adaptive rolling
functions trigger an error.

### NOTICE OF INTENDED FUTURE POTENTIAL BREAKING CHANGES

1. `data.table(x=1, <expr>)`, where `<expr>` is an expression
resulting in a 1-column matrix without column names, will eventually
have names `x` and `V2`, not `x` and `V1`, consistent with
`data.table(x=1, <expr>)` where `<expr>` results in an atomic vector,
for example `data.table(x=1, cbind(1))` and `data.table(x=1, 1)` will
both have columns named `x` and `V2`. In this release, the matrix case
continues to be named `V1`, but the new behavior can be activated by
setting `options(datatable.old.matrix.autoname)` to `FALSE`. See point
5 under Bug Fixes for more context; this change will provide more
internal consistency as well as more consistency with `data.frame()`.

2. The behavior of `week()` will be changed in a future release to
calculate weeks sequentially (days 1-7 as week 1), which is a
potential breaking change. For now, the current "legacy" behavior,
where week numbers advance every 7th day of the year (e.g., day 7
starts week 2), remains the default, and a deprecation warning will be
issued when the old and new behaviors differ. Users can control this
behavior with the temporary option `options(datatable.week = "...")`:
* `"sequential"`: Opt-in to the new, sequential behavior (no warning).
* `"legacy"`: Continue using the legacy behavior but suppress the
deprecation warning.  See
[#2611](https://github.com/Rdatatable/data.table/issues/2611) for
details. Thanks @MichaelChirico for the report and @venom1204 for the
implementation.

### NEW FEATURES

1. New `sort_by()` method for data.tables,
[#6662](https://github.com/Rdatatable/data.table/issues/6662). It uses
`forder()` to improve upon the data.frame method and also matches
`DT[order(...)]` behavior with respect to locale. Thanks @rikivillalba
for the suggestion and PR.

    ```r
    DT = data.table(a=c(1L, 2L, 1L), b=c(3L, 1L, 2L))
    sort_by(DT, ~a + b)
    #    a b
    # 1: 1 2
    # 2: 1 3
    # 3: 2 1
    ```

2. `melt()` now supports using `patterns()` with `id.vars`,
[#6867](https://github.com/Rdatatable/data.table/issues/6867). Thanks
to Toby Dylan Hocking for the suggestion and PR.

3. `print.data.table()` now shows column classes at the bottom of
large tables when `class=TRUE` and `col.names="auto"` (default) for
tables with more than 20 rows,
[#6902](https://github.com/Rdatatable/data.table/issues/6902). This
follows the same behavior as column names at the bottom, making it
easier to see column types for large tables without scrolling back to
the top. Thanks to @TimTaylor for the suggestion and @Mukulyadav2004
for the PR.

4. `as.Date()` method for `IDate` no longer coerces to `double`
[#6922](https://github.com/Rdatatable/data.table/issues/6922). Thanks
@MichaelChirico for the report and PR. The only effect should be on
overly-strict tests that assert `Date` objects have `double` storage,
which is not in general true, especially from R 4.5.0.

5. `as.data.table()` is slightly more efficient at converting arrays
to data.tables,
[#7019](https://github.com/Rdatatable/data.table/pull/7019). Thanks
@eliocamp.

6. `between()` gains the argument `ignore_tzone=FALSE`. Normally, a
difference in time zone between `lower` and `upper` will produce an
error, and a difference in time zone between `x` and either of the
others will produce a message. Setting `ignore_tzone=TRUE` bypasses
the checks, allowing both comparisons to proceed without error or
message about time zones.

7. New helper function `fctr` as an extended version of `factor()`,
[#4837](https://github.com/Rdatatable/data.table/issues/4837). Most
notably, it supports (1) retaining input level ordering by default,
i.e. `levels=unique(x)` as opposed to `levels = sort(unique(x))`; (2)
`rev=` to reverse the levels; and (3) `sort=` to allow more feature
parity with `factor()`. The choice of default is motivated by
convenience in the common case when order of elements needs be
preserved, for example when using `dcast` or adding a legend to a
plot. This also matches the default sort ordering of groups in `by=`.

    ```r
    d = data.table(id1=rep(1:2, each=3L), id2=letters[c(4:3,5L,3:5)], v1=1:6)
    dcast(d, id1 ~ factor(id2))
    #      id1     c     d     e
    # 1:     1     2     1     3
    # 2:     2     4     5     6
    dcast(d, id1 ~ fctr(id2))
    #      id1     d     c     e
    # 1:     1     1     2     3
    # 2:     2     5     4     6
    dcast(d, id1 ~ fctr(id2, sort=TRUE)) # same as factor()
    #      id1     c     d     e
    # 1:     1     2     1     3
    # 2:     2     4     5     6
    dcast(d, id1 ~ fctr(id2, rev=TRUE))
    #      id1     e     c     d
    # 1:     1     3     2     1
    # 2:     2     6     4     5
    ```

8. `groupingsets()` gets a new argument `enclos` for use together with
the `jj` argument in functions wrapping `groupingsets()`, including
the existing wrappers `rollup()` and `cube()`,
[#5560](https://github.com/Rdatatable/data.table/issues/5560). When
forwarding a `j`-expression as `groupingsets(jj = substitute(j))`,
make sure to pass `enclos = parent.frame()` as well, so that the
`j`-expression will be evaluated in the right context. This makes it
possible for `j` to refer to variables outside the
`data.table`. Thanks @sindribaldur for the report and @aitap for the
fix.

9. `isoweek()` is much faster (e.g. 20x) by re-using an implementation
from {base},
[#5111](https://github.com/Rdatatable/data.table/issues/5111). Thanks
@MichaelChirico for the report and PR.

10. `data.table()` and `as.data.table()` with `keep.rownames=TRUE` now
extract row names from named vectors, matching `data.frame()`
behavior. Names from the first named vector in the input are used to
create the row names column (default name `"rn"` or custom name via
`keep.rownames="column_name"`),
[#1916](https://github.com/Rdatatable/data.table/issues/1916). Thanks
to @richierocks for the feature request and @Mukulyadav2004 for the
implementation.

11. New `frev(x)` as a faster analogue to `base::rev()` for atomic
vectors/lists,
[#5885](https://github.com/Rdatatable/data.table/issues/5885). Twice
as fast as `base::rev()` on large inputs, and faster with more
threads. Thanks to Benjamin Schwendinger for suggesting and
implementing.

12. New `cbindlist()` and `setcbindlist()` for concatenating a `list`
of data.tables column-wise, evocative of the analogous `do.call(rbind,
l)` <-> `rbindlist(l)`,
[#2576](https://github.com/Rdatatable/data.table/issues/2576). `setcbindlist()`
does so without making any copies. Thanks @MichaelChirico for the FR,
@jangorecki for the PR, and @MichaelChirico for extensive reviews and
fine-tuning.

    ```r
    l = list(
      data.table(id = 1:3, a = letters[1:3]),
      data.table(b = 4:6, c = 7:9)
    )
    cbindlist(l)
    #    id a b c
    # 1:  1 a 4 7
    # 2:  2 b 5 8
    # 3:  3 c 6 9
    ```

13. New `mergelist()` and `setmergelist()` similarly work _a la_
`Reduce()` to recursively merge a `list` of data.tables,
[#599](https://github.com/Rdatatable/data.table/issues/599). Different
join modes (_left_, _inner_, _full_, _right_, _semi_, _anti_, and
_cross_) are supported through the `how` argument; duplicate handling
goes through the `mult` argument. `setmergelist()` carefully avoids
copies where one is not needed, e.g. in a 1:1 left join. Thanks
Patrick Nicholson for the FR (in 2013!), @jangorecki for the PR, and
@MichaelChirico for extensive reviews and fine-tuning.

    ```r
    l = list(
      data.table(id = c(1L, 2L, 3L), x = c("a", "b", \ 
"c")),
      data.table(id = c(1L, 2L, 4L), y = c("d", "e", \ 
"f")),
      data.table(id = c(1L, 3L, 4L), z = c("g", "h", \ 
"i"))
    )

    # Recursive inner join
    mergelist(l, on = "id", how = "inner")
    #    id x y z
    # 1:  1 a d g

    # Recursive left join (the default 'how')
    mergelist(l, on = "id", how = "left")
    #    id x    y    z
    # 1:  1 a    d    g
    # 2:  2 b    e <NA>
    # 3:  3 c <NA>    h
    ```

14. `fcoalesce()` and `setcoalesce()` gain `nan` argument to control
whether `NaN` values should be treated as missing (`nan=NA`, the
default) or non-missing (`nan=NaN`),
[#4567](https://github.com/Rdatatable/data.table/issues/4567). This
provides full compatibility with `nafill()` behavior. Thanks to
@ethanbsmith for the feature request and @Mukulyadav2004 for the
implementation.

15. New function `isoyear()` has been implemented as a complement to
`isoweek()`, returning the ISO 8601 year corresponding to a given
date,
[#7154](https://github.com/Rdatatable/data.table/issues/7154). Thanks
to @ben-schwen and @MichaelChirico for the suggestion and @venom1204
for the implementation.

16. Multiple improvements have been added to rolling
functions. Request came from @gpierard who needed left aligned,
adaptive, rolling max,
[#5438](https://github.com/Rdatatable/data.table/issues/5438). There
was no `frollmax` function yet. Adaptive rolling functions did not
have support for `align="left"`. `frollapply` did not support
`adaptive=TRUE`. Available alternatives were base R `mapply` or
self-join using `max` and grouping `by=.EACHI`. As a follow up of his
request, the following features have been added:

    - new function `frollmax`, applies `max` over a rolling window.
    - support for `align="left"` for adaptive rolling function.
    - support for `adaptive=TRUE` in `frollapply`.

    - `partial` argument to trim window width to available
      observations rather than returning `NA` whenever window is not
      complete.

    - `give.names` argument that can be used to automatically give the
      names based on the names of `x` and `n`.

    - `frollmean` and `frollsum` no longer treat `Inf` and `-Inf` as
      `NA`s as it used to be for `algo="fast"` (breaking change).

    - `hasNA` argument has been renamed to `has.nf` to convey that it
      is not only related to `NA/NaN` but other non-finite values
      (`Inf/-Inf`) as well.

    Thanks to @jangorecki for implementation and @MichaelChirico and
    others for work on splitting into smaller PRs and reviews.  For a
    comprehensive description about all available features see
    `?froll` manual.

    Adaptive `frollmax` has observed to be around 80 times faster than
    second fastest solution (data.table self-join using `max` and
    grouping `by=.EACHI`). Note that important factor in performance
    is width of the rolling window. Code for the benchmark below has
    been taken from [this SO
    answer](https://stackoverflow.com/a/73408459/2490497).  ```r

    set.seed(108)
    setDTthreads(16)
    x = data.table(
      value = cumsum(rnorm(1e6, 0.1)),
      end_window = 1:1e6 + sample(50:500, 1e6, TRUE),
      row = 1:1e6
    )[, "end_window" := pmin(end_window, .N)
      ][, "len_window" := end_window-row+1L]
    baser = function(x) x[, mapply(function(from, to) max(value[from:to]), row, \ 
end_window)]
    sj = function(x) x[x, max(value), on=.(row >= row, row <= end_window), \ 
by=.EACHI]$V1
    frmax = function(x) x[, frollmax(value, len_window, adaptive=TRUE, \ 
align="left", has.nf=FALSE)]
    frapply = function(x) x[, frollapply(value, len_window, max, adaptive=TRUE, \ 
align="left")]
    microbenchmark::microbenchmark(
      baser(x), sj(x), frmax(x), frapply(x),
      times=10, check="identical"
    )
    #Unit: milliseconds
    #       expr        min         lq       mean     median         uq        \ 
max neval
    #   baser(x) 3094.88357 3097.84966 3186.74832 3163.58050 3251.66753 \ 
3370.33785    10
    #      sj(x) 2221.55456 2255.12083 2306.61382 2303.47883 2346.70293 \ 
2412.62975    10
    #   frmax(x)   17.45124   24.16809   28.10062   28.58153   32.79802   \ 
34.83941    10
    # frapply(x)  272.07830  316.47060  366.94771  396.23566  416.06699  \ 
421.38701    10
    ```

    As of now, adaptive rolling max has no _on-line_ implementation
    (`algo="fast"`), it uses a naive approach
    (`algo="exact"`). Therefore further speed up is still possible if
    `algo="fast"` gets implemented.

17. Function `frollapply` has been completely rewritten. Thanks to
@jangorecki for implementation. Be sure to read `frollapply` manual
before using the function. There are following changes:

    - all basic types are now supported on input/output, not only
      double. Users' code could possibly break if it depends on forced
      coercion of input/output to double type.

    - new argument `by.column` allowing to pass a multi-column subset
    of a data.table into a rolling function, closes
    [#4887](https://github.com/Rdatatable/data.table/issues/4887).

    ```r
    x = data.table(v1=rnorm(120), v2=rnorm(120))
    f = function(x) coef(lm(v2 ~ v1, data=x))
    frollapply(x, 4, f, by.column=FALSE)
    #     (Intercept)         v1
    #           <num>      <num>
    #  1:          NA         NA
    #  2:          NA         NA
    #  3:          NA         NA
    #  4: -0.04648236 -0.6349687
    #  5:  0.09208733 -0.4964023
    #---
    #116: -0.21169439  0.7421358
    #117: -0.19729119  0.4926939
    #118: -0.04217896  0.0452713
    #119:  0.22472549 -0.5245874
    #120:  0.54540359 -0.1638333
    ```

    - uses multiple CPU threads (on a decent OS); evaluation of UDF is
      inherently slow so this can be a great help.

    ```r
    x = rnorm(1e5)
    n = 500
    setDTthreads(1)
    system.time(
      th1 <- frollapply(x, n, median, simplify=unlist)
    )
    #   user  system elapsed
    #  3.078   0.005   3.084
    setDTthreads(4)
    system.time(
      th4 <- frollapply(x, n, median, simplify=unlist)
    )
    #   user  system elapsed
    #  2.453   0.135   0.897
    all.equal(th1, th4)
    #[1] TRUE
    ```

18. New helper `frolladapt` to facilitate applying rolling functions
over windows of fixed calendar-time width in irregularly-spaced data
sets, thereby bypassing the need to "augment" such data with
placeholder rows,
[#3241](https://github.com/Rdatatable/data.table/issues/3241). Thanks
to @jangorecki for implementation.

    ```r
    idx = as.Date("2025-09-05") + c(0,4,7,8,9,10,12,13,17)
    dt = data.table(index=idx, value=seq_along(idx))
    dt
    #        index value
    #       <Date> <int>
    #1: 2025-09-05     1
    #2: 2025-09-09     2
    #3: 2025-09-12     3
    #4: 2025-09-13     4
    #5: 2025-09-14     5
    #6: 2025-09-15     6
    #7: 2025-09-17     7
    #8: 2025-09-18     8
    #9: 2025-09-22     9
    dt[, c("rollmean3","rollmean3days") := list(
      frollmean(value, 3),
      frollmean(value, frolladapt(index, 3), adaptive=TRUE)
      )]
    dt
    #        index value rollmean3 rollmean3days
    #       <Date> <int>     <num>         <num>
    #1: 2025-09-05     1        NA            NA
    #2: 2025-09-09     2        NA           2.0
    #3: 2025-09-12     3         2           3.0
    #4: 2025-09-13     4         3           3.5
    #5: 2025-09-14     5         4           4.0
    #6: 2025-09-15     6         5           5.0
    #7: 2025-09-17     7         6           6.5
    #8: 2025-09-18     8         7           7.5
    #9: 2025-09-22     9         8           9.0
    ```

19. Other new rolling functions: `frollmin`, `frollprod`,
`frollmedian`, `frollvar` and `frollsd`, have been implemented,
resolving long standing issue
[#2778](https://github.com/Rdatatable/data.table/issues/2778). Thanks
to @jangorecki for implementation. Implementation of rolling median is
based on a novel algorithm "sort-median" described by
[@suomela](https://github.com/suomela) in his 2014 paper [Median
Filtering is Equivalent to
Sorting](https://arxiv.org/abs/1406.1717). "sort-median" scales very
well, not only for size of input vector but also for size of rolling
window.

    ```r
    rollmedian = function(x, n) {
      ans = rep(NA_real_, nx<-length(x))
      if (n<=nx) for (i in n:nx) ans[i] = median(x[(i-n+1L):(i)])
      ans
    }
    library(data.table)
    setDTthreads(8)
    set.seed(108)
    x = rnorm(1e5)

    n = 100
    system.time(rollmedian(x, n))
    #   user  system elapsed
    #  2.049   0.001   2.051
    system.time(frollapply(x, n, median, simplify=unlist))
    #   user  system elapsed
    #  3.071   0.223   0.436
    system.time(frollmedian(x, n))
    #   user  system elapsed
    #  0.013   0.000   0.004

    n = 1000
    system.time(rollmedian(x, n))
    #   user  system elapsed
    #  3.496   0.009   3.507
    system.time(frollapply(x, n, median, simplify=unlist))
    #   user  system elapsed
    #  4.552   0.307   0.632
    system.time(frollmedian(x, n))
    #   user  system elapsed
    #  0.015   0.000   0.004

    n = 10000
    system.time(rollmedian(x, n))
    #   user  system elapsed
    # 16.350   0.025  16.382
    system.time(frollapply(x, n, median, simplify=unlist))
    #   user  system elapsed
    # 14.865   0.722   2.267
    system.time(frollmedian(x, n))
    #   user  system elapsed
    #  0.028   0.000   0.005
    ```

    20. `fread()` now supports the `comment.char` argument to skip
    trailing comments or comment-only lines, consistent with
    `read.table()`,
    [#856](https://github.com/Rdatatable/data.table/issues/856). The
    default remains `comment.char = ""` (no comment parsing) for
    backward compatibility and performance, in contrast to
    `read.table(comment.char = "#")`. Thanks to @arunsrinivasan and
    many others for the suggestion and @ben-schwen for the
    implementation.

### BUG FIXES

1. `fread()` no longer warns on certain systems on R 4.5.0+ where the
file owner can't be resolved,
[#6918](https://github.com/Rdatatable/data.table/issues/6918). Thanks
@ProfFancyPants for the report and PR.

2. Joins to extended data.frames, e.g. `x[i, col := x.col1 + i.col2]`
where `i` is a `tbl`, can use the `x.` and `i.` prefix forms,
[#6998](https://github.com/Rdatatable/data.table/issues/6998). Thanks
@MichaelChirico for the bug and PR.

3. Out of sample type bumps now respect `integer64=` selection,
[#7032](https://github.com/Rdatatable/data.table/pull/7032).

4. In rare cases, `data.table` failed to expand ALTREP columns when
assigning a full column by reference. This could result in the target
column getting modified unintentionally if the next call to the
data.table was a modification by reference of the source
column. E.g. in `DT[, b := as.character(a)]` the string conversion
gets deferred and subsequent modification of column `a` would also
modify column `b`,
[#5400](https://github.com/Rdatatable/data.table/issues/5400). Thanks
to @aquasync for the report and Václav Tlapák for the PR.

5. `data.table()` function is now more aligned with `data.frame()`
with respect to the names of the output when one of its inputs is a
single-column matrix object,
[#4124](https://github.com/Rdatatable/data.table/issues/4124),
[#3193](https://github.com/Rdatatable/data.table/issues/3193), and
[#5367](https://github.com/Rdatatable/data.table/issues/5367). Thanks
@PavoDive for the report, @jangorecki for the PR, and @MichaelChirico
for a follow-up for back-compatibility.

6. Including an `ITime` object as a named input to `data.frame()`
respects the provided name, i.e. `data.frame(a = as.ITime(...))` will
have column `a`,
[#4673](https://github.com/Rdatatable/data.table/issues/4673). Thanks
@shrektan for the report and @MichaelChirico for the fix.

7. `fread()` now handles the `na.strings` argument for quoted text
columns, making it possible to specify `na.strings = '""'` and read
empty quoted strings as `NA`s,

[#6974](https://github.com/Rdatatable/data.table/issues/6974). Thanks
to @AngelFelizR for the report and @aitap for the PR.

8. A data.table with a column of class `vctrs_list_of` (from package
{vctrs}) prints as expected,
[#5948](https://github.com/Rdatatable/data.table/issues/5948). Before,
they could be printed messily, e.g. printing every entry in a nested
data.frame. Thanks @jesse-smith for the report, @DavisVaughan and
@r2evans for contributing, and @MichaelChirico for the PR.

9. Fixed incorrect sorting of merges where the first column of a key
is a factor with non-`sort()`-ed levels (e.g. `factor(1:2, 2:1)` and
it is joined to a character column,
[#5361](https://github.com/Rdatatable/data.table/issues/5361). Thanks
to @gbrunick for the report, Benjamin Schwendinger for the fix, and
@MichaelChirico for a follow-up fix caught by revdep testing.

10. Spurious warnings from internal code in `cube()`, `rollup()`, and
`groupingsets()` are no longer surfaced to the caller,
[#6964](https://github.com/Rdatatable/data.table/issues/6964). Thanks
@ferenci-tamas for the report and @venom1204 for the fix.

11. `droplevels()` works on 0-row data.tables,
[#7043](https://github.com/Rdatatable/data.table/issues/7043). The
result will have factor columns `factor(character())`, consistent with
the data.frame method. Thanks @advieser for the report and
@MichaelChirico for the fix.

12. `print(..., col.names = 'none')` now correctly adapts column
widths to the data content, ignoring the original column names and
producing a more compact output,
[#6882](https://github.com/Rdatatable/data.table/issues/6882). Thanks
to @brooksambrose for the report and @venom1204 for the PR.

13. Reference to `.SD` in `...` arguments to `lapply()`,
e.g. ``lapply(list_of_tables, `[`, j=.SD[1L])`` is evaluated
correctly,
[#2982](https://github.com/Rdatatable/data.table/issues/2982). Thanks
@franknarf1 for the report and @MichaelChirico for the fix.

14. Filling columns of class Date with POSIXct (and vice versa) using
`shift()` now yields a clear, informative error message specifying the
class mismatch,
[#5218](https://github.com/Rdatatable/data.table/issues/5218). Thanks
@ashbaldry for the report and @ben-schwen for the fix.

15. `split.data.table()` output list elements retain the S3 class of
the generating data.table, e.g. in `l=split(x, ...)` if `x` has class
`my_class`, so will `l[[1]]` and so on,
[#7105](https://github.com/Rdatatable/data.table/issues/7105). Thanks
@m-muecke for the bug report and @MichaelChirico for the fix.

16. `between()` is now more robust with `integer64`
arguments. Combining small integer `x` with certain large `integer64`
bounds no longer misinterprets the bounds as `double`; if a `double`
bound cannot be losslessly converted into `integer64` for comparison
with `integer64` `x`, an error is signalled instead of returning a
wrong answer with a warning;
[#7164](https://github.com/Rdatatable/data.table/issues/7164). Thanks
@aitap for the bug report and the fix.

17. `t1 - t2`, where one is an `IDate` and the other is a `Date`, are
now consistent with the case where both are `IDate` or both are
`Date`,
[#4749](https://github.com/Rdatatable/data.table/issues/4749). Thanks
@George9000 for the report and @MichaelChirico for the fix.

18. `fwrite` now allows `dec` to be the same as `sep` for edge cases
where only one will be written, e.g. 0-row or 1-column
tables. [#7227](https://github.com/Rdatatable/data.table/issues/7227). Thanks
@MichaelChirico for the report and @venom1204 for the fix.

19. Ellipsis elements like `..1` are correctly excluded when searching
for variables in "up-a-level" syntax inside `[`,
[#5460](https://github.com/Rdatatable/data.table/issues/5460). Thanks
@ggrothendieck for the report and @MichaelChirico for the fix.

20. `forderv` could segfault on keys with long runs of identical bytes
because the single-group branch tail-recursed radix-by-radix until the
C stack ran out. This affected both integer/numeric sorting with many
duplicate columns
([#4300](https://github.com/Rdatatable/data.table/issues/4300)) and
character sorting with long common prefixes
([#7462](https://github.com/Rdatatable/data.table/issues/7462)). This
is a major problem since sorting is extensively used in
`data.table`. Thanks @quantitative-technologies and @DavisVaughan for
the reports, and @ben-schwen for the fix.

21. `[` now preserves existing key(s) when new columns are added
before them, instead of incorrectly setting a new column as key,
[#7364](https://github.com/Rdatatable/data.table/issues/7364). Thanks
@czeildi for the bug report and the fix.

22. `setDTthreads(percent=)` and `setDTthreads(threads=)` now respect
`OMP_NUM_THREADS` and `omp_get_max_threads()`, ensuring consistency
with `setDTthreads()` (no arguments) when OpenMP environment variables
are set,
[#7165](https://github.com/Rdatatable/data.table/issues/7165). Previously,
explicitly setting a thread count or percentage would ignore these
OpenMP limits, potentially exceeding the user's intended thread
cap. Thanks to @bastistician for the report and @ben-schwen for the
fix.

23. `fread()` auto-detects separators for single-column files
consisting solely of quoted values (e.g. `"this_that"\n"2025-01-01
00:00:01"`),
[#7366](https://github.com/Rdatatable/data.table/issues/7366). Thanks
@arunsrinivasan for the report and @ben-schwen for the fix.

24. Rolling functions now ensure there is no nested parallelism. It
could have happened for vectorized input and `adaptive=TRUE`,
[#7352](https://github.com/Rdatatable/data.table/issues/7352). Thanks
@jangorecki for the fix.

25. By-group operations on missing rows (e.g. `foo[c(i, NA), bar,
by=grp]`) now avoid leaving in data from the previous groups,
[#7442](https://github.com/Rdatatable/data.table/issues/7442). Thanks
@aitap for the report and the fix.

26. Grouping by a factor with many groups is now fast again, fixing a
timing regression introduced in
[#6890](https://github.com/Rdatatable/data.table/pull/6890) where
UTF-8 coercion and level remapping were performed unnecessarily,
[#7404](https://github.com/Rdatatable/data.table/issues/7404). Thanks
@ben-schwen for the report and fix.

27. `dogroups()` no longer reads beyond the resized end of
over-allocated data.table list columns,
[#7486](https://github.com/Rdatatable/data.table/issues/7486). While
this didn't crash in practice, it is now explicitly checked for in
recent R versions (r89198+). Thanks @TimTaylor and @aitap for the
report and @aitap for the fix.

28. `rbindlist()` now avoids the crash when working with many
non-UTF-8 column names,
[#7452](https://github.com/Rdatatable/data.table/issues/7452). Thanks
@aitap for the report and the fix.

### NOTES

1. The following in-progress deprecations have proceeded:

   + Argument `logicalAsInt` to `fwrite()` has been removed.
   + Argument `autostart` to `fread()` has been removed.
   + Argument `in.place` to `droplevels` has been removed.
   + It's now an error to set `datatable.nomatch`, which has been warning since \ 
1.15.0.

2. {data.table} now depends on R 3.4.0 (2017).

3. Changes to `fread()` output and errors:

   + When the size of the file exceeds the size of the address space,
   `fread()` now signals an informative error instead of trying to map
   its size modulo the address space.

   + On non-Windows systems, `fread()` now prints the reason why the
   file couldn't be opened, which could also be due to it being too
   large to map.

   + With `verbose=TRUE`, file sizes are now printed using correct
   binary SI prefixes (the sizes have always been reported as bytes
   denominated in powers of `2^10`, so e.g. `1024*1024` bytes was
   reported as `1 MB` where `1 MiB` or `1.05 MB` is correct).

4. The default `format_list_item()` method (and hence
`print.data.table()`) annotates truncated list items with their
length,
[#605](https://github.com/Rdatatable/data.table/issues/605). Thanks
Matt Dowle for the original report (2012!) and @MichaelChirico for the
fix.

5. A GitHub Actions workflow is now in place to warn the entire
maintainer team, as well as any contributor following the GitHub
repository, when the package is at risk of archival on CRAN
[#7008](https://github.com/Rdatatable/data.table/issues/7008). Thanks
@tdhock for the original report and @Bisaloo and @TysonStanley for the
fix.

6. Using a double vector in `set()`'s `i=` and/or `j=` no longer
throws a warning about preferring integer,
[#6594](https://github.com/Rdatatable/data.table/issues/6594). While
it may improve efficiency to use integer, there's no guarantee it's an
improvement and the difference is likely to be minimal. The coercion
will still be reported under `datatable.verbose=TRUE`. For
package/production use cases, static analyzers such as
`lintr::implicit_integer_linter()` can also report when numeric
literals should be rewritten as integer literals.

7. In rare situations a data.table object may lose its internal
attribute that holds a self-reference. New helper function
`.selfref.ok()` tests just that. It is only intended for technical use
cases. See manual for examples.

8. Retain important information in the error message about the source
of the error when `i=` fails, e.g. pointing to `charToDate()` failing
in `DT[date_col == "20250101"]`,
[#7444](https://github.com/Rdatatable/data.table/issues/7444). Thanks
@jan-swissre for the report and @MichaelChirico for the fix.

9. Internal use of declared non-API R functions `SETLENGTH`,
`TRUELENGTH`, `SET_TRUELENGTH`, and `SET_GROWABLE_BIT` has been
eliminated. Most usages have been migrated to R's experimental
resizable vectors API (thanks to @ltierney, introduced in R 4.6.0,
backported for older R versions),
[#7451](https://github.com/Rdatatable/data.table/pull/7451). Uses of
`TRUELENGTH` for marking seen items during grouping and binding
operations (aka free hash table trick) have been replaced with proper
hash tables,
[#6694](https://github.com/Rdatatable/data.table/pull/6694). The new
hash table implementation uses linear probing with power of 2 tables
and automatic resizing. Additionally, `chmatch()` now hashes the
needle (`x`) instead of the haystack (`table`) when `length(table) >>
length(x)`, significantly improving performance for lookups into large
tables. We've benchmarked the refactored code and find the performance
satisfactory, but please do report any edge case performance
regressions we may have missed. Thanks to @aitap, @ben-schwen,
@jangorecki and @HughParsonage for implementation and reviews.

-------------------------------------------------------------------------------------
## data.table [v1.17.8](https://github.com/Rdatatable/data.table/milestone/41) \ 
(6 July 2025)

1. Internal functions used to signal errors are now marked as
non-returning, silencing a compiler warning about potentially
unchecked allocation failure. Thanks to Prof. Brian D. Ripley for the
report and @aitap for the fix,
[#7070](https://github.com/Rdatatable/data.table/pull/7070).

-------------------------------------------------------------------------------------
## data.table [v1.17.6](https://github.com/Rdatatable/data.table/milestone/40) \ 
(15 June 2025)

1. On a heavily loaded machine, a `forder` thread could try to perform
a zero-length copy from a null pointer, which was de-facto harmless
but is against the C standard and was caught by additional CRAN
checks,
[#7051](https://github.com/Rdatatable/data.table/issues/7051). Thanks
to @helske for the report and @aitap for the PR.

-------------------------------------------------------------------------------------
## data.table [v1.17.4](https://github.com/Rdatatable/data.table/milestone/39) \ 
(25 May 2025)

1. The C code now avoids passing invalid data pointers from 0-length
vectors to `memcpy()`, which previously caused undefined
behaviour. Thanks to Prof. Brian D. Ripley for the report and Michael
Chirico for the fix,
[#6911](https://github.com/Rdatatable/data.table/pull/6911).

-------------------------------------------------------------------------------------
## data.table [v1.17.2](https://github.com/Rdatatable/data.table/milestone/38) \ 
(7 May 2025)

### BUG FIXES

1. `fwrite(compress="gzip")` once again produces a gzip header when
the column names are missing or disabled,
[@6852](https://github.com/Rdatatable/data.table/issues/6852). Thanks
@maxscheiber for the report and @aitap for the fix.

2. `fread(keepLeadingZeros=TRUE)` now correctly parses dates with
components with leading zeros as dates instead of strings,
[#6851](https://github.com/Rdatatable/data.table/issues/6851). Thanks
@TurnaevEvgeny for the report and @ben-schwen for the fix.

3. `as.data.table()` on `x` avoids an infinite loop if the output of
the corresponding `as.data.frame()` method has the same class as the
input,
[#6874](https://github.com/Rdatatable/data.table/issues/6874). Concretely,
we had `class(x) = c('foo', 'data.frame')` and
`class(as.data.frame(x)) = c('foo', 'data.frame')`, so
`as.data.frame.foo` wound up getting called repeatedly. Thanks
@matschmitz for the report and @ben-schwen for the fix.

4. By-reference sub-assignments to factor columns now match the levels
in UTF-8, preventing their duplication when the same level exists in
different encodings,
[#6886](https://github.com/Rdatatable/data.table/issues/6886). Thanks
@iagogv3 for the report and @aitap for the fix.

5. `fwrite()` now avoids a crash when translating strings into a
different encoding,
[#6883](https://github.com/Rdatatable/data.table/issues/6883). Thanks
@filipemsc for the report and @aitap for the fix.

6. Custom binary operators from the `lubridate` package now work with
objects of class `IDate` as with a `Date` subclass,
[#6839](https://github.com/Rdatatable/data.table/issues/6839). Thanks
@emallickhossain for the report and @aitap for the fix.

7. `as.data.table()` now properly handles keys: specifying keys sets
them, omitting keys preserves existing ones, and setting `key=NULL`
clears them,
[#6859](https://github.com/Rdatatable/data.table/issues/6859). Thanks
@brookslogan for the report and @Mukulyadav2004 for the fix.

### NOTES

1. Continued work to remove non-API C functions,
[#6180](https://github.com/Rdatatable/data.table/issues/6180). Thanks
Ivan Krylov for the PRs and for writing a clear and concise guide
about the R API: https://aitap.codeberg.page/R-api/.

-------------------------------------------------------------------------------------
## data.table [v1.17.0](https://github.com/Rdatatable/data.table/milestone/34)  \ 
(20 Feb 2025)

### POTENTIALLY BREAKING CHANGES

1. In `DT[, variable := value]`, when value is class `POSIXlt`, we
automatically coerce it to class `POSIXct` instead,
[#1724](https://github.com/Rdatatable/data.table/issues/1724). Thanks
to @linzhp for the report, and Benjamin Schwendinger for the fix.

### NEW FEATURES

1. New function `rowwiseDT()` for creating a data.table object
"row-wise", often convenient for readability of small,
literally-defined tables. Thanks to @shrektan for the suggestion and
PR and @tdeenes for the idea of the `name=` syntax. Inspired by
`tibble::tribble()`.

```r
library(data.table)
rowwiseDT(
  a=,b=,c=,  d=,
  1, 2, "a", 2:3,
  3, 4, "b", list("e"),
  5, 6, "c", ~a+b
)
#>        a     b      c      d
#>    <num> <num> <char> <list>
#> 1:     1     2      a    2,3
#> 2:     3     4      b      e
#> 3:     5     6      c ~a + b
```

2. Limited support for subsetting or aggregating columns of type
`expression`,
[#5596](https://github.com/Rdatatable/data.table/issues/5596). Thanks
to @tsp for the report, and @ben-schwen for the fix.

3. `groupingsets.data.table()`, `cube.data.table()`, and
`rollup.data.table()` gain a `label` argument, which allows the user
to specify a label for each grouping variable, to be included in the
grouping variable column in the output in rows where the variable has
been aggregated,
[#5351](https://github.com/Rdatatable/data.table/issues/5351). Thanks
to @markseeto for the request, @jangorecki and @markseeto for
specifying the desired behaviour, and @markseeto for implementing.

    ```r
    DT = data.table(V1 = rep(c("a1", "a2"), each = 5),
                    V2 = rep(rep(c("b1", "b2"), c(3, 2)), 2),
                    V3 = rep(c("c1", "c2"), c(3, 7)),
                    V4 = rep(1:2, c(6, 4)),
                    V5 = rep(1:2, c(9, 1)),
                    V6 = rep(c(1.1, 1.2), c(2, 8)))

    # Call groupingsets() and specify a label for V1, a different
      label for the other character grouping
    # variables, a label for the integer grouping variables, and a
      label for the numeric grouping variable.
    groupingsets(DT, .N, by = c("V1", "V2", "V3", \ 
"V4", "V5", "V6"),
                 sets = list(c("V1", "V2", "V3"), \ 
c("V1", "V4"), c("V4", "V6"), \ 
"V2", "V5", character()),
                 label = list(V1 = "All values", character = \ 
"Total", integer = 999L, numeric = NaN))

    #             V1     V2     V3    V4    V5    V6     N
    #         <char> <char> <char> <int> <int> \ 
<num> <int>
    #  1:         a1     b1     c1   999   999   NaN     3
    #  2:         a1     b2     c2   999   999   NaN     2
    #  3:         a2     b1     c2   999   999   NaN     3
    #  4:         a2     b2     c2   999   999   NaN     2
    #  5:         a1  Total  Total     1   999   NaN     5
    #  6:         a2  Total  Total     1   999   NaN     1
    #  7:         a2  Total  Total     2   999   NaN     4
    #  8: All values  Total  Total     1   999   1.1     2
    #  9: All values  Total  Total     1   999   1.2     4
    # 10: All values  Total  Total     2   999   1.2     4
    # 11: All values     b1  Total   999   999   NaN     6
    # 12: All values     b2  Total   999   999   NaN     4
    # 13: All values  Total  Total   999     1   NaN     9
    # 14: All values  Total  Total   999     2   NaN     1
    # 15: All values  Total  Total   999   999   NaN    10
    ```

4. `patterns()` in `melt()` combines correctly with user-defined
`cols=`, which can be useful to specify a subset of columns to reshape
without having to use a regex, for example `patterns("2", \ 
cols=c("y1",
"y2"))` will only give `y2` even if there are other columns in the
input matching `2`,
[#6498](https://github.com/Rdatatable/data.table/issues/6498). Thanks
to @hongyuanjia for the report, and to @tdhock for the PR.

5. `setcolorder()` gains `skip_absent` to ignore unrecognized columns
(i.e. columns included in `neworder` but not present in the data),
[#6044,
#6068](https://github.com/Rdatatable/data.table/pull/6044). Default
behavior (`skip_absent=FALSE`) remains unchanged, i.e. unrecognized
columns result in an error. Thanks to @sluga for the suggestion and
@sluga & @Nj221102 for the PRs.

6. `fread()` gains `logicalYN` argument to read columns consisting
only of strings `Y`, `N` as `logical` (as opposed to character),
[#4563](https://github.com/Rdatatable/data.table/issues/4563). The
default is controlled by option `datatable.logicalYN`, itself
defaulting to `FALSE`, for back-compatibility -- some smaller tables
(especially sharded tables) might inadvertently read a "true" string
column as `logical` and cause bugs. This is particularly important for
tables with a column named `y` or `n` -- automatic header detection
under `logicalYN=TRUE` will see these values in the first row as being
"data" as opposed to column names. A parallel option was not included
for `fwrite()` at this time -- users looking for a compact
representation of logical columns can still use
`fwrite(logical01=TRUE)`. We also opted for now to check only `Y`, `N`
and not `Yes`/`No`/`YES`/`NO`.

7. `fwrite()` with `compress="gzip"` produces compatible gz files when
composed of multiple independent chunks owing to parallelization,
[#6356](https://github.com/Rdatatable/data.table/issues/6356).
Earlier `fwrite()` versions could have issues with HTTP upload using
`Content-Encoding: gzip` and `Transfer-Encoding: chunked`. Thanks to
@oliverfoster for report and @philippechataignon for the fix. Thanks
also @aitap for pre-release testing that found some possible memory
leaks in the initial fix.

8. `fwrite()` gains a new parameter `compressLevel` to control
compression level for gzip,
[#5506](https://github.com/Rdatatable/data.table/issues/5506). This
parameter balances compression speed and total compression, and
corresponds directly to the analogous command-line parameter,
e.g. `compressLevel=4` corresponds to passing `-4`; the default, `6`,
matches the command-line default, i.e. equivalent to passing
`-6`. Thanks @mgarbuzov for the request and @philippechataignon for
implementing.

### BUG FIXES

1. `fwrite()` respects `dec=','` for timestamp columns (`POSIXct` or
`nanotime`) with sub-second accuracy,
[#6446](https://github.com/Rdatatable/data.table/issues/6446). Thanks
@kav2k for pointing out the inconsistency and @MichaelChirico for the
PR.

2. The data.table-only attribute `$.internal.selfref` is no longer set
for
data.frames. [#5286](https://github.com/Rdatatable/data.table/issues/5286). Thanks
@OfekShilon for the report and fix.

3. Tagging/naming arguments of `c()` in `j=c()` should now more
closely follow base R conventions for concatenation of named lists
during grouping,
[#2311](https://github.com/Rdatatable/data.table/issues/2311). Naming
an `lapply(.SD, FUN)` call as an argument of `c()` in `j` will now
always cause that tag to get prepended (with a single dot separator)
to the resulting column names. Additionally, naming a `list()` call as
an argument of `c()` in `j` will now always cause that tag to get
prepended to any names specified within the list call. This bug only
affected queries with (1) `by=` grouping (2)
`getOption("datatable.optimize") >= 1L` and (3) `lapply(.SD, FUN)` in
`j`.

    While the names returned by `data.table` when `j=c()` will now
    mostly follow base R conventions for concatenating lists, note
    that names which are completely unspecified will still be named
    positionally, matching the typical behavior in `j` and
    `data.table()`. according to position in `j` (e.g. `V1`, `V2`).

    Thanks to @franknarf1 for reporting and @myoung3 for the PR.

    ```r
    # tag 'mean' prepended to lapply()-named columns
    names(mtcars[, c(mean=lapply(.SD,sum)), by="cyl", \ 
.SDcols=c("am", "carb")])
    # [1] "cyl" "mean.am" "mean.carb"

    # tag 'mean' is prepended to the first named sublist, 'sum' to the
    second names(mtcars[, c(mean=list(a=mean(hp), b=mean(wt)),
    sum=lapply(.SD, sum)), by="cyl", .SDcols=c("am", \ 
"carb")])

    # [1] "cyl" "mean.a" "mean.b" \ 
"sum.am" "sum.carb"

    # strict base naming would result in names c("", "b", \ 
"c") here
    names(mtcars[, c(list(mean(hp), b=mean(wt)), c=list(mean(cyl)))])
    # [1] "V1" "b" "c"
    ```

4. Queries like `DT[, min(x):max(x)]` now work as expected, i.e. the
same as `DT[, seq(min(x), max(x))]` or `with(DT, min(x):max(x))`,
[#2069](https://github.com/Rdatatable/data.table/issues/2069). Shorthand
like `DT[, a:b]` meaning "select from columns `a` through `b`" still
works. Thanks to @franknarf1 for reporting, @jangorecki for the fix,
and @MichaelChirico for follow-ups ensuring back-compatibility.

5. `fread()` performance improves when specifying `Date` among
`colClasses`,
[#6105](https://github.com/Rdatatable/data.table/issues/6105). One
implication of the change is that the column will be an `IDate` (which
also inherits from `Date`), which may affect code strongly relying on
the column class to be `Date` exactly; computations with `IDate` and
`Date` columns should otherwise be the same. If you strongly prefer
the `Date` class, run `as.Date()` explicitly following
`fread()`. Thanks @scipima for the report and @MichaelChirico for the
fix.

6. `dt[, col]` now returns a copy of `col` also when it is a list
column, as in any other case,
[#4877](https://github.com/Rdatatable/data.table/issues/4877). Thanks
to @tlapak for reporting and the PR.

7. `rbindlist` and `rbind` binding `bit64::integer64` columns with
`character`/`complex`/`list` columns now works,
[#5504](https://github.com/Rdatatable/data.table/issues/5504). Thanks
to @MichaelChirico for the request and @ben-schwen for the PR.

8. Fixed possible segfault in `setDT(df); attr(df, key) <- value;
set(df, ...)`, i.e. adding columns to an object with `set()` that was
converted to data.table with `setDT()` and later had attributes add
with `attr<-`,
[#6410](https://github.com/Rdatatable/data.table/issues/6410). Thanks
to @hongyuanjia for the report and @ben-schwen for the PR. Note that
`setattr()` should be preferred for adding attributes to a data.table.

9. `DT[1, on=NULL]` now works for returning the first row,
[#6579](https://github.com/Rdatatable/data.table/issues/6579). Thanks
to @Kodiologist for the report and @tdhock for the PR.

10. `tables()` now returns the correct size for data.tables over 2GiB,
[#6607](https://github.com/Rdatatable/data.table/issues/6607). Thanks
to @vlulla for the report and the PR.

11. `rbindlist(l, use.names=TRUE)` can now handle different encodings
for the column names in different entries of `l`,
[#5452](https://github.com/Rdatatable/data.table/issues/5452). Thanks
to @MEO265 for the report, and Benjamin Schwendinger for the fix.

12. Added a `data.frame` method for `format_list_item()` to fix error
printing data.tables with columns containing 1-column data.frames,
[#6592](https://github.com/Rdatatable/data.table/issues/6592). Thanks
to @r2evans for the bug report and fix.

13. Auto-printing gets some substantial improvements - Suppression in
`knitr` documents is now done by implementing a method for
`knit_print` instead of looking up the call stack,

[#6589](https://github.com/Rdatatable/data.table/pull/6589). The old
way was fragile and wound up broken by some implementation changes in
{knitr}. Thanks to @jangorecki for the report

[#6509](https://github.com/Rdatatable/data.table/issues/6509) and
@aitap for the fix.  - `print()` methods for S3 subclasses of
data.table (e.g. an object of class `c("my.table", "data.table",
"data.frame")`) no longer print where plain data.tables wouldn't,
e.g. `myDT[, y := 2]`,

[#3029](https://github.com/Rdatatable/data.table/issues/3029). The
improved detection of auto-printing scenarios has the added benefit of
_allowing_ print in highly explicit statements like `print(DT[, y :=
2])`, obviating our recommendation since v1.9.6 to append `[]` to
signal "please print me".

14. Joins of `integer64` and `double` columns succeed when the
`double` column has lossless `integer64` representation,
[#4167](https://github.com/Rdatatable/data.table/issues/4167) and
[#6625](https://github.com/Rdatatable/data.table/issues/6625). Previously,
this only worked when the double column had lossless _32-bit_ integer
representation. Thanks @MichaelChirico for the reports and fix.

15. `DT[order(...)]` better matches `base::order()` behavior by (1)
recognizing the `method=` argument (and erroring since this is not
supported) and (2) accepting a vector of `TRUE`/`FALSE` in
`decreasing=` as an alternative to using `-a` to convey "sort `a`
decreasing",
[#4456](https://github.com/Rdatatable/data.table/issues/4456). Thanks
@jangorecki for the FR and @MichaelChirico for the PR.

16. Assignment with `:=` to an S4 slot of an under-allocated
data.table now works,
[#6704](https://github.com/Rdatatable/data.table/issues/6704). Thanks
@MichaelChirico for the report and fix.

17. `as.data.table()` method for `data.frame`s (especially those with
extended classes) is more consistent with `as.data.frame()` with
respect to rention of attributes,
[#5699](https://github.com/Rdatatable/data.table/issues/5699). Thanks
@jangorecki for the report and fix.

18. Grouped queries on keyed tables no longer return an incorrectly
keyed result if the _ad hoc_ `by=` list has some function call (in
particular, a function which happens to return a strictly decreasing
function of the keys), e.g. `by=.(a = rev(a))`,
[#5583](https://github.com/Rdatatable/data.table/issues/5583). Thanks
@AbrJA for the report and @MichaelChirico for the fix.

19. An integer overflow in `fread()` with lines longer than `2^(31/2)`
bytes is prevented,
[#6729](https://github.com/Rdatatable/data.table/issues/6729). The
typical impact was no worse than a wrong initial allocation size,
corrected later. Thanks to @TaikiSan21 for the report and @aitap for
the fix.

20. Fixed a memory issue causing segfaults in `forder`,
[#6797](https://github.com/Rdatatable/data.table/issues/6797). Thanks
@dkutner for the report and @MichaelChirico for the fix.

21. `setDT(get0('var'))` now correctly modifies `var` by reference,
consistent with the long-standing behavior of `setDT(get('var'))`,
[#6864](https://github.com/Rdatatable/data.table/issues/6864). Thanks
to @rikivillalba for the report and @venom1204 for the fix.

22. `fread()` could fail to read Mac CSV files (with `\r` line
endings) if the file contained any `\n` character, such as a final
`\r\n`. This was fixed by detecting the predominant line ending in a
sample of the file,
[#4186](https://github.com/Rdatatable/data.table/issues/4186). Thanks
to @MPagel for the report and @ben-schwen for the fix.

23. By reference assignments (':=') with functions that modified the
data.table by reference
e.g. (`foo=function(DT){modify(DT);return(1L)}`, `DT[,a:=foo(DT)]`)
returned a malformed data.table due to the modification of the
targeted named column index ("a") during the j expression evaluation
[#6768](https://github.com/Rdatatable/data.table/issues/6768). Thanks
@AntonNM for the report and fix.

### NOTES

1. There is a new vignette on joins! See
`vignette("datatable-joins")`. Thanks to Angel Feliz for authoring it!
Feedback welcome. This vignette has been highly requested since 2017:
[#2181](https://github.com/Rdatatable/data.table/issues/2181).

2. Tests run again when some Suggests packages are missing,
[#6411](https://github.com/Rdatatable/data.table/issues/6411). Thanks
@aadler for the note and @MichaelChirico for the fix.

3. Some grouping operations run much faster under `verbose=TRUE`,
[#6286](https://github.com/Rdatatable/data.table/issues/6286). Thanks
@joshhwuu for the report and fix. This overhead was not present on
Windows. As a rule, users should expect `verbose=TRUE` operations to
run more slowly, as extra statistics might be calculated as part of
the report; here was a case where the overhead was particularly high
and the fix was particularly easy.

4. `set()` and `:=` now provide some extra guidance for common
incorrect approaches to assigning `NULL` to some rows of a list
column. The correct way is to put `list(list(NULL))` on the RHS of
`:=` (or `.(.(NULL))` for short). Thanks to @MichaelChirico for the
suggestion and @Nj221102 for the implementation.

5. Improved the error message when trying to write code like `DT[,
":="(a := b, c := d)]` (which should be `DT[, ":="(a = b, c \ 
= d)]`),
[#5296](https://github.com/Rdatatable/data.table/issues/5296). Thanks
@MichaelChirico for the suggestion & fix.

6. `measurev()` was implemented and documented in v1.15.0, for use
within `melt()`, and it is now exported (dependent packages can now
use without a NOTE from CRAN check).

7. The `dcast()` and `melt()` generics no longer attempt to redirect
to {reshape2} methods when passed non-`data.table`s. If you're still
using {reshape2}, you must use namespace-qualification:
`reshape2::dcast()`, `reshape2::melt()`. We have been warning about
the deprecation since v1.12.4 (2019). Please note that {reshape2} is
retired.

8. `showProgress` in `[` is disabled for "trivial" grouping
(`.NGRP==1L`),
[#6668](https://github.com/Rdatatable/data.table/issues/6668). Thanks
@MichaelChirico for the request and @joshhwuu for the PR.

9. `key<-`, marked as deprecated since 2012 and unusable since
v1.15.0, has been fully removed.

10. The following in-progress deprecations have proceeded:

   + Using `fwrite(logicalAsInt=)` has been upgraded from a warning
   (since v1.15.0) to an error. It will be removed in the next
   release.

   + Using `fread(autostart=)` has been upgraded to an error. It has
   been warning since v1.11.0 (6 years ago). The argument will be
   removed in the next release.

   + Using `droplevels(in.place=TRUE)` (warning since v1.16.0) has
   been upgraded from warning to error. The argument will be removed
   in the next release.

   + Use of `:=` and `with=FALSE` in `[` has been upgraded from
   warning (since v1.15.0) to error. Long ago (before 2014), this was
   needed when, e.g., assigning to a vector of column names defined
   outside the table, but `with=FALSE` is no longer needed to do so:
   `DT[, (cols) := ...]` works fine.

11. Better handling of multibyte characters in `print()`, added in
1.16.0, has the side effect of possibly ignoring invisible characters
like `\n` or `\t` for the purposes of counting width for
`datatable.prettyprint.char`. That's because we switched to using
`strtrim()` over `substring()`, the latter of which is explicitly
discouraged for the purposes of truncating strings, whereas the former
of which has platform-dependent behavior for whether invisible
characters count towards string width.

Log message:
(textproc/R-data.table)  Updated 1.14.10 to 1.16.4

Related NEWS.md extends over 800 lines.
The head lines are:
   5:# data.table [v1.16.4]   4 December 2024
  11:# data.table [v1.16.2]  (9 October 2024)
  35:# data.table [v1.16.0] (25 August 2024)
 246:# data.table [v1.15.4] (27 March 2024)
 256:# data.table [v1.15.2] (27 Feb 2024)
 264:# data.table [v1.15.0] (30 Jan 2024)

And Full part is found at
  https://github.com/Rdatatable/data.table/blob/master/NEWS.md

Log message:
(textproc/R-data.table) +TEST_DEPENDS+= R-nanotime, make test does not pass yet

Log message:
(textproc/R-data.table) Updated 1.14.0 to 1.14.10

# data.table [v1.14.10](https://github.com/Rdatatable/data.table/milestone/20)

## NOTES

1. Maintainer of the package for CRAN releases is from now on Tyson
  Barrett (@tysonstanley),
  [#5710](https://github.com/Rdatatable/data.table/issues/5710).

2. Updated internal code for breaking change of `is.atomic(NULL)` in
  R-devel,
  [#5691](https://github.com/Rdatatable/data.table/pull/5691). Thanks to
  Martin Maechler for the patch.

3. Fix multiple test concerning coercion to missing complex numbers,
  [#5695](https://github.com/Rdatatable/data.table/issues/5695) and
  [#5748](https://github.com/Rdatatable/data.table/issues/5748). Thanks
  to @MichaelChirico and @ben-schwen for the patches.

4. Fix multiple format warnings (e.g., -Wformat)
  [#5712](https://github.com/Rdatatable/data.table/pull/5712),
  [#5781](https://github.com/Rdatatable/data.table/pull/5781),
  [#5880](https://github.com/Rdatatable/data.table/pull/5800),
  [#5786](https://github.com/Rdatatable/data.table/pull/5786). Thanks to
  @MichaelChirico and @jangorecki for the patches.

# data.table \ 
[v1.14.8](https://github.com/Rdatatable/data.table/milestone/28?closed=1)  (17 \ 
Feb 2023)

## NOTES

1. Test 1613.605 now passes changes to `as.data.frame()` in R-devel,
  [#5597](https://github.com/Rdatatable/data.table/pull/5597). Thanks to
  Avraham Adler for reporting.

2. An out of bounds read when combining non-equi join with `by=.EACHI`
  has been found and fixed thanks to clang ASAN,
  [#5598](https://github.com/Rdatatable/data.table/issues/5598). There
  was no bug or consequence because the read was followed (now preceded)
  by a bounds test.

3. `.rbind.data.table` (note the leading `.`) is no longer exported
  when `data.table` is installed in R>=4.0.0 (Apr 2020),
  [#5600](https://github.com/Rdatatable/data.table/pull/5600). It was
  never documented which R-devel now detects and warns about. It is only
  needed by `data.table` internals to support R<4.0.0; see note 1 in
  v1.12.6 (Oct 2019) below in this file for more details.

# data.table \ 
[v1.14.6](https://github.com/Rdatatable/data.table/milestone/27?closed=1)  (16 \ 
Nov 2022)

## BUG FIXES

1. `fread()` could leak memory,
  [#3292](https://github.com/Rdatatable/data.table/issues/3292). Thanks
  to @patrickhowerter for reporting, and Jim Hester for the fix. The fix
  requires R 3.4.0 or later. Loading `data.table` in earlier versions
  now highlights this issue on startup, asks users to upgrade R, and
  warns that we intend to upgrade `data.table`'s dependency from 8 year
  old R 3.1.0 (April 2014) to 5 year old R 3.4.0 (April 2017).

## NOTES

1. Test 1962.098 has been modified to pass latest changes to `POSIXt`
  in R-devel.

2. `test.data.table()` no longer creates `DT` in `.GlobalEnv`, a CRAN
  policy violation,
  [#5514](https://github.com/Rdatatable/data.table/issues/5514). No
  other writes occurred to `.GlobalEnv` and release procedures have been
  improved to prevent this happening again.

3. The memory usage of the test suite has been halved,
  [#5507](https://github.com/Rdatatable/data.table/issues/5507).

# data.table \ 
[v1.14.4](https://github.com/Rdatatable/data.table/milestone/26?closed=1)  (17 \ 
Oct 2022)

## NOTES

1. gcc 12.1 (May 2022) now detects and warns about an always-false
  condition (`-Waddress`) in `fread` which caused a small efficiency
  saving never to be invoked,
  [#5476](https://github.com/Rdatatable/data.table/pull/5476). Thanks to
  CRAN for testing latest versions of compilers.

2. `update.dev.pkg()` has been renamed `update_dev_pkg()` to get out
  of the way of the `stats::update` generic function,
  [#5421](https://github.com/Rdatatable/data.table/pull/5421). This is a
  utility function which upgrades the version of `data.table` to the
  latest commit in development which has passed all tests. As such we
  don't expect any backwards compatibility concerns. Its manual page was
  causing an intermittent hang/crash from `R CMD check` on Windows-only
  on CRAN which we hope will be worked around by changing its name.

3. Internal C code now passes `-Wstrict-prototypes` to satisfy the
  warnings now displayed on CRAN,
  [#5477](https://github.com/Rdatatable/data.table/pull/5477).

4. `write.csv` in R-devel no longer responds to
  `getOption("digits.secs")` for `POSIXct`,
  [#5478](https://github.com/Rdatatable/data.table/issues/5478). This
  caused our tests of `fwrite(, dateTimeAs="write.csv")` to fail on
  CRAN's daily checks using latest daily R-devel. While R-devel
  discussion continues, and currently it seems like the change is
  intended with further changes possible, this `data.table` release
  massages our tests to pass on latest R-devel. The idea is to try to
  get out of the way of R-devel changes in this regard until the new
  behavior of `write.csv` is released and confirmed. Package updates are
  not accepted on CRAN if they do not pass the latest daily version of
  R-devel, even if R-devel changes after the package update is
  submitted. If the change to `write.csv()` stands, then a future
  release of `data.table` will be needed to make `fwrite(,
  dateTimeAs="write.csv")` match `write.csv()` output again in that
  future version of R onwards. If you use an older version of
  `data.table` than said future one in the said future version of R,
  then `fwrite(, dateTimeAs="write.csv")` may not match `write.csv()` if
  you are using `getOption("digits.secs")` too. However, you can always
  check that your installation of `data.table` works in your version of
  R on your platform by simply running `test.data.table()`
  yourself. Doing so would detect such a situation for you: test 1741
  would fail in this case. `test.data.table()` runs the entire suite of
  tests and is always available to you locally. This way you do not need
  to rely on our statements about which combinations of versions of R
  and `data.table` on which platforms we have tested and support; just
  run `test.data.table()` yourself. Having said that, because test 1741
  has been relaxed in this release in order to be accepted on CRAN to
  pass latest R-devel, this won't be true for this particular release in
  regard to this particular test.

    ```R
    $ R --vanilla
    R version 4.2.1 (2022-06-23) -- "Funny-Looking Kid"
    > DF = data.frame(A=as.POSIXct("2022-10-01 01:23:45.012"))
    > options(digits.secs=0)
    > write.csv(DF)
    "","A"
    "1",2022-10-01 01:23:45
    > options(digits.secs=3)
    > write.csv(DF)
    "","A"
    "1",2022-10-01 01:23:45.012

    $ Rdevel --vanilla
    R Under development (unstable) (2022-10-06 r83040) -- "Unsuffered \ 
Consequences"
    > DF = data.frame(A=as.POSIXct("2022-10-01 01:23:45.012"))
    > options(digits.secs=0)
    > write.csv(DF)
    "","A"
    "1",2022-10-01 01:23:45.012
    ```

5. Many thanks to Kurt Hornik for investigating potential impact of a
  possible future change to `base::intersect()` on empty input,
  providing a patch so that `data.table` won't break if the change is
  made to R, and giving us plenty of notice,
  [#5183](https://github.com/Rdatatable/data.table/pull/5183).

6. `datatable.[dll|so]` has changed name to `data_table.[dll|so]`,
  [#4442](https://github.com/Rdatatable/data.table/pull/4442). Thanks to
  Jan Gorecki for the PR. We had previously removed the `.` since `.` is
  not allowed by the following paragraph in the Writing-R-Extensions
  manual. Replacing `.` with `_` instead now seems more consistent with
  the last sentence.

    > ... the basename of the DLL needs to be both a valid file name
      and valid as part of a C entry point (e.g. it cannot contain
      ‘.’): for portable code it is best to confine DLL names to be
      ASCII alphanumeric plus underscore. If entry point R_init_lib is
      not found it is also looked for with ‘.’ replaced by ‘_’.

# data.table \ 
[v1.14.2](https://github.com/Rdatatable/data.table/milestone/24?closed=1)  (27 \ 
Sep 2021)

## NOTES

1. clang 13.0.0 (Sep 2021) requires the system header `omp.h` to be
  included before R's headers,
  [#5122](https://github.com/Rdatatable/data.table/issues/5122). Many
  thanks to Prof Ripley for testing and providing a patch file.

Log message:
textproc: Replace RMD160 checksums with BLAKE2s checksums

All checksums have been double-checked against existing RMD160 and
SHA512 hashes

Unfetchable distfiles (fetched conditionally?):
./textproc/convertlit/distinfo clit18src.zip

Log message:
textproc: Remove SHA1 hashes for distfiles

Log message:
textproc/R-data.table: Release maintainership

Narrowing my focus to biology packages

Log message:
(textproc/R-data.table) updated  1.12.2 to 1.14.0

NEWS.md from 1.12.2 to 1.14.0 has over 1,000 lines, See following
URL for full text:
https://github.com/Rdatatable/data.table/blob/master/NEWS.md

Here is only for 'POTENTIALLY BREAKING CHANGES' in 1.140.0

# data.table \ 
[v1.14.0](https://github.com/Rdatatable/data.table/milestone/23?closed=1)

## POTENTIALLY BREAKING CHANGES

1. In v1.13.0 (July 2020) native parsing of datetime was added to
`fread` by Michael Chirico which dramatically improved
performance. Before then datetime was read as type character by
default which was slow. Since v1.13.0, UTC-marked datetime
(e.g. `2020-07-24T10:11:12.134Z` where the final `Z` is present) has
been read automatically as POSIXct and quickly. We provided the
migration option `datatable.old.fread.datetime.character` to revert to
the previous slow character behavior. We also added the `tz=` argument
to control unmarked datetime; i.e. where the `Z` (or equivalent UTC
postfix) is missing in the data. The default `tz=""` reads unmarked
datetime as character as before, slowly. We gave you the ability to
set `tz="UTC"` to turn on the new behavior and read unmarked datetime
as UTC, quickly. R sessions that are running in UTC by setting the TZ
environment variable, as is good practice and common in production,
have also been reading unmarked datetime as UTC since v1.13.0, much
faster. Note 1 of v1.13.0 (below in this file) ended `In addition to
convenience, fread is now significantly faster in the presence of
dates, UTC-marked datetimes, and unmarked datetime when tz="UTC" is
provided.`.

    At `rstudio::global(2021)`, Neal Richardson, Director of
    Engineering at Ursa Labs, compared Arrow CSV performance to
    `data.table` CSV performance, [Bigger Data With Ease Using Apache
    \ 
Arrow](https://rstudio.com/resources/rstudioglobal-2021/bigger-data-with-ease-using-apache-arrow/). \ 
He
    opened by comparing to `data.table` as his main point. Arrow was
    presented as 3 times faster than `data.table`. He talked at length
    about this result. However, no reproducible code was provided and
    we were not contacted in advance in case we had any comments. He
    mentioned New York Taxi data in his talk which is a dataset known
    to us as containing unmarked
    datetime. [Rebuttal](https://twitter.com/MattDowle/status/1360073970498875394).

    `tz=`'s default is now changed from `""` to `"UTC"`. If \ 
you have
    been using `tz=` explicitly then there should be no change. The
    change to read UTC-marked datetime as POSIXct rather than
    character already happened in v1.13.0. The change now is that
    unmarked datetimes are now read as UTC too by default without
    needing to set `tz="UTC"`. None of the 1,017 CRAN packages
    directly using `data.table` are affected. As before, the migration
    option `datatable.old.fread.datetime.character` can still be set
    to TRUE to revert to the old character behavior. This migration
    option is temporary and will be removed in the near future.