Vignettes

[arunsrinivasan]

HTML vignette series:

Planned for v1.9.8

• Quick tour of data.table
• Keys and fast binary search based subset
• Secondary indices and auto indexing
• Joins vignette. a) joins vs subsets -- extending binary search based subset to joins + conditional / non-equi joins, rolling and interval joins. b) by=.EACHI, join + update feature. c) Document i.col usage as filed in Docs: explain and document the i.col notation for joins #1038. d) Also cover about performance/advantages from on performing slower than double setkey #1232.
• [ ] Cover get() and mget(). E.g., http://stackoverflow.com/q/33785747/559784 covered in programming on data.table #4304
• Add about on= argument rationale in FAQ ([Documentation] Use of the on= argument for joins #1623).
• FAQ 5.3 needs to mention that it's a shallow copy that's done in order to restore over-allocation. Thanks to Jan for linking it in := changes address of a data table #1729.

---

Future releases

• data.table internals, performance aspects and expressiveness
• Reading multiple files (fread + rbindlist), ordering, ranking and set operations
• IDateTime vignette
• Document the difference between data.table() and data.frame() somewhere - relevant issues: Creation of data.table using a list #968, data.table(x) != as.data.table(x) #877. Perhaps slightly more in detail in the FAQ.
• coursera FAQ
• Advanced data.table usage:
  • NSE
  • ...
• Timings vignette (moving [R-Forge #1133] Add r-help example to timings.Rnw #520 here to get everything in one place, but not sure if we need it as a vignette since we've the Wiki with benchmarks/timings).
• fread+fwrite vignette, include also Convenience features of fread wiki, also fread (and fwrite) vignette #2855

---

Finished:

• Introduction to data.table - data.table syntax, general form, subset rows in i, select / do in j and aggregations using by.
• Reference Semantics (add/update/delete columns by reference, and see that we can combine with i and by in the same way as before)
• Efficient reshaping using data.tables
• Link to this answer on SO on by=.EACHI until the vignette is done.

---

Minor:

• Operations using integer64, and promoting it for large integers.

---

Notes (to update current vignettes based on feedbacks): Please let me know if I missed anything..

Introduction to data.table:

• order in i.
• Explain how to name columns in j while selecting/computing.
• Emphasise that keyby is applied after obtaining the result on the computed result, not on the original data.table.
• Mention new updates to .SDcols and cols in with=FALSE being able to select columns as colA:colB.

Reference semantics:

• Also explain all other relevant set* functions here.. (setnames, setcolorder etc..)
• Mainly set.
• Explain that 1b) the := operator is just defining ways to use it - the example there doesn't work as it just shows two different ways of using it -- Following this comment.

Keys and fast binary search based subsets:

• Add an example of subset using integer/double keys.
• Difference in "nomatch" default in binary search based subsets.
• replacing NAs with binary search based subsets possible?

FAQ (most appropriate here, I think).

• Update FAQ with issue on external pointer being NULL when reading an R object from file, for example, using readRDS(). Update this SO post.
• Explain with example, on over allocating the data.table using alloc.col(), and when to use it (when you need to create multiple columns), and why. Update this SO post.

[jangorecki]

fread is a least worth to mention.
Above points are related mainly to data transformation, fread is more a data extraction so it might be skipped in such vignette, yet IMO it is worth to mention such data.table capabilities.

edit: which one you are going to use: Rnw or Rmd?

[arunsrinivasan]

Agreed, and updated.

[matthieugomez]

I'm curious about what makes a cold by faster than say tapply. One part of the answer is gforce, but what about user written functions? I could not find anything about this. There's a nice post about panda : http://wesmckinney.com/blog/?p=489
One could even compare it with sapply. For instance, suppose I start from a list of vectors. Is it ever worth it to append all the vectors in one column in a data.table and use by instead of sapply ?

[arunsrinivasan]

@matthieugomez interesting question! Would be nice to cover this as well. Keep'em coming :-).

[gsee]

I would be interested to learn about IDateTime and some of the use cases for it.

[arunsrinivasan]

@gsee updated.

[markdanese]

Being new to R and data.table (since March), I would say that there needs to be a basic outcome-oriented introduction as opposed to the current function-oriented one. In other words, it is one thing to read what each parameter in data.table does, but they often make little sense without having a use-case in mind. While there are examples of output, many people need to go the other direction. That is, they know what output they need, but they don't know what function/parameter/setting is most appropriate to use. It would be helpful to have a simple recipe approach to get them started.

How to I create subsets of my data?
How do I do an operation on subsets of my data to create a new or updated data set?
How do I add a new column?
How do I delete a column?
How do I create a single variable?
How do I create multiple variables?
How do I do different operations on different subsets of my data? (.BY)
How do I use data.table in a function and pass in data.table names and columns on which to operate?
How do I do multiple sequential operations on the same data.table?
Can I select a subset of data and do an operation on it at the same time?
When do I need to be careful about creating/updating variables by reference?
How do I select one observation per group (first, last)?
How do I set a key and how is it different from setting an index?
Under what conditions does my key get deleted when I do an operation on my data.table?
Can I just use the regular "merge" syntax or do I need to use data.table syntax (Y[X])?
How do I collapse a list of lists into one big data.table? What if the columns are in different order?

There are probably a ton of other items all on SO that could be edited into a simple compilation of questions and answers.

[arunsrinivasan]

@markdanese Thanks for your suggestions. These are all great to have, but probably as a separate wiki, as they're very particular about certain tasks. The objective of the vignette is to introduce to data.table syntax illustrating how flexible and powerful it can be, so that you are able to do these tasks yourself.

I'm writing the vignettes now (as fast as I can), and the format is more or less in this fashion (Q&A) and explaining the answer with an example. Once I've the first vignette polished, I intend to post it here to get some feedback.. It'd be great to know what you think as well.

Thanks again.

[vlulla]

Further extension of the idea of the wiki page: The FAQ and Code Fragments (Advanced) links listed on http://www.ats.ucla.edu/stat/r/ might be a useful resource for contrasting traditional tasks in R with data.table way. I did something like this in a blog post (http://vijaylulla.com/wp/2014/11/12/grouping-in-r-using-data-table/) to show it to my colleague. Sorry for the shameless self promotion.

[arunsrinivasan]

I've finished the Introduction to data.table vignette (see link on top). It'd be great to know what you think.

Thanks to @jangorecki and @brodieG for great feedbacks, and of course @mattdowle :-).