Make (Rcpp)Eigen use threads the way (Rcpp)Armadillo does

[eddelbuettel]

While doing some work updating and overhauling (Rcpp)Armadillo last summer and fall, we also revisited / simplified some of the threading (including as always a redo of what macOS does; let's leave that aside for now).

I carried some of that over to RcppEigen as well so that by default we get all cores (if a user opts in). I still have that code here and had been meaning to reach out to 'interested parties': @jaganmn @bbolker @bgoodri @SteveBronder come to mind.

My current thinking is that we should be able to set something up that 'does no harm' (i.e. defaults to a max of two threads unless told otherwise to not cause harm at CRAN) but goes beyond if suitable local environment variables or config settings are found. So the current status quo, as I recall it, is that

• package RcppEigen ignores the existence of OpenMP around R, RcppEigen:::EigenNbThreads() (which existed for years) return 1
• just by adding 'use OpenMP where is exists' we can do better, i.e. on my machine I now get all cores (which would upset CRAN)
• we just decide which throttling mechanism we like
  • ie maybe borrow the data.table one which uses 50%
  • and/or combine with a configuration variable as I once did for tiledb

Thoughts? I'll try what I have here (uncommitted, and it isn't much yet) into another branch

[eddelbuettel]

I.e. following data.table may be enough (though we may want to use 100% instead of 50% or...)

$ Rscript -e 'data.table:::getDTthreads()'   # by default 50% of all, here 0.5 * 32 threads
[1] 16
$ OMP_THREAD_LIMIT=2 Rscript -e 'data.table:::getDTthreads()'   # CRAN setting
[1] 2
$ 

[eddelbuettel]

And I have a simple version of this now:

$ Rscript -e 'RcppEigen:::EigenNbThreads()'   # no limit by default
[1] 32
$ OMP_THREAD_LIMIT=4 Rscript -e 'RcppEigen:::EigenNbThreads()'
[1] 4
$ Rscript -e 'options(Ncpus=6); RcppEigen:::EigenNbThreads()'
[1] 6
$ 

This would 'do the Right Thing (TM)' at CRAN via OMP_THREAD_LIMIT being set to two, also allows option Ncpus to be set, and if all else fails uses all machine cores.

PS In the branch feature/openmp.

PPS And once more just comparison in a 'pristine' r2u container:

$ Rscript -e 'RcppEigen:::EigenNbThreads()'        # using CRAN RcppEigen         
$ # .... installing from the feature/openmp branch ... not shown
$ Rscript -e 'RcppEigen:::EigenNbThreads()'       # new behavior using OpenMP
[1] 32
$ OMP_THREAD_LIMIT=2 Rscript -e 'RcppEigen:::EigenNbThreads()'   # 'CRAN-like' when limit is 2
[1] 2
$