snp_readBGI() when using an outdated version of package {bit64}.snp_cor() and bed_cor() now use less memory.Remove parameter info from snp_cor() and bed_cor() because this correction is not useful after all.
snp_cor() and bed_cor() now return NaNs when e.g. the standard deviation is 0 (and warn about it). Before, these values were not reported (i.e. treated as 0).
snp_readBGI().snp_manhattan() when non-ordered (chr, pos) are provided.snp_ancestry_summary() by allowing to estimate ancestry proportions after PCA projection (instead of directly using the allele frequencies).Add function bed_cor() (similar to snp_cor() but with bed files/objects directly).
Add functions snp_ld_scores() and bed_ld_scores().
snp_ancestry_summary() to estimate ancestry proportions from a cohort using only its summary allele frequencies.snp_scaleAlpha(), which is similar to snp_scaleBinom(), but has a parameter alpha that controls the relation between the scaling and the allele frequencies.snp_cor() now also uses the upper triangle (@uplo = "U") when the sparse correlation matrix is diagonal, so that it is easier to use with e.g. as_SFBM().type in snp_asGeneticPos() to also be able to use interpolated genetic maps from here.return_flip_and_rev to snp_match() for whether to return internal boolean variables "_FLIP_" and "_REV_".$perc_kept in the output of snp_ldsplit(), the percentage of initial non-zero values kept within the blocks defined.snp_prodBGEN().snp_prodBGEN() to compute a matrix product between BGEN files and a matrix (or a vector). This removes the need to read an intermediate FBM object with snp_readBGEN() to compute the product. Moreover, when using dosages, they are not rounded to two decimal places anymore.Trade new parameter num_iter_change for a simpler allow_jump_sign.
Change defaults in LDpred2-auto to use 500 burn-in iterations (was 1000 before) followed by 200 iterations (500 before). Such a large number of iterations is usually not really needed.
as_SFBM(corr0, compact = TRUE). Make sure to reinstall {bigsnpr} after updating to {bigsparser} v0.5.Prepare for incoming paper on (among other things) improved robustness of LDpred2-auto:
add parameter shrink_corr to shrink off-diagonal elements of the LD matrix,
add parameter num_iter_change to control when starting to shrink the variants that change sign too much,
also return corr_est, the “imputed” correlations between variants and phenotypes, which can be used for post-QCing variants by comparing those to beta / sqrt(n_eff * beta_se^2 + beta^2).
Replace parameter s by delta in snp_lassosum2(). This new parameter delta better reflects that the lassosum model also uses L2-regularization (therefore, elastic-net regularization).
Now detect strong divergence in lassosum2 and LDpred2-grid, and return missing values for the corresponding effect sizes.
snp_ldsc() when using blocks with different sizes.info to snp_cor() to correct correlations when they are computed from imputed dosage data.snp_readBGEN() now also returns frequencies and imputation INFO scores.rsid to snp_asGeneticPos() to also allow matching with rsIDs.snp_lassosum2() to train the lassosum models using the exact same input data as LDpred2.report_step in snp_ldpred2_auto() to report some of the internal sampling betas.snp_readBGEN() when using BGEN files containing ~.thr_r2 in snp_cor().snp_ldsplit(). Instead, report the best splits for a range of numbers of blocks desired.snp_ldsplit() now makes more sense. Also fix a small bug that prevented splitting the last block in some cases.snp_ldsplit() for optimally splitting variants in nearly independent blocks of LD.file.type = "--gzvcf" for using gzipped VCF in snp_plinkQC().snp_assocBGEN(); prefer reading small parts with snp_readBGEN() as a temporary bigSNP object and do the association test with e.g. big_univLinReg().snp_thr_correct() for correcting for winner’s curse in summary statistics when using p-value thresholding.Use a better formula for the scale in LDpred2, useful when there are some variants with very large effects (e.g. explaining more than 10% phenotypic variance).
Simplify LDpred2; there was not really any need for initialization and ordering of the Gibbs sampler.
return_sampling_betas in snp_ldpred2_grid() to return all sampling betas (after burn-in), which is useful for assessing the uncertainty of the PRS at the individual level (see https://doi.org/10.1101/2020.11.30.403188).$postp_est, $h2_init and $p_init in LDpred2-auto.snp_readBGEN() to make sure of the expected format.snp_fst() for computing Fst.could not find function "ldpred2_gibbs_auto".as_SFBM(corr0) instead of bigsparser::as_SFBM(as(corr0, "dgCMatrix")). This should also use less memory and be faster.sparse to enable getting also a sparse solution in LDpred2-auto.bigsparser::as_SFBM().01 or 1 for chromosomes in BGI files.snp_match(). Also now remove duplicates by default.All 3 LDpred2 functions now use an SFBM as input format for the correlation matrix.
Allow for multiple initial values for p in snp_ldpred2_auto().
Add function coef_to_liab() for e.g. converting heritability to the liability scale.
alpha of function snp_cor() to 1.Add functions snp_ldpred2_inf(), snp_ldpred2_grid() and snp_ldpred2_auto() for running the new LDpred2-inf, LDpred2-grid and LDpred2-auto.
Add functions snp_ldsc() and snp_ldsc2() for performing LD score regression.
Add function snp_asGeneticPos() for transforming physical positions to genetic positions.
Add function snp_simuPheno() for simulating phenotypes.
snp_pcadapt(), bed_pcadapt(), snp_readBGEN() and snp_fastImputeSimple().Parallelization of clumping algorithms has been modified. Before, chromosomes were imputed in parallel. Now, chromosomes are processed sequentially, but computations within each chromosome are performed in parallel thanks to OpenMP. This should prevent major slowdowns for very large samples sizes (due to swapping).
Use OpenMP to parallelize other functions as well (possibly only sequential until now).
Can now run snp_cor() in parallel.
Parallelization of snp_fastImpute() has been modified. Before this version, chromosomes were imputed in parallel. Now, chromosomes are processed sequentially, but computation of correlation between variants and XGBoost models are performed using parallelization.
snp_subset() as alias of method subset() for subsetting bigSNP objects.bed_light internally to make parallel algorithms faster because they have to transfer less data to clusters. Also define differently functions used in big_parallelize() for the same reason.object 'obj.bed' not found in snp_readBed2().backingfile to subset.bigSNP().byrow to bed_counts().Add memory-mapping on PLINK (.bed) files with missing values + new functions:
bed()bed_MAF()bed_autoSVD()bed_clumping()bed_counts()bed_cprodVec()bed_pcadapt()bed_prodVec()bed_projectPCA()bed_projectSelfPCA()bed_randomSVD()bed_scaleBinom()bed_tcrossprodSelf()download_1000G()snp_modifyBuild()snp_plinkKINGQC()snp_readBed2()sub_bed()Add 3 parameters to autoSVD(): alpha.tukey, min.mac and max.iter.
Remove option for changing ploidy (that was only partially supported).
Automatically apply snp_gc() to pcadapt.
snp_fastImputeSimple(): fast imputation via mode, mean or sampling according to allele frequencies.snp_readBGEN() that could not handle duplicated variants or individuals.snp_grid_PRS(), it now stores not only the FBM, but also the input parameters as attributes (the whole result basically).Add 3 SCT functions snp_grid_*() to improve from Clumping and Thresholding (preprint coming soon).
Add snp_match() function to match between summary statistics and some SNP information.
is.size.in.bp is deprecated.read_as for snp_readBGEN(). It is now possible to sample BGEN probabilities as random hard calls using read_as = "random". Default remains reading probabilities as dosages.For memory-mapping, now use mio instead of boost.
snp_clumping() (and snp_autoSVD()) now has a size that is inversely proportional to thr.r2.
snp_pruning() is deprecated (and will be removed someday); now always use snp_clumping().
snp_assocBGEN() for computing quick association tests from BGEN files. Could be useful for quick screening of useful SNPs to read in bigSNP format. This function might be improved in the future.snp_readBGEN() to read UK Biobank BGEN files in bigSNP format.Add parameter is.size.in.bp to snp_autoSVD() for the clumping part.
Change the threshold of outlier detection in snp_autoSVD() (it now detects less outliers). See the documentation details if you don’t have any information about SNPs.
snp_gene (as a gist) to get genes corresponding to ‘rs’ SNP IDs thanks to package {rsnps} from rOpenSci. See README.snp_fastImpute. Also store information in an FBM (instead of a data frame) so that imputation can be done by parts (you can stop the imputation by killing the R processes and come back to it later). Note that the defaults used in the Bioinformatics paper were alpha = 0.02 and size = 500 (instead of 1e-4 and 200 now, respectively). These new defaults are more stringent on the SNPs that are used, which makes the imputation faster (30 min instead of 42-48 min), without impacting accuracy (still 4.7-4.8% of errors).