RINS_HOWTO

NOTE.  THIS FILE IS NOW MOSTLY OUTDATED AND KEPT ONLY AS A HISTORICAL REFERENCE







RINS (Rapid Identification of Non-human Sequence)

http://khavarilab.stanford.edu/resources.html


Still having some issues so trying to install the versions that were available when the paper was published.


Downloaded ...

http://khavarilab.stanford.edu/downloads/rins_core.tar.gz
https://s3.amazonaws.com/changseq/kqu/rins/rins.tar.gz

http://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=Download
( ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.2.26+-universal-macosx.tar.gz )
 
http://users.soe.ucsc.edu/~kent/src/
( http://users.soe.ucsc.edu/~kent/src/blatSrc34.zip )
( http://users.soe.ucsc.edu/~kent/src/blatSrc35.zip )


http://bowtie-bio.sourceforge.net/
( http://sourceforge.net/projects/bowtie-bio/files/bowtie/0.12.8/bowtie-0.12.8-src.zip/download )
( http://sourceforge.net/projects/bowtie-bio/files/bowtie2/2.0.0-beta7/bowtie2-2.0.0-beta7-source.zip/download )

http://trinityrnaseq.sourceforge.net
( http://sourceforge.net/projects/trinityrnaseq/files/trinityrnaseq_r2012-06-08.tgz/download )


--------------------------------------------------
Preparation (for usage on a Mac using tcsh)

	XCode and it's command line tools must be installed
	MacPorts must be installed
	( don't know which ports )

	sudo port install git-core
	sudo port install libpng


	R and some of these Perl libs are only needed for some tests.

	sudo port install R


	trinity/sample_data/test_Trinity_Coding_Extraction
	will need some Perl libs ...
cpan install URI
cpan install URI::Escape


	trinity/sample_data/test_edgeR_diff_expr/example_edgeR_analysis_pipeline
	trinity/sample_data/test_edgeR_diff_expr/testEdgeRfuncs
	will require 'R'
	also the 'edgeR' library (which I can't find just yet)
	Download and unzip package source for limma and edgeR
http://www.bioconductor.org/packages/2.10/bioc/html/limma.html
http://www.bioconductor.org/packages/2.10/bioc/html/edgeR.html
	R CMD INSTALL /Users/jakewendt/limma
	R CMD INSTALL /Users/jakewendt/edgeR


	echo $PATH should include /opt/local/bin

	Add the following to your .tcshrc (or modified to the appropriate *rc file)
	( could also add "source ~/rins/ccls/rinsrc" to your .tcshrc )

setenv PATH ${PATH}:${HOME}/rins/ccls:${HOME}/rins/core/:${HOME}/rins/blast:${HOME}/rins/blat:${HOME}/rins/bowtie:${HOME}/rins/trinity
#	for simplicity
setenv RINS_SRC_DIR /Volumes/BUF-Fileshare/RestrictedData/StaffFolders/WendtJake/RINS

	open a new window and close this one





--------------------------------------------------
Install our own scripts 
(if you are reading this you may have already done this somewhere else.)

mkdir -p ~/rins
cd ~/rins
git clone git://github.com/ccls/sequencing.git ccls


--------------------------------------------------
Install BLAST
	
mkdir -p ~/rins/blast
mkdir ~/temporary_rins
cd ~/temporary_rins
tar xfz $RINS_SRC_DIR/ncbi-blast-2.2.26+-universal-macosx.tar.gz
cd ncbi-blast-2.2.26+
mv bin/* ~/rins/blast/

#	Compiling from source is problematic and time consuming.
#	tar xfz $RINS_SRC_DIR/ncbi-blast-2.2.26+-src.tar.gz
#	cd ncbi-blast-2.2.26+-src/c++
#	# adding --with-64 only delays the failures
#	./configure --prefix=$HOME/rins
#	make
#	make install

--------------------------------------------------
Install BLAT

There is a difference between 34 and 35.  I don't know exactly what it is
but 35 finds nothing and 34 finds a bunch.  Perhaps a change in parameters?
34 also requires a few tweeks just to get it to compile?

mkdir -p ~/rins/blat
cd ~/temporary_rins
unzip $RINS_SRC_DIR/blatSrc35.zip
cd blatSrc
make C_INCLUDE_PATH=/opt/local/include/ PNGLIB=/opt/local/lib/libpng.a BINDIR=${HOME}/rins/blat/

	#	make seems to do a make install (unconventional, and poor design)

--------------------------------------------------
Install BOWTIE

mkdir -p ~/rins/bowtie
cd ~/temporary_rins
unzip $RINS_SRC_DIR/bowtie-0.12.8-src.zip
cd bowtie-0.12.8
make
mv bowtie bowtie-build bowtie-inspect ${HOME}/rins/bowtie/


--------------------------------------------------
Install BOWTIE2 (accidental extra)

mkdir -p ~/rins/bowtie2
cd ~/temporary_rins
unzip $RINS_SRC_DIR/bowtie2-2.0.0-beta7-source.zip
cd bowtie2-2.0.0-beta7
make
mv bowtie2 bowtie2-build bowtie2-align bowtie2-inspect ${HOME}/rins/bowtie/

#	No need for another dir since filenames are different

	#	doesn't seem to be a make install
	#	not really sure what needs to be copied to bin
	#	no subdirs so just moving the 'new' stuff


--------------------------------------------------
Install TRINITY (a bit of a clusterf*k to 'install')

#	Trying to use older version so params match and can compare output.
#	2011-08-20 version will require additional stuff
#		cd Inchworm
#		make clean
#		./configure --prefix=`pwd`
#		make
#		make install
#		cd ../Chrysalis
#		make clean UNSUPPORTED=yes
#		make UNSUPPORTED=yes


mkdir -p ~/rins/trinity
cd ~/temporary_rins
tar xfz $RINS_SRC_DIR/trinityrnaseq_r2012-06-08.tgz
cd trinityrnaseq_r2012-06-08
make

	#	This is excessive but what to do?
	#	Very poor software design to not have a 'make install'.
mv * ${HOME}/rins/trinity/


--------------------------------------------------
Install RINS Core (a bunch of perl scripts)

mkdir -p ~/rins/core
cd ~/temporary_rins
tar xfz $RINS_SRC_DIR/rins_core.tar.gz
cd rins_core
mv * ${HOME}/rins/core/


--------------------------------------------------
How about we test something. (This data set is HUMUNGOUS!)
( 6GB tar.gz, 18GB untared )

mkdir ~/rins_test
cd ~/rins_test
tar xfz $RINS_SRC_DIR/rins.tar.gz




edit config.txt




completes, but generates some errors and result.txt has no data


> rins.pl -c config.txt -o result.txt
Tue Aug 28 15:37:52 PDT 2012
step 1 change fastq files to fasta files
already fasta format, copy fasta files instead
step 2 chop reads
1000000
1000000
step 3 blat chopped reads
Loaded 340163412 letters in 32101 sequences
Error: sequence name gi_77020114_ref_NC_007461.1__Chlamydia_phage_4,_complete_genome is repeated in the database, all names must be unique.
Loaded 340163412 letters in 32101 sequences
Error: sequence name gi_77020114_ref_NC_007461.1__Chlamydia_phage_4,_complete_genome is repeated in the database, all names must be unique.
step 4 find blat out candidate reads
step 5 compress raw reads
step 6 pull reads from blat_out_candidate fasta files
step 7 align compressed reads to human genome reference using bowtie
step 8 1 iteration
de novo assembly using Trinity
Error, do not understand options: --paired_fragment_length 300 --run_butterfly
cp: trinity_output/Trinity.fasta: No such file or directory
blastn trinity output against human genome
clean up blastn outputs
step 8 2 iteration
blat chopped reads
Loaded 0 letters in 0 sequences
Searched 56836892 bases in 567692 sequences
Loaded 0 letters in 0 sequences
Searched 56769200 bases in 567692 sequences
find blat out candidate reads
de novo assembly using Trinity
rm: trinity_output: No such file or directory
Error, do not understand options: --paired_fragment_length 300 --run_butterfly
cp: trinity_output/Trinity.fasta: No such file or directory
blastn trinity output against human genome
clean up blastn outputs
rm: trinity_output: No such file or directory
step 9 detect species of non human sequences
blastn trinity output against non-human genome
Loaded 0 letters in 0 sequences
Searched 0 bases in 0 sequences
Loaded 0 letters in 0 sequences
Searched 0 bases in 0 sequences
write results
Attempting to mail jakewendt@berkeley.edu the complete notice
Tue Aug 28 15:39:34 PDT 2012



removed duplicates from virus index (no idea if same, different, better, worse)
gi_77020114_ref_NC_007461.1__Chlamydia_phage_4,_complete_genome
gi_282599175_ref_NC_013693.1__Shigella_phage_phiSboM-AG3,_complete_genome
gi_306478709_ref_NC_014473.1__Croton_yellow_vein_virus,_complete_genome
gi_240017693_ref_NC_011052.2__Sweet_potato_leaf_curl_Spain_virus,_complete_genome
gi_157311299_ref_NC_009821.1__Enterobacteria_phage_Phi1,_complete_genome

Not doing this anymore.  Looks like there are just short of 2000 duplicate sequences.


Truncated virus.fa at second  ...
gi_77020114_ref_NC_007461.1__Chlamydia_phage_4,_complete_genome


Sadly, it appears that the rins.pl script is a bit out of sync with
the other software too. Had to modify some params for calls to Trinity.pl.



blat output psl files are empty (step 3)

wrote script to rename duplicate sequence names (doesn't really seem necessary)
still blat output is empty













I manually ran blat on the chopped and unchopped fa files and
found that the unchopped files seem to actually produce a non-empty
psl output file?  Investigating.


Also seems that bowtie is silently seg faulting!!!
That's always nice.

> bowtie -n 3 -p 6 -f -S /Users/jakewendt/rins_test/rins/indexes/hg18 compress_leftlane.fa compress_leftlane.sam
Segmentation fault


It would seem that the included hg18 "database" is corrupt and does not pass the 

blastdbcheck -db indexex/hg18

I recreated the hg18 ebwt files from the included hg18.fa file

cd indexes; bowtie-build hg18.fa hg18
The resulting files are the same size, but 1 does differ.

> diff ~/rins_test/rins/*indexes/hg18.1.ebwt
Binary files /Users/jakewendt/rins_test/rins/indexes/hg18.1.ebwt and /Users/jakewendt/rins_test/rins/new_indexes/hg18.1.ebwt differ



























--------------------------------------------------
### My Upgrades

I have updated the parameters used to call Trinity.pl as 
some of the existing ones are no longer used.

I have "uniquified" the virus sequence names in the virus db.

I have modified all system calls to print command before execution.

I have added, and am still adding, checks along the way.

blastn created human_contig.txt not formated as expected and generates divide by 0
	blastn_cleanup.pl human_contig.txt Trinity.fasta clean_blastn.fa 0.8
	my ($front, $back) = split (/len[:]/, $data[0]);
Created my own version

Similar problem with write_result.pl
	file format is even more different here?

Old blastn output "-outfmt 6" format ...
comp7_c0_seq1_FPKM_all:46727.836_FPKM_rel:46727.836_len:377_path:[0,586,641,781,805,811] chr3  99.26 136 0 1 242 377 197266575 197266441 2e-62  244
New blastn output "-outfmt 6" format ...
comp430_c0_seq1 chrX  97.24 398 10  1 1 398 108184401 108184005 0.0  673





SAM_filter_out_unmapped_reads.pl raises divide by zero when total_count is 0
I don't know if the total_count == 0 is bad anyway, but there is no check.
This is part of Trinity and the script that calls it explicitly specifies
the path so I had to actually edit the Trinity version. Or would have to
modify and include the script that calls it, and the script that calls that,
and the script that calls that, and .........



I have also modified to the script to have many defaults such as
* command line defaults
    * input config file (-c) defaults to config.txt
    * output file (-o) defaults to results.txt
* config file defaults
    * blat_bin: blat (expected to be in PATH)
    * bowtie_bin: bowtie (expected to be in PATH)
    * bowtie_build_bin: bowtie-build (expected to be in PATH)
    * trinity_script: Trinity.pl (expected to be in PATH)
    * blastn_bin: blastn (expected to be in PATH)
    * raw_read_length: 100
    * chop_read_length: 25
    * minIdentity: 80
    * compress_ratio_thrd: 0.5
    * iteration: 2
    * bowtie_threads: 6
    * bowtie_mismatch: 3
    * paired_fragment_length: 300
    * min_contig_length: 300
    * trinity_threads: 6
    * blastn_evalue_thrd: 0.05
    * similarity_thrd: 0.8

I am considering logging the entire output of the scripts, but it is simply easier for the user to use something like 'tee'.  In addition, some of these applications, particularly the java ones, are very resource intensive.  They should be run nicely or you may not be able to use your computer for anything else while it is running.

	nice +20 rins.pl | tee -a my_log_file &




--------------------------------------------------
### Some continued confusion


blat finds nothing with chopped reads.  Why?  I would expect it to be easier to find matches for smaller pieces?  For the moment, I skip the chopping.

blatSrc35 is the latest, but blatSrc34 is what was used in RINS development.  HUGE difference, but 34 is 5 YEARS OLDER!  In software terms, pyramid times!

#	users.soe.ucsc.edu/~kent/src/
#	[ ]	blatSrc34.zip	20-Apr-2007 16:40 	2.0M
#	[ ]	blatSrc35.zip	13-Aug-2012 20:07 	1.8M


From the rins.pl script ...
  system "$compress_script blat_out_candidate_leftlane.fa $compress_ratio_thrd > compress_leftlane.names";
  system "$compress_script blat_out_candidate_leftlane.fa $compress_ratio_thrd > compress_rightlane.names";
The second line creates compress_rightlane.names from the blat_out_candidate_leftlane.fa file???  Is this correct?  I would think not, but perhaps it is irrelevant if the contents are the same.  Of course, if they are the same, why is there a left and a right????  This needs resolved before any substantial science can be done.  For the moment, I have "corrected" this to use left for left and right for right.
This is a typo and has been fixed in some versions of the script.







--------------------------------------------------
Some data would be nice ....


http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.2bit

twoBitToFa hg19.2bit hg19.fa

#	generate ebwt files
bowtie-build hg19.fa hg19



Get SRR073726.fastq for use in the test_cahpv test...

ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA026/SRA026688/SRX031910/SRR073726.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA026/SRA026688/SRX031910/SRR073726_1.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_database/dra/fastq/SRA026/SRA026688/SRX031910/SRR073726_2.fastq.bz2

http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR073/SRR073726/SRR073726_1.fastq.gz
http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR073/SRR073726/SRR073726_2.fastq.gz

--------------------------------------------------