BACE - DNA Extraction

Followed Promega DNA Extraction Kit Protocol

Results were better than Spin Column method, but still not great for some samples.

EMP - Align OTUs

Took the rep set of sequences and pulled out Curtobacterium OTUs only (Curtobacterium were assigned by GreenGenes database)

Aligned curto only sequences with SINA

Sequences are really short (~150 bp) - see how they incorporate into sequenced data from BACE litter (align with all sequences that were a hit for Microbacteriaceae)

BioCluster

Gained access to the BioCluster

Login: ssh username@hpc.oit.uci.edu

Help: cat /data/help/cheat-sheet.txt

Guidebook on how to use the BioCluster created by Kevin Thornton:

Example to run jobs on the cluster:

Batch jobs are jobs that contain all of necessary information and instructions to run inside a script. You create a script with your favorite editor (like emacs) and then submit the script to the scheduler to run.

Some jobs can run for days, weeks, or longer so batch is the way to go for such work. Once you submit a job to the scheduler, you can log off and come back at a later time and check on the results.

Serial batch jobs are usually the simplest to use. Serial jobs run with only one core and are also the slowest since they only consume 1-core per job.

Consider the following serial job script available from the HPC demo account.

• cat ~demo/serial.sh

#!/bin/bash
#$ -N TEST
#$ -q free64
#$ -m beas

date  > out

Grid Engine Directive	What It Does
#!/bin/bash	Running shell to use ( the bash shell )
#$ -N TEST	Our Job Name is TEST. If output is produced to standard out, you will see a file name TEST.o<jobid> and TEST.e<jobid> for errors (if any occurred)
#$ -q free64	Request the free64 queue
#$ -m beas	Send you email of job status (b)egin, (e)rror, (a)bort, (s)suspend

The first line #!/bin/bash is the shell to use. Grid Engine (GE) directives start with #$. GE directives are needed in order to tell the scheduler what queue to use, how many cores to use, whether to send email or not, etc.

The last line in our serial.sh script is the program to run. In this example it is a simple date program writing the output to out file.

date > out

Now that we have a basic understanding let’s run our first serial batch job on the HPC Cluster. First create a test directory, change to the test directory, copy the demo serial.sh script to our new directory and submit the job.

From your HPC account, do the following:

$ mkdir serial-test
$ cd serial-test
$ cp ~demo/serial.sh .
$ qsub serial.sh
$ qstat -u $USER

After you submit the job (qsub), GE will respond with a job ID:

Your job 1961 ("TEST") has been submitted

and qstat will display something similar to this:

job-ID  prior   name   user     state submit/start  queue       slots

  1961 0.00000  TEST  jfarran   qw    08/16/2012                 1

The state of our job is qw queue wait (meaning the job is sitting in the queue waiting for a compute node). The core count (slots) shows as 1 (this is the default which is one core).

When we run qstat -u $USER again a few seconds later, we see:

job-ID  prior   name   user    state submit/start  queue               slots

  1961 0.50659  TEST  jfarran   r   08/16/2012    free64@compute-7-11   1

The scheduler found compute-7-11 on free64 queue available with 1 core (slots) and started our job #1961 on it. The job state changed from queue wait qw to running r.

Once you submit your job (qsub), things happen rather quickly so you may need to type qstat repeatedly and fast to see your job. Or open a new window and run: watch -d "qstat -u $USER"

Once the job completes you will get an email notification and the qstat output will be empty.

Now do an ls and you will see the following files:

out  serial.sh

The serial.sh is the batch job we submitted and file out is the output from the date program. To see the output type:

$ cat out

BACE, Curto - Phylogeny

Used ARB-Silva SINA to align sequences and ARB to generate phylogeny

***May take out sequences (see below post; specifically AB_3.17L, AB_3.19L, AB_3.27L) that do not have full 16S gene - do not fit great into phylogeny

EDIT: new phylogeny with scale generated by ARB tree generator 

BACE, Curto - Sequence Data

Received sequence data from Beckman Genomics Institute

Trimmed sequences:
   Eliminated below 5% probability and trimmed first 20 bp (length of primers)

Primers used:
Forward Primer 

AGAGTTTGATCCTGGCTCAG

Reverse Primer

AAGGAGGTGATCCAGCCGCA

Assembled de novo:

   Samples with No contig overlap - could not assemble (n = 12) - used either F or R strand for id

      AB 3.02L - gel shows blurry line at 1500 bp

      AB 3.04L - multiple bands 

      AB 3.05L - bright band at 1500 bp

      AB 3.12L - bright, smeared band at 1500 bp

      AB 3.17L - multiple bands ~1500 bp

      AB 3.19L - very faint band at 1500 bp

      AB 3.27L - no visible band

      AB 3.37L - bright, smeared band at 1500 bp

      *AB 3.04B - multiple bands

      *AB 3.09B - multiple bands

      AB 3.13B - bright band at 1500 bp

      *Curto 145 (redo) - no visible band

      *low quality read percentage - did not include 

Blast samples - blast against nr/nt database

BACE, Curto - PCR

Performed PCR:

	1 Rxn (μl)	Rxn Number	Total
		96
dH2O	11.3		1084.8
Premix F	15		1440.0
pA (50μM)	0.2		19.2
pH' (50μM)	0.2		19.2
Taq (5units/μl)	0.3		28.8
DNA	3.0

rDNA 16S Gene for 1500 bp 

Primers Used:

Forward Primer - pA -> 27f / E8F

Reverse Primer - pH' -> 1525f / E1541F

ThermoCycler:
95 for 4:00 min
*95 for 0:40 min
*55.5 for 0:30 min
*72 for 2 min
72 for 3:30 min
Hold at 4
*Repeat for 30 cycles

LOOKED REALLY GOOD!

Transfer 20 uL to new plate - READY FOR SEQ

Claudia sent out to Beckman Genomics Institute on 1/13/15

Curto - PCR

Took Kristen's Curtobacterium strains from deep freeze and plated.

Performed PCR:

	1 Rxn (μl)	Rxn Number	Total
		12
dH2O	13.3		159.6
Premix F	15		180
pA (50μM)	0.2		2.4
pH' (50μM)	0.2		2.4
Taq (5units/μl)	0.3		3.6
DNA	1

rDNA 16S Gene for 1500 bp 

Primers Used:

Forward Primer - pA -> 27f / E8F

Reverse Primer - pH' -> 1525f / E1541F

5 of the 9 strains amplified

Need to reevaluate strains that failed

EMP Biom Files Pt. VII

Went back to the hdf5 file and attempted to solve problem in MatLab (Rich helped a lot!)

Solved the array issue:

%% Load the data
a=h5info('name of hdf file');% return structured array of the hdf
% hieracrchy for reference
observdata=h5read('name of hdf file','/observation/matrix/data');
observIndices=h5read('name of hdf file','/observation/matrix/indices');
observIndptr=h5read('name of hdf file','/observation/matrix/indptr');
ids=h5read('name of hdf file','/observation/ids');

%% Get the OTU indices
% You need a cell array of strings of your desired OTUs stored as variable
% qList

for i=1:length(qList)
    qInd=find(strcmp(qList,ids));
end

%% find the data

outmat=zeros(length(qList),length(sampleIndptr));

for i=1:length(qInd)
    p=observdata(observIndptr(qInd(i)):observIndptr(qInd(i)+1));
    pI=observIndices(observIndptr(qInd(i)):observIndptr(qInd(i)+1));
    for j=1:length(pI)
        outmat(i,pI(j)+1)=p(j); %plus one to correct for matlab python coordinate changes
    end
end

Merged files and got the following with all metadata!!!!!

EMP Biom Files Pt. VI - Green Planet

Got access to the Green Planet server and am able to login.

Was informed by Chad Cantwell that QIIME may or may not be installed in the new environment (server was updated recently and it may not be running)

Contacted  Steve Hatosy (from Adam's lab) and was super helpful.
When you login to the server (through ssh), type: . /sopt/qiime/set_paths_1.4
This should set the paths to the QIIME script

Code to run:
split_otu_table_by_taxonomy.py
-i full_emp_table_w_tax.biom #open source biom file
-L 5 #split taxonomy at Family level
-o ./L5/ #file directory

>> SystemError: Negative size passed to PyString_FromStringAndSize

This is (I think) due to large size of the input file (biom file 2.63 GB). From my correspondence with people in the Knight Lab, the file should need ~30GB of memory to load.

Protocol for Cluster:

1. ssh -Y abchase@gplogin3.ps.uci.edu
2. enter login information
3. pwd #gets cluster directory 
4. exit cluster
5. from local terminal. you can upload files to the cluster:
1. scp /Users/MartinyLab/Desktop/alexs-stuff/EMP/EMPopen/full_emp_table_w_tax.biom abchase@gplogin1.ps.uci.edu:/home/abchase
6. Run QIIME code with cluster file directory

BACE - Isolate Strains

12/8/2014
Got done streaking the cultures grown on BACE media (n=39). Took no cultures from 1:10000 dilutions

12/9/2014
Got done streaking the cultures grown on Loma Ridge media (n=40). Took no cultures from 1:10000 dilutions