Curto - Kristen's Isolates

Need to find Kristen's Curtobacterium isolates. Jen sent spreadsheet with isolates that have been deep frozen "~/Documents/Research/m.curtobacterium/bacterial-strains/bacterial-cultures.xlsx"

From Lab Intranet, strains to consider (n=9):

109
115
136
145
163
171
183*
190*
213**

*originally classified as Okibacterium in Kristen's spreadsheet
**not listed in spreadsheet

Kristen says the spreadsheet in outdated but Intranet should be correct.

Streaked strains on to new LB plates and, eventually, sequence 16S to verify that they are Curtobacterium

BACE - Culture litter

1. Took Boston litter and ran through autoclaved sieves (2000 um -> 250 um -> 25 um)
1. ***NEXT TIME - GRIND UP LITTER BEFOREHAND
2. Scraped water and residue in bottom two filters into 50mL conicals
3. Add litter residue to vacuum filter (no vacuum attached) with 100 um filter
4. Rinse conicals 2x with autoclaved DI H2O
5. Total of 200 mL of litter residue and add to new set of 50mL conicals
6. Dilute each tube to 1x, 1:100, 1:1000, 1:10000
7. Plate 100 uL of each dilution on media plates:
1. For this experiment, plated on both Loma Ridge Media and BACE Media

EMP Biom Files Pt. V

Still working on open source files. not really sure how to access array data from hdf5 files

However, wrote short code to organize the 2 OTUs from the closed_ref_emp_table to combine with merged mapping file:

import os
import csv

mydir = os.path.expanduser("~/Desktop/alexs-stuff/EMP/")


in_file = mydir + "EMP_10k_merged_mapping_final.txt" #master mapping file

#need txt file with sample ids that had curto hits
wanted_file = mydir + "EMPclosed/sample-ids-curto.txt" 

out_file = mydir + "EMPclosed/curto-samples.csv"

wanted = set()

with open(wanted_file) as f:
for line in f:
line = line.strip()
if line != "":
wanted.add(line)

count = 0

with open(in_file, "rb") as tsvin, open(out_file, "wb") as csvout:
tsvin = csv.reader(tsvin, delimiter = '\t')
csvout = csv.writer(csvout)

for row in tsvin:

if row[0] in wanted:
count = count + 1
csvout.writerows([row])


print "Converted %i records" % count

Output looks like this in excel after some editing:
Total samples: 136 with 111 in the merged mapping file

EMP Biom Files Pt. IV

Got in touch with Daniel MacDonald from the Knight Lab:

Sent him the full_emp biom file and he said it is fine but takes about ~30GB to parse (really prohibitive). Converted the open reference biom file into hdf5 format:

ftp://thebeast.colorado.edu/pub/full_emp_table_w_tax.hdf5

Wrote the following code. Only outputs one column (OTUs), but did confirm that curt OTUs are present in the file

import os

import h5py

mydir = os.path.expanduser("~/Desktop/alexs-stuff/")

in_file = mydir + "EMP/EMPopen/full_emp_table_hdf5.h5"

wanted_file = mydir + "EMP/greengenes-curto-only.txt"

out_file = mydir + "EMP/emp-curto-only.txt"

wanted = set()

with open(wanted_file) as f:

for line in f:

line = line.strip()

if line != "":

wanted.add(line)

hdf5_file = h5py.File(in_file, "r")

count = 0

with open(out_file, "w") as h:

for keys in hdf5_file["observation"]["ids"]:

if keys in wanted:

count = count + 1

h.write(keys + "\n")

print "Converted %i records" % count

hdf5_file.close()

EMP Biom Files Pt. III

Got in touch with Sean Gibbons and he was able to forward some code:
https://github.com/klocey/rare-bio/blob/master/tools/ConvertBiom/ConvertBiom.py
***script scans through giant biom file in smaller pieces, rather than loading entire file into memory

Output is a sparse abundance matrix
each row is: OTU, site, number of reads

Example (first 10 lines from open reference biom file):
0 0 7.0
0 1 10.0
0 2 13.0
0 3 7.0
0 4 2.0
0 5 3.0
0 6 3.0
0 7 3.0
0 158 320.0
0 159 32.0

Not really sure how to interpret data

EMP Biom Files Pt. II

***ALL DONE ON THE MAC IN THE LAB***

Need to make biom file into classic format to pull out Curto files
Should look something like:

Sample

Taxonomy

OTU 1 

OTU 2

…

OTU n

Biom Convert

biom convert
-i full_emp_table_w_tax_closedref.biom
-o full_emp_closedref_taxonomy.txt
--biom-to-classic-table
--header-key taxonomy 

Generated .txt file but too large to export into Excel - ERROR - not enough memory

Need to breakdown master .biom file:

split_otu_table_by_taxonomy.py
-i full_emp_table_w_tax_closedref.biom
-L 3 #level3 taxonomic split (class level)
-o ./L3/

Need to split further - maybe at L5:

split_otu_table_by_taxonomy.py
-i full_emp_table_w_tax_closedref.biom
-L 5 #level5 taxonomic split (family level)
-o ./L5/

Problem: only 2 Curto OTUs present in outputted .biom file. Asked Sean Gibbons what he thought:
"Yep, any of the OTUs with 'New' in the name did not hit the reference database, so you won't find them in the closed ref table. Number-only labels are Greengenes IDs, and those should all be in the closed ref table."

So - try again with open reference database? crashed last time when I ran previous code on the open reference biom file (see Pt I)
RESULT: SystemError: Negative size passed to PyString_FromStringAndSize

^Probably due to too large of input file
full_emp_table_w_tax.biom is 2.63 GB - probably crashes as a security measure

EMP Biom Files

Not too sure what to do with .biom files from S.Gibbons

Two files:

full_emp_table_w_tax_closedref.biom
full_emp_table_w_tax.biom

Need to figure out how to pull Curtobacterium metadata from above master files

***Neither attempt has been able to utilize the open reference .biom file - ERROR***

Attempt 1:

• QIIME - filter_samples_from_otu_table.py
• Under "List-based Filtering":
• -i full_emp_table_w_tax_closedref.biom
• --sample_id_fp curto-only.txt
• RESULT: nothing
• PROBLEM: curto-only.txt contains OTUs, not individual samples

Attempt 2:

• QIIME - filter_otus_from_otus_table.py
• Use feature to extract Curtobacterium OTUs
• -i full_emp_table_w_tax_closedref.biom
• -e curto.only.txt #this excludes samples from new .biom
• --negate_ids_to_exclude 
• RESULT: generates new .biom file
• PROBLEM: I think it only contains a handful of OTUs, or not working?
• Summary table from .biom
• Num samples: 15481
• Num observations: 2
• Total count: 197
• Table density (fraction of non-zero values): 0.005
• Compared to Summary table from master .biom
• Num samples: 15481
• Num observations: 69444
• Total count: 654448644
• Table density (fraction of non-zero values): 0.016
• Generated by "biom summarize-table" function 

EMP Update

I was able to figure out which OTUs from the rep_set file were Curtobacterium:

• Searched taxonomic assignment file from S.Gibbons for "Microbacteriaceae" n=2713

searchfile = open("rep_set_tax_assignments.txt", "r")
for line in searchfile:
    if "f__Microbacteriaceae" in line: print line
searchfile.close()

• Created a smaller fasta file by pulling out Microbacteriaceae sequences from giant 'rep_set.fna' file from S.Gibbons

from Bio import SeqIO
fasta_file = "rep_set.fna" #input fasta file
wanted_file = "microbacteriaceae-only.txt" #input interesting sequence IDs, one per line
result_file = "microbacteriaceae-only.fasta" #output fasta file
wanted = set()
with open(wanted_file) as f:
    for line in f:
        line = line.strip()
        if line != "":
            wanted.add(line)
fasta_sequences = SeqIO.parse(open(fasta_file),'fasta')
count = 0
with open(result_file, "w") as f:
    for seq in fasta_sequences:
        if seq.id in wanted:
         count = count + 1
         SeqIO.write([seq], f, "fasta")

print "Coverted %i records" % count

• QIIME - assign_taxonomy.py on new 'microbacteriaceae-only.fasta' 
• Aligned with GreenGenes core set (same reference as GenBank protocol)
• Performed above procedure to generate 'curtobacterium-only.fasta' n=53

BACE - Leaf Litter

Jen contacted Jeff Dukes at Purdue to get leaf litter from their grassland site in Boston.

Hopefully, this will be shipped next week (11/17).
***Claudia sent them a box on 11/13/14***

Need to find the leaf litter media from lab intranet (Kristen could help).

Found the following:

-       Grind litter for 30 seconds to break into smaller chunks. Continue until you have ~200ml of ground litter.

-       Add litter and 1 liter of DI water to large flask. Cover top with foil.

-       Place on stir plate for 24 hours.

-       Allow litter to settle for 24-48 hours.

-       Decant liquid (siphon or scoop) into clean flask (you should have ~800 ml of media).

-       Filter media through 100 μm, 8 μm, 3 μm, and 0.8 μm membranes (I use larger sizes first to decrease the use of expensive 0.8 μm filters).

-       Transfer media into autoclavable jug. Add 15g agar and fill with DI water until final volume reached 1 litter.

-       Autoclave (Liquid cycle)

Earth Microbiome (EMP)

EMP has been down for months. Trying to get access to their databases.
Heard from EMP: rebuilding database. any day now it will be up and running

Got access to the EMP files from Jack Gilbert and Sean Gibbons

• open reference OTU table (.biom)
• closed reference OTU table (.biom)
• rep sequences (.fna)
• phylogenetic tree (.tre)
• metadata file (.txt)
• taxonomic assignments (.txt) - from emp_10k_rdp