Configuring the config files

Phylociraptor uses two main config files for setting analysis options:

• The data/config.yaml file, which controls settings of the running behavior of phylociraptor.

• The data/sample.csv file which contains information about the used samples.

Additionally there is a cluster config file, which control the behavior on HPC clusters with different job submission systems. We provide individual cluster config file for different submission systems.

The config.yaml file

When you download phylociraptor the file is called config.yaml.template , it is located in the data/ directory and you have to first rename it to config.yaml. As you can see from the file extension, the file is in YAML format. Here is how the complete file looks like (without comments). We also added line numbers for this example, so that is easier to refer to different settings in this documentation.

species: data/plath.csv
busco:
		set: "metazoa_odb9"
		threads: 8
		augustus_species: fly
		additional_parameters: ""

filtering:
		dupseq: "persample"
		cutoff: 0.5
		minsp: 3
		seq_type: "aa"
		min_parsimony_sites: 10

alignment:
		method: mafft
		threads: 8
		parameters: --quiet --auto

trimming: 
		method: trimal 
		parameters: -gappyout

tree:
		method: "iqtree raxml"

speciestree:
		method: "astral"

iqtree:
		threads: 16 
		bootstrap: 1000
		model: MFP
		additional_params: ""
		maxmem: "" 
raxmlng:
		threads: 16
		bootstrap: 1000
		additional_params: ""

Line 1: Here we specify the samples configuration file path (We will talk about it later). It is important that you give a relative path here. It is best to place the file in your data directory inside your phylociraptor folder.

Lines 2-6: Here you can specify the settings for the BUSCO runs (which will run individually for each genome specified in the file in line 1). Line 3 refers to the BUSCO set path. Your you should give the web address to the BUSCO set. You can look here to see which sets are available. Currently phylociraptor uses BUSCO v3, so make sure you specify the correct download link in the config file. Line 4 sets the number of threads which should be used for each BUSCO run. This will depend on the type and power of your computer environment. Increasing the number of threads can speed up the BUSCO analyses. Usually you can leave this at the standard setting of 8 threads.

Line 5 specifies which models Augustus should uses to predict genes as part of the BUSCO step. Augustus offers many different pretrained species models and it is generally a good idea to use a species which is closely related to the species you want to analyse. It will improve BUSCO performance and results in more (and more accurate) genes available for phylogenomic reconstruction.

Line 6 Here you can specify additional parameters which will be passed on to BUSCO. Check the BUSCO website and manual for possible options. We have not tested all possible parameters and we can not guarantee that all of them will work with phylociraptor.

Line 7-12: This sections controls the filtering behaviour of Orthology inference and alignments. Filtering will take place at different steps in the analysis workflow (-m filter-orthology and -m filter-align). This allows to test the effect of different filtering parameters on the dataset. We will go into additional detail later.

Note

cutoff should be between 0 (no filtering) and 1 (only keep samples with 100% BUSCO completeness). Both extremes are probably unrealistic.

minsp should be between 3 and n (n is the number of samples in your analysis).

Line 8: Occasionaly BUSCO will place more than one sequences in files for single-copy BUSCO genes. This flag allows to set the behaviour how phylociraptor handles these cases. Possible options are: “persample” which will remove the duplicated sequences only for the species in which it occurs. This means that for the affected sample, this gene will not be used. “pergene” will discard the whole gene for all samples. This gene will thus not be used for phylogenomic reconstruction at all.

Line 9: The cutoff value sets the minimum proportion of BUSCO completeness (complete single-copy BUSCO genes) a sample needs to have to be included in downstream analysis. This flag allows to exclude samples with poor BUSCO performance eg. due to a suboptimal assembly. The default is 0.5 which will include samples which have at least 50% BUSCO completeness. Your should provide a value between 0 (no-cutoff) to 1 (only genomes with 100% BUSCO completness). Keep in mind that 1 is very unrealistic for most genomes!

Line 10: Not all BUSCO genes will be detected with equal efficiency, thus your alignments for different BUSCO genes will contain a varying number of genes. minsp lets you set the minimum number of sequences need to be in an alignment for it to be included in the phylogenomic reconstruction. The default is 3, which means that at least 3 sequences need to be present for a gene to be included in the analysis. You should use a value between 3 and the total number of samples you have.

Line 11: seq_type lets you specify whether you want to reconstruct trees based on nucleotide “nu” or amino-acid “aa” data. The default is “aa”.

Line 12: With min_parsimony_sites you can set how many parsimony informative sites an alignment needs to have to be included in the analysis. The default value is 10.

Line 13-16: In this section you can change settings about the alignment step.

Line 14: You can specify the alignment method here. Currently we only support mafft but we plan support for additional aligners in the future.

Line 15: Specify the number of threads for the alignment step. As for BUSCO this depends on your computer environment. The alignment step is not the most time-consuming step, so a smaller number of threads should not impact performance too much. The default value is 8, which is usually fine.

Line 16: Here you can pass additional parameters to mafft eg. alignment methods. By default phylociraptor will tell mafft to determine the best alignment strategy automatically (–auto). For additional options see mafft website.

Line 17-19: Here you can set how to control the alignment trimming step, to filter out ambiguous position, misalignments and more.

Line 18: This lets you control the trimming method. Currently trimal and alicut are the supported options here. To familiarize how these programs work it is a good idea to study the respective websites and manuals.

Line 19: Because trimming can be done in many different ways, you can specify different trimming parameters here which will be passed on to the specified trimming method.

Line 20-33 This part of the config file lets you specify the (species)tree reconstruction methods and set different additional parameters if you wish.

Line 21 Specify the maximum-likelihood tree reconstruction methods here. These should be under quotes and seperated by a space. Currently we support iqtree and raxml-ng (specify raxml in the config file). We plan to include additionla methods in the future.

Line 23 Here you can specify the species tree reconstruction method. Currently we only support astral but we plan to include additional methods in the future.

Line 24-29 These settings change the behaviour of iqtree.

Line 25: Lets you change the number of threads for your iqtree run. The way this works is a bit different from other threading settings. For iqtree the number of threads will rever to the maximum number of threads. When phylociraptor is run an a HPC cluster with a job sheduler it will let the sheduler know how many threads should be requested for the iqtree job. Internally iqtree will then determine the optimal number of threads from 1 to number of specified threads (using -nt AUTO -ntmax {threads}).

Line 26: Lets you specify the number of bootstrap replicates. This will use the Ultra-Fast bootstraping (-bb) option of iqtree.

Line 27: Here you can specify the substitution model used by iqtree. The default setting is MFP which will determine the best substitution model for each gene partition as part of the iqtree run. Of you can also specify a specific model here as well, although it is very unrealistic to assume that the same model is optimal for all genes. In case you ran phylociraptor -m modeltest this setting will be ignored, since the best model for each genes was already inferred. To optimize runtimes we recommend to identify the best substitution models with phylociraptor -m modeltest before running -m tree. The modeltesting step is highly parallelized and probably a lot faster than model selection as part of the tree reconstruction.

Line 28: You can also specify additional parameters for iqtree here. Keep in mind however that iqtree offers a myriad of different options and applications, most of which have not been tested with phylociraptor.

Line 29: We provide an extra option to limit the amount of memory iqtree can use. This can be necessary for very large datasets on HPC clusters.

Line 31: Specify the number of threads raxml should use here.

Line 32: Specify the number of bootstrap replicates raxml will calculate here.

Line 33: Here you can pass additional arguments to raxml. Similar to iqtree, raxml offers many different options. We have not tested them all for compatibility with phylociraptor.

The samples CSV file

This is the second most important file phylociraptor needs to run and probably the most important file for you. In this file you specify which samples should be analyzed and whether they should be downloaded automatically or if you provide them as local files. Here is an example for a sample file which is part of phylociraptor when you clone it from github.

species,Organism_Groups,Size(Mb),Chromosomes,Organelles,Plasmids,Assemblies,web_local
"Ascobolus_immersus","Eukaryota;Fungi;Ascomycetes",59.5299,0,0,0,1,web
"Ascodesmis_nigricans","Eukaryota;Fungi;Ascomycetes",27.3852,0,0,0,1,web
"Cerataphis_brasiliensis_yeast-like_symbiont","Eukaryota;Fungi;Ascomycetes",25.4655,0,0,0,1,web
"Choiromyces_venosus","Eukaryota;Fungi;Ascomycetes",126.035,0,0,0,1,web
"Endocalyx_cinctus","Eukaryota;Fungi;Ascomycetes",44.7739,0,0,0,1,web
"Margaritispora_aquatica","Eukaryota;Fungi;Ascomycetes",42.5173,0,0,0,1,web
"Morchella_conica","Eukaryota;Fungi;Ascomycetes",52.4255,0,0,0,2,web
"Neolecta_irregularis","Eukaryota;Fungi;Ascomycetes",14.1826,0,0,0,1,web
"Nilaparvata_lugens_yeast-like_symbiont","Eukaryota;Fungi;Ascomycetes",26.8096,0,0,0,1,web
"Pneumocystis_carinii","Eukaryota;Fungi;Ascomycetes",7.66146,0,0,0,1,web
"Pneumocystis_jirovecii","Eukaryota;Fungi;Ascomycetes",8.39624,0,0,0,3,web
"Protomyces_sp._C29","Eukaryota;Fungi;Ascomycetes",11.928,0,0,0,1,web
"Sclerotium_cepivorum","Eukaryota;Fungi;Ascomycetes",56.335,0,0,0,1,web
"Taxomyces_andreanae","Eukaryota;Fungi;Ascomycetes",43.1525,0,0,0,1,web
"Terfezia_boudieri","Eukaryota;Fungi;Ascomycetes",63.2346,0,0,0,1,web
"Amphirosellinia_nigrospora","Eukaryota;Fungi;Ascomycetes",48.1778,0,0,0,1,web

This file is very similar to the TSV file you can download from the NCBI Genome Browser . The reason for this is to make it easy to use files downloaded from the Genome Browser with phylociraptor. To make a TSV file you will to make only minimal changes:

1. Make sure the header line is compatible with phylocirpator:

This is how it should look like:

species,Organism_Groups,Size(Mb),Chromosomes,Organelles,Plasmids,Assemblies,web_local

2. Add information to the last column to let phylocirpator know if a genomes should be downloaded web or is provided locally local.

If you have just downloaded a file from NCBI and you want to reformat the whole file to make it compatible with phylocirpator you can run this command. Make sure to check you header line after this again!

perl -p00e 's/\n/,web\n/g' genomes.csv > genomes_added_web.csv

Now you can add your local files to the file at the end if you want. As said above, most of the information in the file is not necessary for phylociraptor, however to maintain the structure of the file you have to make sure to currecntly provide the fild seperators. This is how a line for a locally provided species should look like:

Gouania hofrichteri,,,,,,,data/assemblies/Gouania_hofrichteri_4616STDY8352653_CONSENSUS_filtered20.fasta

As you can see the last column contains the path to the assembly which should be used for this species. phylociraptor will realize that this is a locally provided file and not try to download it. This path has to be relative to the working directory of phylociraptor. To ensure that there are no problems with the paths to locally provided files it is a good idea to place the assemblies inside the data/assemblies directory.

HPC Cluster config files

phylociraptor can be easily scaled to run on large HPC clusters. This has the benefit that many individual tasks can run in parallel which can drastically decrease overall runtime of the analysis. HPC clusters are usually very different with many individual (cluster specific) settings. Usually, individual jobs on HPC systems are managed by batch job-sheduling software which assigns resources (memory, CPUs and runtime) and starts jobs. Many different job schedulers exist.

We tested phylociraptor on clusters running SLURM , SGE and TORQUE.

When used on HPC clusters phylociraptor will automatically create and submit individual jobs for each task (there is no need to use batch submission files and commands such as sbatch or qsub). To make this work, phylociraptor needs to have some specific information on the underlying cluster. This information is provided through cluster configuration files in YAML format.

Phylociraptor provides cluster configuration files templates for SLURM, SGE and TORQUE clusters.

Note

Cluster configuration files have to be altered to fit your own computer infrastructure. Usually only minor changes have to be made, most settings should work out of the box.

Adjusting the SLURM cluster config template

Before you make any changes to the file, it is a good idea to make a copy:

$ cp data/cluster-config-SLURM.yaml.template data/cluster-config-SLURM.yaml

Here are the first 30 lines of the config file for a SLURM cluster.

__default__:
   time: "72:00:00"
   ntasks: 1
   job-name: default
   hint: nomultithread
   mem: 4G
   partition: mem_0096 
   qos: mem_0096
   output: $(pwd)/log/slurm-%j.out
   error: $(pwd)/log/slurm-%j.err
busco:
   job-name: BUSCO 
orthology:
   job-name: ORTHOLOGY
filter_orthology:
   job-name: FILTORTH
filter_alignments:
   job-name: filtalgn
part_filter_align:
   job-name: FAL
align_trim:
   job-name: alntri
filter_align:
   job-name: FAL
get_filter_statistics:
   job-name: getfilstat 
run_busco:
   job-name: rBUSCO
   mem: 32G

Typically it will only be necessary to change the highlighted lines in the example above. QOS (Quality of service) and partition are usually cluster specific and need to be changed. The walltime limit time: "72:00:00" will also depend on your cluster environment.

Make sure to check your cluster’s documentation to find the correct values.

Other values (like mem, ntasks, etc. ) don’t have to be changed. The provided values have been tested with differently sized datasets on different clusters and should work for the majority of datasets.