A Protein Secondary Structure Prediction Server
JPred RESTful API (v.1.5)
(command line access to JPred jobs submission, monitoring, and results retrieval)
Table of contents
- Introduction
- Flexibility
- Programming examples
- How to start
- Additional functions
- Checking for updates
- Checking job status just once
- Checking for quota (number of jobs allowed to submit on a given day)
- Checking time (in sec) left before quota re-fill
- Mass-submission, scheduler programs (submition + status check + retrieval of results automation!)
- Help, making it better
- Updates
Introduction
The RESTful API allows JPred users to submit jobs from the command line (Unix/Linux shell, Terminal window in Mac/Windows OS). Here are some examples:
Single sequence submission:
perl jpredapi submit mode=single format=fasta email=name@domain.com file=single_fasta.example name=TestJob3 or: perl jpredapi submit mode=single format=raw email=name@domain.com name=TestJob1 seq=MQVWPIEGIKKFETLSYLPPLTVEDLLKQIEYLLRSKWVPCLEFSKVGFVYRENHRSPGYYDGRYWTMWKLPMFGCTDATQVLKELEEAKKAYPDAFVRIIGFDNVRQVQLISFIAYKPPGC
Multiple alignment as input submission:
perl jpredapi submit mode=msa format=fasta email=name@domain.com file=msa_fasta.example name=TestJob4 perl jpredapi submit mode=msa format=msf email=name@domain.com file=msa_msf.example name=TestJob5 perl jpredapi submit mode=msa format=blc email=name@domain.com file=msa_blc.example name=TestJob6
Batch job submission (Please note that batch mode is kept for REST API for full compatibility with regular JPred submissions. This implies that the batch submissions are limited to 200 jobs per submission, and to 1000 per day - as is the case for regular, through the JPred web site submission. To use JPred REST API in full capacity - please use individual job submissions (as shown above) wrapped in your code in loops - allowing you submit beyond any set threshold. Current record is 10,000+ sequence based jobs per day and 40,000+ MSA-based jobs in a day. Also instead of writing your submission and results retrieval script yourself - consider using one of the provided for download by us. Here is the link.):
perl jpredapi submit mode=batch format=fasta email=name@domain.com file=batch_fasta.example name=TestJob7
Check submission_all_types_example.csh script file provided for examples.
Check *.example files for example of inputs in all valid combinations of mode/format: batch_fasta.example, msa_blc.example, msa_fasta.example, msa_msf.example, single_fasta.example, single_raw.example
Valid combinations of mode/format are: mode=single format=raw, mode=single format=fasta, mode=msa format=fasta, mode=msa format=msf, mode=msa format=blc, mode=batch format=fasta
Note that the JPred API supports all the types of jobs that the JPred web-server does (single sequence in Raw/FASTA, batch job (multiple sequences in FASTA format) and multiple sequence alignment (MSF, BLC or FASTA formats). Examples of supported formats are given on formats page and other details are described on F.A.Q. and Help pages.
Back to the top of the page...
Flexibility
The script is flexible - input file names and their suffixes are free; the order of parameters is free, many have defaults and are optional. The script also checks the validity of the input and reports on problems. It also returns any errors/warnings/messages the JPred server produces while the job is running.
Back to the top of the page...
Programming examples
The JPred RESTful API allows the user to write (shell) programs which simplify submission, monitoring, and results retrieval. We will provide more advanced scripts in the near future, however a simple example is:
...for submission ("silent" option supresses most of the output, leaving only essential):
#!/bin/csh perl jpredapi submit mode=single format=fasta email=name@domain.com file=InputFile1.txt name=TestJob1 silent perl jpredapi submit mode=single format=fasta email=name@domain.com file=InputFile2.txt name=TestJob2 silent perl jpredapi submit mode=single format=fasta email=name@domain.com file=InputFile3.txt name=TestJob3 silent perl jpredapi submit mode=single format=fasta email=name@domain.com file=InputFile4.txt name=TestJob4 silent perl jpredapi submit mode=single format=fasta email=name@domain.com file=InputFile5.txt name=TestJob5 silent
Output of the above script would include JPred job ids:
Created JPred job with jobid: jp_OpHCA0J Created JPred job with jobid: jp_S6uasf5 Created JPred job with jobid: jp_cmUrnkU Created JPred job with jobid: jp_4mCdrmJ Created JPred job with jobid: jp_aU4tS10
...using the job ids and the following script will automatically monitor and retrieve jobs like so:
#!/bin/csh perl jpredapi status jobid=jp_OpHCA0J getResults=yes checkEvery=300 silent & perl jpredapi status jobid=jp_S6uasf5 getResults=yes checkEvery=300 silent & perl jpredapi status jobid=jp_cmUrnkU getResults=yes checkEvery=300 silent & perl jpredapi status jobid=jp_4mCdrmJ getResults=yes checkEvery=300 silent & perl jpredapi status jobid=jp_aU4tS10 getResults=yes checkEvery=300 silent &
The above script would check status of each of the jobs once every 300 sec (5 min) and get results (downloaded into a local directory) when the job is completed. Example output:
Fri Feb 13 11:50:24 2015--->Your job is 10% complete... Fri Feb 13 11:50:24 2015--->Your job is 10% complete... Fri Feb 13 11:50:24 2015--->Your job is 10% complete... Fri Feb 13 11:50:24 2015--->Your job is 10% complete... Fri Feb 13 11:50:24 2015--->Job jp_OpHCA0J finished. Results available at the following URL: http://www.compbio.dundee.ac.uk/jpred4/results/jp_OpHCA0J/jp_OpHCA0J.results.html Job results archive is now available at: jp_OpHCA0J/jp_OpHCA0J.tar.gz Fri Feb 13 11:55:26 2015--->Job jp_cmUrnkU finished. Results available at the following URL: http://www.compbio.dundee.ac.uk/jpred4/results/jp_cmUrnkU/jp_cmUrnkU.results.html Fri Feb 13 11:55:26 2015--->Job jp_4mCdrmJ finished. Results available at the following URL: http://www.compbio.dundee.ac.uk/jpred4/results/jp_4mCdrmJ/jp_4mCdrmJ.results.html Fri Feb 13 11:55:26 2015--->Job jp_S6uasf5 finished. Results available at the following URL: http://www.compbio.dundee.ac.uk/jpred4/results/jp_S6uasf5/jp_S6uasf5.results.html Job results archive is now available at: jp_cmUrnkU/jp_cmUrnkU.tar.gz Job results archive is now available at: jp_4mCdrmJ/jp_4mCdrmJ.tar.gz Fri Feb 13 11:55:27 2015--->Job jp_aU4tS10 finished. Results available at the following URL: http://www.compbio.dundee.ac.uk/jpred4/results/jp_aU4tS10/jp_aU4tS10.results.html Job results archive is now available at: jp_aU4tS10/jp_aU4tS10.tar.gz Job results archive is now available at: jp_S6uasf5/jp_S6uasf5.tar.gz
On the first round of status checks in this example there were 4 jobs out of five 10% done with one already finished and results immediately downloaded. Eventually (in the next 5 min, second round of checks) the rest of the jobs were done and results downloaded. The script also outputs URLs to results directories if one prefers to review results online. Note that archives like 'jp_S6uasf5.tar.gz' can be unarchived using 'tar -xzf jp_S6uasf5.tar.gz' (tar is a standard archive command, which should be available on virtually any Linux/OS-X computer or through applications like 7-zip under MS-Windows).
Check one_job_retrieval.csh script file provided. One can use it to retrieve JPred job results like so: 'source one_job_retrieval.csh jobId' - where jobId is the JPred job id returned at submission (id looks like: jp_OpHCA0J).
Back to the top of the page...
How to start
Download the linked here archive, open it in the working directory ('tar xzf jpredapi.tar.gz'). The archive consists of the main client script (jpredapi) as well as some input examples and script examples.
The main scripts do not require any special libraries, so should run on virtually any computer with Perl installed. Check if the documentation shows (see next paragraph). If it doesn't - check the path to perl using 'which perl' (or ask your IT-specialist/computer-administrator), copy the path into the first line of jpredapi script (open it with any text editor). Try checking for documentation again. This should be the only (if any) edit to the script. Please contact us if more assistance is needed.
To access detailed documentation with description of the options, etc. just run the script without any parameters:
perl jpredapi
Here is the copy of what you should see:
JPred RESTful API client v.1.4 documentation.
General:
please note that jobs run through this client program will be submitted to the JPred 4 server:
http://www.compbio.dundee.ac.uk/jpred4
A tutorial about the JPred API is available at:
http://www.compbio.dundee.ac.uk/jpred4/api.shtml
Email:
Usage of your email address is optional on single sequence job submission, but if you use one - you will be
notified when the job is complete or sent some diagnostics if it crashed. Email is obligatory for batch job submission.
Results from batch submissions are sent by email and link to individual job results and to an archive containing ALL results.
If you use email for your submissions we may ocassionaly contact you about major updates to the JPred server and JPred API.
========== COMMANDS ==========
1.
To print this documentation run jpredapi without arguments like so:
'perl jpredapi'
2.
To submit a JPred job do:
'perl jpredapi submit file=filename mode=batch format=fasta email=name@domain.com name=my_test_job skipPDB=on'
or
'perl jpredapi submit mode=single format=raw email=name@domain.com name=my_test_job skipPDB=on seq=MQVWPIEGIKKFETLSYLPPLTVEDLLKQIEYLLRSKWVPCLEFSKVGFVYRENHRSPGYYDGRYWTMWKLPMFGCTDATQVLKELEEAKKAYPDAFVRIIGFDNVRQVQLISFIAYKPPGC'
where:
'file=filename' - filename is the name of a file with the job input, see example_submission_sequence.txt (file name/suffix could be any, e.g. it does not have to be something like name.fasta for FASTA format file)
'seq=AAAABBB' - instead of passing input file, for one-sequence submission you can submit jobs with sequences passed right through command line
'format=batch' - defines format. Possible values are:
- batch (multiple sequence submission, file in FASTA format)
- seq (single sequence submission in raw/FASTA format)
- or multiple sequence submission formats: msf, blc, fasta
'email=name@domain.com' - defines email address where job report will be sent (optional for all but batch submissions). We DO recommend to use email.
'name=my_test_job_number_3' - defines (optional) job name
'skipPDB=on' - possible values are: on and off. If on - PDB check will not be performed - this is default for JPred API (see JPred documentation for more details on PDB check). Since default is defined, parameter is optional.
NOTE:
order of parameters is not important, optional parameters could be left out.
NOTE:
adding the word 'silent' to the command line will suppress all but essential output, e.g.:
'perl jpredapi submit file=filename mode=batch format=fasta email=name@domain.com name=my_test_job skipPDB=on silent'
2.1
Check submission_all_types_example.csh script file provided for further examples.
2.2
Check *.example files for example of inputs in all valid combinations of mode/format: batch_fasta.example msa_blc.example msa_fasta.example msa_msf.example single_fasta.example single_raw.example
2.2.1
Valid combinations of mode/format are: mode=single format=raw, mode=single format=fasta, mode=msa format=fasta, mode=msa format=msf, mode=msa format=blc, mode=batch format=fasta
3.
To check JPred job status do:
'perl jpredapi status jobid=id [getResults=yes] [checkEvery=60] [silent]'
where:
'job=jobid' - jobid is JPred job id returned on submission
'getResults=yes' - if job successful jpredapi will download job results archive when ready (it is default, so, parameter could be left out)
'checkEvery=60' - how often to check for job status in seconds (default 60 seconds = 1 min, minimum is 10 seconds) - it is optional parameter, you may leave it out, script will take care of it
NOTE: if using 'checkEvery=once' the program will perform status check just once and exit
'silent' - adding the word 'silent' to the command line will suppress all but essential output
3.1
Check one_job_retrieval.csh script file provided. One can use it to retrieve JPred job results like so:
'source one_job_retrieval.csh jobId' - where jobId is the JPred job id returned at submission (id looks like: jp_OpHCA0J)
4.
To check how many jobs you have already submitted on a given day (out of 1000 maximum allowed jobs per user per day) do:
'perl jpredapi quota email=name@domain.com'
where:
'email=name@domain.com' - defines email address you use for job submissions
5.
To check how much time (in sec) left before more jobs are allowed from a given user (limit is 1000 jobs per user per day) do:
'perl jpredapi sectonewday'
Back to the top of the page...
Additional functions
Below we describe additional functions available in 'jpredapi' program (as in official version).
Checking for updates
Every time a user runs the jpredapi client it checks if they are running the latest version. If not, then a warning is printed suggesting to download the up to date version (link is then provided), and the code is paused for 20 seconds allowing users to react.
Checking job status just once
Checking job status with the command:
'perl jpredapi status jobid=id [getResults=yes] [checkEvery=once] [silent]'
the program will perform only one check and exit. - Note "checkEvery=once" vs. "checkEvery=60".
We added the "check only once" function on users' request. Otherwise, when using "checkEvery=N" and N is an interval in seconds (60 = 1 min, 300 = 5 min - the latter is recommended as 5 min is the median time it takes for JPred to complete a job once started) the program will perform status checks in a loop until job is finished at which stage the program will attempt to download the job results (if "getResults=yes").
Checking for quota (number of jobs allowed to submit on a given day)
The default limit of number of jobs allowed per user per day is 1000. We ask JPred users to respect the limit. - The limit is chosen based on the strength of the cluster JPred uses.
To check how many jobs one still allowed to submit on a given day do:
perl jpredapi quota email=name\@domain.com
where 'email=name\@domain.com' - defines email address one uses for job submissions.
Checking time (in sec) left before quota re-fill
To check how much time (in sec) left before more jobs are allowed from a given user (limit is 1000 jobs per user per day) do:
perl jpredapi sectonewday
Back to the top of the page...
Mass-submission, scheduler programs (2 options)
1. Simple to use (and fine tune to your needs) shell-script from JPred team
This is a combination of two shell scripts. The first one prepares inputs: single sequence FASTA-format files (this could be skipped if your inputs are already prepared). The second (main) script takes care of:
- schedule-submitting (controlled submission) all the jobs - whether 1, 100, or 100,000 (there is no limit);
- submits >>1000 jobs per user per day;
- checks (and logs) status of the jobs;
- retrieves results (all the result files JPred produces or only files of interest).
All with virtually "a push of a button". All automated. Script is deliberately written in a simple fashion (in case one would want to fine tune it), it's only about 150 lines of code. It's a shell-script - no extra requirements, no installations. The script is based on the JPred REST API.
Before downloading and using the script - please read through a few pages tutorial: direct link to PDF file available from the JPred Help page.
As with everything else JPred - if you have questions, need assistance, or have feedback to share - please contact us.
Back to the top of the page...
2. Perl based code from one of the JPred "power" users
Based on the official JPred REST API functionality and client code described above Fabian Egli wrote an advanced mass-submission/scheduler program (in Perl). His code provides a JPred user with advanced functionality for an easy submission of hundreds of JPred jobs in a sustainable manner. Sustainable here means: automatic control over how many jobs are running in parallel, which prevents JPred cluster from being overloaded, allowing other users to submit requests and obtain results with minimum waiting time (**). Fabian's program in automatic regime would check if more jobs are allowed, would check for complete/failed jobs, download selected result file types locally, etc. For full details on the code, how-to, and for download please check Fabian's GitHub repository page.
**: Note that using monitoring tools one could easily check the current load of the JPred cluster as well as recent and long-term history, which we hope will assist JPred users in scheduling (relevant to mass submissions of [multiple] thousands of jobs). From the monitoring and statistics tools we use on JPred - we know that median time a job takes is about 5 min and given multiple nodes JPred uses for its jobs - JPred is capable of going through up to hundreds of jobs per hour. - Which means that even if at submission JPred notifies you that your job is for example 1000s in line, please don't be concerned, the job will likely run at the latest within a few hours. We recommend to provide your email address at submission to allow automatic notification once results are available.
Back to the top of the page...
Help, making it better
For help - you are always welcome to contact us.
You are also welcome to contact us with comments and suggestions for improvements and added functionality.
Back to the top of the page...
Updates
24 Mar 2015: V.1.4 -> V.1.5
Implemented additional functions: "Checking time (in sec) left before quota re-fill". To check how much time (in sec) left before more jobs are allowed from a given user (limit is 1000 jobs per user per day).
17 Mar 2015: V.1.3 -> V.1.4
Implemented additional functions: "Checking job status just once" and "Checking for quota (number of jobs allowed to submit on a given day)".
17 Mar 2015: V.1.2 -> V.1.3
Significantly improved error check and reporting at the submission stage.
05 Mar 2015: V.1.1 -> V.1.2
Client script from now on features automatic version check.
Details: Every time a user runs the jpredapi client it checks if they are running the latest version. If not, then a warning is printed suggesting to download the up to date version (link is then provided), and the code is paused for 20 seconds allowing users to react.
