Queues and workers

Each worker below is a separate process consuming its own queue. Because every worker runs one job at a time, the number waiting is the number of jobs ahead of a new submission.

Scheduled housekeeping

Publishes the sweep that fails prediction jobs whose worker died without reporting anything.

Scheduled Last swept 98s ago, every 300s; 0 abandoned job(s) failed on that run.

Platform usage

Aggregate counts only. Access tokens are counted, never stored or shown here — there is nothing on this page that identifies a user or a submission.

• Last 7 days
• Last 30 days
• Historically

Reading these numbers. Online predictors counts submissions, which is what users asked for whether or not it succeeded. The legacy APIs and ACPYPESERV count completed predictions, because those are the only rows carrying both a token and a timestamp — a failed legacy job stores no result. The two are therefore not directly comparable. "Distinct tokens" is summed across platforms, so one person using both services counts once for each.

Time limits and retries

Every prediction gets a fixed budget. The limit is the same for all tools and both queue systems, so a heavy job cannot occupy a worker indefinitely while other users wait.

Soft limit — 890 seconds

The prediction is interrupted and asked to stop. It gets the chance to clean up its temporary files, and the failure is recorded against your job with the reason.

Hard limit — 900 seconds

Ten seconds later the worker process is terminated outright. This is the backstop for a job wedged somewhere it cannot be interrupted gracefully.

ACPYPE jobs — 900 seconds

ACPYPESERV runs on a separate queue system with the same budget.

Attempts — up to 3

A job interrupted by a transient problem — a dropped database or broker connection — is retried automatically with a growing delay. A job that ran out of time is not retried: a second attempt would only spend the same budget again, so you are told immediately instead of waiting three times as long for the same answer.

Worker recycling — every 100 jobs

Each worker restarts after this many jobs. The prediction libraries hold on to memory, and a periodic restart keeps a long-lived worker from growing without bound. It does not interrupt a job in progress.

If your job exceeds the limit

The most common cause is input size: a multiple sequence alignment with many sequences, or a very long sequence. Try splitting the input and submitting the parts separately — several smaller jobs will finish where one large one will not. The result page always shows the recorded reason for a failure.