Development of a workflow system for the CARMEN Neuroscience Portal
Colin Ingram (Newcastle university), Mark Jessop (University of York), Michael Weeks (University of York ), Jim Austin (University of York), Martyn Fletcher (University of York )
CARMEN has relatively specific workflow requirements due to its cloud execution model and the use of the it's NDF data format (www.carmen.org.uk/standards/CarmenDataSpecs.pdf). Hence, although we evaluated current workflow tools, such as Taverna (www.taverna.org.uk/) and E-Science Central (www.esciencecentral.co.uk/), it was found that neither met the functional requirements and a bespoke workflow service was developed. The CARMEN Workflow Tool is Java-based and designed to make use of CARMEN Services and NDF. The workflow tool supports both data and control flow, and allows parallel execution of services. Using a Service Invocation API to invoke CARMEN services simplifies the workflow infrastructure substantially. This API allows the workflow engine to make use of our dynamic service deployment and execution system, achieving scalable heterogeneous distributed processing. The complete workflow tool consists of a graphical design tool, a workflow engine, and access to a library of CARMEN services and common workflow tasks.
The poster will provide an overview of the main design considerations for the workflow system and provide an overview of its use within CARMEN. We provide details of the workflow execution engine and describe the XML scripting approach that has been developed to control the workflow orchestration. The workflow description XML schema follows a similar form to myGrid’s SCUFL (Simple Conceptual Unified Flow Language - www.mygrid.org.uk/dev/wiki/display/developer/SCUFL2) script but modified to suit our service and data description formats. Example workflows for neurophysiology will be presented, demonstrating the flexibility of the workflow tool to support complex analysis pipelines.