Discriminating tools helps potential users of the tool or users of its outputs make informed decision about whether the tool and/or its outputs are fit for their purposes. The following are helpful concepts the RC uses for discriminating model types:
- Tool-Grade (using adaptation of NASA’s Technological Readiness Levels)
- Report Cards
What exactly is the tool itself? Tools can be defined as1:
a: something (such as an instrument or apparatus) used in performing an operation or necessary in the practice of a vocation or profession. a scholar’s books are his tools
b: an element of a computer program (such as a graphics application) that activates and controls a particular function. a drawing tool
Here, we are referring to tools as a computer program (similar to b), that takes input data, operates on it and produces output(s). We do not consider tools to be the many individual commands or functions within a specific tool (some might call such tools a toolset, or toolbox). However, we recognize that for many audiences and end-users, the output data alone is itself a “tool” to inform practice or decision making (e.g. BRAT). However, in those instances the user might be more effectively consuming the outputs of a tool (or model, similar to definition a above) via something like the Riverscapes Viewer or our Riverscapes Warehouse .
RCtools are deployed to users thorugh a variety of interfaces:
- : ArcGIS AddIn ArcGIS AddIn Toolbar in ArcGIS
- : ArcPy Toolbox in ArcGIS
- QGIS Plugin = Toolbar in QGIS
- : CLI = Command Line Interface
- : GUI = Graphical User Interface
- : Web = interactive web site
- : API = Application Programming Interface
- : App = Mobile App
Most tools have just one deployment interface, some have multiple.
We classify the grade of our tools along a continuum of growth from innovative research ideas, through to operational tools in development that (with a little love and patience) can be run by someone other than the developer, on through to more broadly deployable professional tools that are robust and usable by any user in very different settings. Tools that have the potential to be efficiently run with greater repetition and/or across greater spatial extents or at greater resolution may be upgraded to production-grade tools whereas those that can then be deployed to much broader audiences in the most accessible platforms (e.g. Mobile Aps, Web Aps) and with ease of sharing maybe ultimately be launched as commercial-grade tools.
Our RC Technical Committee ranks a tool’s grade status using the following criteria:
|Tool Status||Badge||Vetted in
|TR3||Proof of Concept||
None or Not Applicable: • Minimal or In Progress: • Functional: • Fully Developed:
NOTE - The RC does not track concepts or proof of concept tools in its listing.
Tool Grades Explained
Below we elaborate definitions for each tool-grade.
Concept-Grade is the necessary pre-cursor stage to development of any tool, in which the ideas are articulated for what the tool does, what its inputs are, what its outputs are and who its audience might be.
Research begins with ideas, and we formalize those ideas into hypotheses and concepts. Before we make an actual “tool”, we might articulate a concept, sketch out how it might work (a conceptual model or diagram), or even write some pseudo code. Concepts are typically what gets pitched or proposed for research proposals, and sometimes in agenda-setting or state-of-the-science papers.
RC does not track “tools” before they exist, but the idea of a concept-grade is helpful and explit from NASA’s original technological readniess levels.
Proof-of-Concept-Grade is a stage in which some preliminary form of a tool has been built, and the tool has been demonstrated to “work” by producing reasonalbe outputs for an example dataset.
The vast majority of research papers published in the riverscape sciences that use a tool the author(s) developed are using proof-of-concept-grade tools. That is to say, the results produced are scientifically defensible, the methods described are also defensible, and the investigators executed those methods with tools (e.g. source code and scripts) or workflows that they authored and developed. However, another investigator wishing to repeat the work or apply the workflow to their own dataset would be unlikely able to use the tool of the authors (even if it was OpenSource) without significant refactoring or manipulation.
Research-Grade tools are those that are supported by peer-reviewed science, have formally been packaged up, and maybe even made open-source, but it is a tool that has really only been applied and validated within the researcher’s own lab and/or study-sites.
Research-Grade tools are typically those that researchers have built to make their life easier, but the idea of the tool might be aluring to other potential users. Such tools might be associated with a methodological contribution, that other researchrs and users would like to apply or replicate, but for them to actually get the tool to work on their machine, with their data might be difficult to impossible. These tools are sort of “buyer beware”, in that they are not really made (yet) for other users (and in the case of open-source, the “buyers” haven’t paid anything… “you get what you pay for”).
If the RC is successful, we will make it easier. They are for more researchers to lift their tools from a proof of concept to research-grade. Research-grade tools are the minimum level required to achieve some degree of F-A-I-R-ness. However, the outputs of such tools can reach broader audiences more effectively, if these tools are made to output Produces Riverscapes Projects that can be easily consumed from a cloud-based Warehouse and explored with the RiversCapes Viewer..
Research-grade tools straddle the “research” and “development” TRLs, where some degree of validation of the outputs and its potential utility has been demonstrated. However, the tool’s application or utility for users outside the developers research group is limited.
Operational-Grade is defined as.
Operational-grade tools are firmly in the “development” TRL realm and are based on validated and demonstrated techniques.
Professional-Grade is defined as.
Production-Grade is defined as.
Commercial-Grade is defined as.
Technological Readiness Levels
These tool-grade ideas are based on the concept of Technological Readiness Level (TRLs), as originally developed by NASA. The TRLs provide a way to discriminate between concepts and products that are in research phases, in development phases, or ready for deployment to broader audiences or market. TRLs are illustrated below (from TWI-global and formally defined by the European Union: