Developers Guide

Prerequisites

Please see the details page for prerequisites to build and test the EPICS Archiver Appliance. An installation of Tomcat is required to build successfully; this is located using the environment variable TOMCAT_HOME. Use something like

[ epicsarchiverap ]$ echo $TOMCAT_HOME
/opt/local/tomcat/latest
[ epicsarchiverap ]$ ls -l $TOMCAT_HOME/
drwxr-x---   3 mshankar cd  4096 Oct 29 18:25 bin
-rw-r-----   1 mshankar cd 19182 May  3  2019 BUILDING.txt
drwx------   3 mshankar cd   254 Jul 29 14:41 conf
drwx------   2 mshankar cd   238 May 22 15:43 conf_from_install
drwxr-xr-x+  2 mshankar cd   238 May 22 15:44 conf_original
-rw-r-----   1 mshankar cd  5407 May  3  2019 CONTRIBUTING.md
drwxr-x---   2 mshankar cd  4096 Sep 17 18:13 lib
-rw-r-----   1 mshankar cd 57092 May  3  2019 LICENSE
drwxr-x---   2 mshankar cd   193 Nov 11 16:58 logs
-rw-r-----   1 mshankar cd  2333 May  3  2019 NOTICE
-rw-r-----   1 mshankar cd  3255 May  3  2019 README.md
-rw-r-----   1 mshankar cd  6852 May  3  2019 RELEASE-NOTES
-rw-r-----   1 mshankar cd 16262 May  3  2019 RUNNING.txt
drwxr-x---   2 mshankar cd    30 Sep 17 18:19 temp
drwxr-x---  11 mshankar cd   205 Nov 11 16:58 webapps
drwxr-x---   3 mshankar cd    22 May 22 15:55 work
[ epicsarchiverap ]$

By default, Tomcat sets up a HTTP listener on port 8080. You can change this in the Tomcat server.xml to avoid collision with other folks running Tomcat. For example, here I have changed this to 17665.

<Connector port="17665" protocol="HTTP/1.1"
                connectionTimeout="20000"
                redirectPort="8443" />

To run the unit tests, please make a copy of your Tomcat configuration (preferably pristine) into a new folder called conf_original. The unit tests that use Tomcat copy the conf_original folder to generate new configurations for each test.

    cd ${TOMCAT_HOME}
    cp -R conf conf_original

Gradle will do this step for you if you forget.

Building

The EPICS archiver appliance is shared on GitHub using Git as the source control repository. We use Gradle for building. The default target builds the install package and the various wars and places them into the build/distributions folder.

$ ls build/distributions
archappl_v1.1.0-31-ge02e1f1.dirty.tar.gz

The Gradle build script will build into the default build directory build. You don’t need to install Gradle, instead you can use the wrapper as ./gradlew, or install it and run from the epicsarchiverap folder:

$ gradle
BUILD SUCCESSFUL in 16s
12 actionable tasks: 10 executed, 2 up-to-date

The build can then be found in epicsarchiverap/build/distributions or the war files in epicsarchiverap/build/libs.

Deploying

To deploy the EPICS archiver appliance, you simply have to copy the .war files generated by the build to Tomcat’s webapps folder. Tomcat will expand the war file on startup. It is often more convenient to deploy all the WAR’s into the same tomcat container for development. One can use something like

pushd ${TOMCAT_HOME}/webapps && rm -rf retrieval* && popd && cp ../retrieval.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf engine* && popd && cp ../engine.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf etl* && popd && cp ../etl.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf mgmt* && popd && cp ../mgmt.war ${TOMCAT_HOME}/webapps

to deploy all the same webapps on the same Tomcat instance. To set the locations of the various stores,

export ARCHAPPL_SHORT_TERM_FOLDER=/arch/sts/ArchiverStore
export ARCHAPPL_MEDIUM_TERM_FOLDER=/arch/mts/ArchiverStore
export ARCHAPPL_LONG_TERM_FOLDER=/arch/lts/ArchiverStore

Running Tomcat

Start Tomcat using the catalina.sh run or the catalina.sh start commands. The catalina.sh startup script is found in the Tomcat bin folder. catalina.sh run starts Tomcat and leaves it running in the console so that you can Ctrl-C to terminate. catalina.sh start starts Tomcat in the background and you will need to run catalina.sh stop to stop the process.

To bring up the management app, bring up http://:17665*YourMachineHere*/mgmt/ui/index.html in a recent version of Firefox/Google chrome.

Running the unit tests

Gradle creates temporary directories for all the unit tests. If you wish to clean them first you can use gradle clean. You then have the following options:

gradle test # Runs all unit tests except slow tests
gradle unitTests # Runs all unit tests
gradle epicsTests # Runs all integration tests that require only an epics installation
gradle integrationTests # Runs all tests that require a tomcat installation and optionally an epics installation
gradle flakyTests # Runs all tests that can fail due to system resources
gradle allTests # Runs all tests (not recommended)

Or run individual tests with:

gradle test -tests PolicyExecutionTest
gradle integrationTests --tests PvaGetArchivedPVsTest --info

If you cancel an integrationTest early, or it gets stuck for some reason it’s possible to kill any tomcats running with

gradle shutdownAllTomcats

If you wish to run the current development version locally for testing, it’s possible to use:

gradle testRun

Formatting with Spotless

The gradle build script build.gradle includes the Spotless Plugin which tracks the formatting of the code. To run the formatter run:

gradle spotlessApply

The build script checks that the changes in the current git branch are up-to-date with origin/master branch. So make sure your local origin/master is up-to-date with the home repository master branch to pass the CI checks.

Side projects

The sources for the Archive Viewer integration and the client library for the PB/HTTP protocol are in separate projects in the same repository.

  1. PB/HTTP

  2. Archive Viewer

Information useful to developers

  1. If you unfamiliar with servlet containers; here’s a small overview that a few collaborators found useful

    • Reading up on a few basics will help; there are several good sources of information on the net; but don’t get bogged down by the details.

    • Please do use Eclipse/Netbeans/Intelij to navigate the code. This makes life so much easier.

    • To get a quick sense of what a class/interface does, you can use the javadoc. Some attempts have been made to have some Javadoc in most classes and all interfaces

    • We use Tomcat purely as a servlet container; that is, a quick way of servicing HTTP requests using Java code.

    • A WAR file is basically a ZIP file (you can use unzip) with some conventions. For example, all the libraries (.jar’s) that the application depends on will be located in the WEB-INF/lib folder.

    • The starting point for servlet’s in a WAR file is the file WEB-INF/web.xml. For example, in the mgmt.war’s WEB-INF/web.xml, you can see that all HTTP requests matching the pattern /bpl/* are satisfied using the Java class org.epics.archiverappliance.mgmt.BPLServlet.

    • If you navigate to this class in Eclipse, you’ll see that it is basically a registry of BPLActions.

    • For example, the HTTP request, /mgmt/bpl/getAllPVs is satisfied using the GetAllPVs class. Breaking this down,

      1. /mgmt gets you into the mgmt.war file.

      2. /bpl gets you into the BPLServlet class.

      3. /getAllPVs gets you into the GetAllPVs class.

    • From a very high level

      • The engine.war establishes Channel Access monitors and then writes the data into the short term store (STS).

      • The etl.war file moves data between stores - that is from the STS to the MTS and from the MTS to the LTS and so on.

      • The retrieval.war gathers data from all the stores, stitches them together to satisfy data retrieval requests.

      • The mgmt.war manages all the other three and holds configuration state.

    • In terms of configuration, the most important is the PVTypeInfo; you can see what one looks like by looking at http://machine:17665/mgmt/bpl/getPVTypeInfo?pv=MYPV:111:BDES

    • The main interfaces are the ones in the org.epics.archiverappliance package.

    • The ConfigService class does all configuration management.

    • The customization guide is also a good guide to way in which this product can be customized.