# Developers Guide

## Prerequisites

Please see the [details](./details#system-requirements) page for
prerequisites to build and test the EPICS Archiver Appliance. An
installation of Tomcat is required to build successfully; this is
located using the environment variable `TOMCAT_HOME`. Use something like

```bash
[ epicsarchiverap ]$ echo $TOMCAT_HOME
/opt/local/tomcat/latest
[ epicsarchiverap ]$ ls -l $TOMCAT_HOME/
drwxr-x---   3 mshankar cd  4096 Oct 29 18:25 bin
-rw-r-----   1 mshankar cd 19182 May  3  2019 BUILDING.txt
drwx------   3 mshankar cd   254 Jul 29 14:41 conf
drwx------   2 mshankar cd   238 May 22 15:43 conf_from_install
drwxr-xr-x+  2 mshankar cd   238 May 22 15:44 conf_original
-rw-r-----   1 mshankar cd  5407 May  3  2019 CONTRIBUTING.md
drwxr-x---   2 mshankar cd  4096 Sep 17 18:13 lib
-rw-r-----   1 mshankar cd 57092 May  3  2019 LICENSE
drwxr-x---   2 mshankar cd   193 Nov 11 16:58 logs
-rw-r-----   1 mshankar cd  2333 May  3  2019 NOTICE
-rw-r-----   1 mshankar cd  3255 May  3  2019 README.md
-rw-r-----   1 mshankar cd  6852 May  3  2019 RELEASE-NOTES
-rw-r-----   1 mshankar cd 16262 May  3  2019 RUNNING.txt
drwxr-x---   2 mshankar cd    30 Sep 17 18:19 temp
drwxr-x---  11 mshankar cd   205 Nov 11 16:58 webapps
drwxr-x---   3 mshankar cd    22 May 22 15:55 work
[ epicsarchiverap ]$
```

By default, Tomcat sets up a HTTP listener on port 8080. You can change
this in the Tomcat server.xml to avoid collision with other folks
running Tomcat. For example, here I have changed this to 17665.

```xml
<Connector port="17665" protocol="HTTP/1.1"
                connectionTimeout="20000"
                redirectPort="8443" />
```

To run the unit tests, please make a copy of your Tomcat configuration
(preferably pristine) into a new folder called `conf_original.` The unit
tests that use Tomcat copy the conf_original folder to generate new
configurations for each test.

```bash
    cd ${TOMCAT_HOME}
    cp -R conf conf_original
```

Gradle will do this step for you if you forget.

## Building

The EPICS archiver appliance is shared on
[GitHub](https://github.com/slacmshankar/epicsarchiverap) using Git as
the source control repository. We use [Gradle](http://gradle.org/) for
building. The default target builds the install package and the various
wars and places them into the `build/distributions` folder.

```bash
$ ls build/distributions
archappl_v1.1.0-31-ge02e1f1.dirty.tar.gz
```

The Gradle build script will build into the default build directory
`build`. You don\'t need to install Gradle, instead you can use the
wrapper as `./gradlew`, or install it and run from the `epicsarchiverap`
folder:

```bash
$ gradle
BUILD SUCCESSFUL in 16s
12 actionable tasks: 10 executed, 2 up-to-date
```

The build can then be found in `epicsarchiverap/build/distributions` or
the war files in `epicsarchiverap/build/libs`.

## Deploying

To deploy the EPICS archiver appliance, you simply have to copy the
`.war` files generated by the build to Tomcat\'s webapps folder. Tomcat
will expand the war file on startup. It is often more convenient to
deploy all the WAR\'s into the same tomcat container for development.
One can use something like

```bash
pushd ${TOMCAT_HOME}/webapps && rm -rf retrieval* && popd && cp ../retrieval.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf engine* && popd && cp ../engine.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf etl* && popd && cp ../etl.war ${TOMCAT_HOME}/webapps
pushd ${TOMCAT_HOME}/webapps && rm -rf mgmt* && popd && cp ../mgmt.war ${TOMCAT_HOME}/webapps
```

to deploy all the same webapps on the same Tomcat instance. To set the
locations of the various stores,

```bash
export ARCHAPPL_SHORT_TERM_FOLDER=/arch/sts/ArchiverStore
export ARCHAPPL_MEDIUM_TERM_FOLDER=/arch/mts/ArchiverStore
export ARCHAPPL_LONG_TERM_FOLDER=/arch/lts/ArchiverStore
```

## Running Tomcat

Start Tomcat using the `catalina.sh run` or the `catalina.sh start`
commands. The `catalina.sh` startup script is found in the Tomcat bin
folder. `catalina.sh run` starts Tomcat and leaves it running in the
console so that you can Ctrl-C to terminate. `catalina.sh start` starts
Tomcat in the background and you will need to run `catalina.sh stop` to
stop the process.

To bring up the management app, bring up
<http://*YourMachineHere*:17665/mgmt/ui/index.html> in a recent
version of Firefox/Google chrome.

## Running the unit tests

Gradle creates temporary directories for all the unit tests. If you wish
to clean them first you can use `gradle clean`. You then have the
following options:

```bash
gradle test # Runs all unit tests except slow tests
gradle unitTests # Runs all unit tests
gradle epicsTests # Runs all integration tests that require only an epics installation
gradle integrationTests # Runs all tests that require a tomcat installation and optionally an epics installation
gradle flakyTests # Runs all tests that can fail due to system resources
gradle allTests # Runs all tests (not recommended)
```

Or run individual tests with:

```bash
gradle test -tests PolicyExecutionTest
gradle integrationTests --tests PvaGetArchivedPVsTest --info
```

If you cancel an integrationTest early, or it gets stuck for some reason
it\'s possible to kill any tomcats running with

```bash
gradle shutdownAllTomcats
```

If you wish to run the current development version locally for testing,
it\'s possible to use:

```bash
gradle testRun
```

## Formatting with Spotless

The gradle build script `build.gradle` includes the [Spotless Plugin](https://github.com/diffplug/spotless)
which tracks the
formatting of the code. To run the formatter run:

```bash
gradle spotlessApply
```

The build script checks that the changes in the current git branch are
up-to-date with `origin/master` branch. So make sure your local
`origin/master` is up-to-date with the [home repository](https://github.com/slacmshankar/epicsarchiverap) master
branch to pass the CI checks.

## Side projects

The sources for the [Archive Viewer](sysadmin/archiveviewer) integration and
the client library for the [PB/HTTP](developer/pb_pbraw) protocol are in
separate projects in the same repository.

1. [PB/HTTP](https://github.com/slacmshankar/epicsarchiverap_pbrawclient/)
2. [Archive Viewer](https://github.com/slacmshankar/epicsarchiverap_archiveviewer/)

## Information useful to developers

1. If you unfamiliar with servlet containers; here\'s a small overview
   that a few collaborators found useful
   - Reading up on a few basics will help; there are several good
     sources of information on the net; but don\'t get bogged down by
     the details.
   - Please do use Eclipse/Netbeans/Intelij to navigate the code.
     This makes life so much easier.
   - To get a quick sense of what a class/interface does, you can use
     the [javadoc](../_static/javadoc/index.html). Some attempts have been made to
     have some Javadoc in most classes and all interfaces
   - We use Tomcat purely as a servlet container; that is, a quick
     way of servicing HTTP requests using Java code.
   - A WAR file is basically a ZIP file (you can use unzip) with some
     conventions. For example, all the libraries (.jar\'s) that the
     application depends on will be located in the WEB-INF/lib
     folder.
   - The starting point for servlet\'s in a WAR file is the file
     `WEB-INF/web.xml`. For example, in the mgmt.war\'s
     `WEB-INF/web.xml`, you can see that all HTTP requests matching
     the pattern `/bpl/*` are satisfied using the Java class
     `org.epics.archiverappliance.mgmt.BPLServlet`.
   - If you navigate to this class in Eclipse, you\'ll see that it is
     basically a registry of BPLActions.
   - For example, the HTTP request, `/mgmt/bpl/getAllPVs` is
     satisfied using the `GetAllPVs` class. Breaking this down,
     1. `/mgmt` gets you into the mgmt.war file.
     2. `/bpl` gets you into the BPLServlet class.
     3. `/getAllPVs` gets you into the GetAllPVs class.
   - From a very high level
     - The engine.war establishes Channel Access monitors and then
       writes the data into the short term store (STS).
     - The etl.war file moves data between stores - that is from
       the STS to the MTS and from the MTS to the LTS and so on.
     - The retrieval.war gathers data from all the stores, stitches
       them together to satisfy data retrieval requests.
     - The mgmt.war manages all the other three and holds
       configuration state.
   - In terms of configuration, the most important is the
     `PVTypeInfo`; you can see what one looks like by looking at
     <http://machine:17665/mgmt/bpl/getPVTypeInfo?pv=MYPV:111:BDES>
   - The main interfaces are the ones in the
     [`org.epics.archiverappliance`](../_static/javadoc/org/epics/archiverappliance/package-summary.html)
     package.
   - The
     [ConfigService](../_static/javadoc/org/epics/archiverappliance/config/ConfigService.html)
     class does all configuration management.
   - The [customization guide](../sysadmin/customization) is also a good
     guide to way in which this product can be customized.