Dynamic (re)-assignment of PVs to appliances

The EAA primarily uses clustering to support archiving millions of PVs. When a PV PV1 is added to EAA, the mgmt component in EAA assigns the PV to one of the appliances in the cluster. That appliance then establishes ca/pva_monitors to the PV and starts writing samples into a sequence of stores. In the following example, if PV1 is assigned to appliance0, then the data for PV1 is stored in STS_0, MTS_0, LTS_0 with the ETL on appliance0 moving data from STS_0 → MTS_0 → LTS_0.

        flowchart TB
  subgraph appliance0
    direction LR
    STS_0 --> MTS_0 --> LTS_0
  end
  subgraph appliance1
    direction LR
    STS_1 --> MTS_1 --> LTS_1
  end
  appliance0 <--JSON/HTTP--> appliance1

The EAA supports a resharding operation that can move the PV PV1 from appliance0 to appliance1. This is a heavyweight operation as it involves first copying the data from STS_0, MTS_0, LTS_0 to STS_1, MTS_1, LTS_1, moving the PV to appliance1 and then finally deleteing the data from STS_0, MTS_0, LTS_0. The reshard operation can take several seconds/minutes to complete.

The reshard BPL is a perfectly adequate approach for installations where the number of appliances in the cluster and the set of PV’s being archived is reasonably static. In more modern cloud-centric environments, devops engineers expect to be able to change the number of appliances in the cluster by changing a single attribute in a YAML file. A more lightweight operation for moving PV’s from one appliance to another would help support this.

However, such a lightweight reassign operation requires careful setup of the data stores a-priori as the EAA is a stateful entity. This setup involves sharing some of the later data stores among all the appliances in the cluster and applying consolidateOnShutdown on the earlier data stores that are specific to the appliance.

For example, in a typical installation, the STS and MTS are local on the appliance hardware while the LTS and XLTS are on remote network storage. Often the STS is a RAMDisk, and thus has a consolidateOnShutdown attribute on the data store to move the data off the STS onto the MTS when the appliance is being gracefully shutdown or the PV is being paused.

        flowchart LR
  subgraph Appliance_Hardware
    direction LR
    STS["STS with consolidateOnShutdown"]
    STS --> MTS
  end
  subgraph Network_Storage
    direction LR
    LTS --> XLTS
  end
  Appliance_Hardware --> Network_Storage

One can configure the data stores such that the LTS and XLTS are shared between the appliances. That is, both appliance0 and appliance1 write to the exact same location on the LTS and XLTS. But as the PVs are distributed between the appliances on a one-to-one basis, there should be no conflicts when writing data as no two appliances will archive the same PV.

        flowchart TB
  subgraph appliance0
    direction LR
    STS_0
    MTS_0
    STS_0 --> MTS_0
  end
  subgraph appliance1
    direction LR
    STS_1
    MTS_1
    STS_1 --> MTS_1
  end
  subgraph Network_Storage
    direction LR
    LTS --> XLTS
  end
  appliance0 --> Network_Storage
  appliance1 --> Network_Storage

Note that there may be some performance impact from a shared LTS/XLTS as data files from both appliance0 and appliance1 will now be stored in the same folders on LTS/XLTS. And since retrieval typically involves doing an ls in a folder with many thousands of files, increasing the number of files in your LTS/XLTS folder may have performance implications and some impact on the metadata servers for your file system.

However, in such a configuration with a shared LTS/XLTS, if we configure both the STS and MTS to have consolidateOnShutdown, then, to move a PV from appliance0 to appliance1

We pause the PV on appliance0. This should then flush all the data for the PV into the shared LTS
We change the appliance assignment for the PV in the cluster from appliance0 to appliance1.
We resume the PV on appliance1. Since the LTS is shared, appliance1 should now be able to access the data for the PV from the shared LTS and there is no loss of data.

        flowchart LR
  subgraph Appliance_Specific_Storage
    direction LR
    STS["STS with consolidateOnShutdown"]
    MTS["MTS with consolidateOnShutdown"]
    STS --> MTS
  end
  subgraph Shared_Network_Storage
    direction LR
    LTS --> XLTS
  end
  Appliance_Specific_Storage --> Shared_Network_Storage

With recent releases, the EAA appliances now react to changes in PVTypeInfo. So these multiple steps for PV reassignment can be accomplished by simply changing the appliance assignment for the PV in the cluster/PVTypeInfo from appliance0 to appliance1. And this is what the reassignAppliance BPL does.

Executing the reassignAppliance BPL in an installation that is not configured to support it can result in data loss with data for a PV being stranded across multiple appliances in the cluster. Therefore, this BPL is disabled by default. To enable this for your installation, please use the org.epics.archiverappliance.mgmt.bpl.ReassignAppliance property in your archappl.properties.