Package org.epics.archiverappliance.config

Interface ConfigService

All Known Implementing Classes:
ConfigServiceForTests, DefaultConfigService
public interface ConfigService
Interface for appliance configuration. One gets to a config service implementation thru dependency injection of one kind or the other. In a servlet container, this is initialized by ArchServletContextListener (which is registered as part of the web.xml). ArchServletContextListener is also used in the unit tests that involve tomcat. Guice is a good option for this but it takes over the dispatch logic from tomcat and we'll need to investigate if that has any impact.
Author:
mshankar

  • Field Details

    • ARCHAPPL_APPLIANCES

      static final String ARCHAPPL_APPLIANCES
      This is the environment variable that points to the file containing the various appliances in this cluster. This list of appliances is expected to be the same for all appliances in the cluster; so it is perfectly legal to place it in NFS somewhere and point to the same file/location from all appliances in the cluster. It is reasonably important that all appliances see the same list of cluster members or we tend to have split-brain effects (See wikipedia). The format of the file itself is simple XML like so 
       <appliances>
   <appliance>
     <identity>archiver</identity>
     <cluster_inetport>archiver:77770</cluster_inetport>
     <mgmt_url>http://archiver.slac.stanford.edu:77765/mgmt/bpl</mgmt_url>
     <engine_url>http://archiver.slac.stanford.edu:77765/engine/bpl</engine_url>
     <etl_url>http://archiver.slac.stanford.edu:77765/etl/bpl</etl_url>
     <retrieval_url>http://archiver.slac.stanford.edu:77765/retrieval/bpl</retrieval_url>
     <data_retrieval_url>http://archiver.slac.stanford.edu:77765/retrieval/</data_retrieval_url>
   </appliance>
 </appliances>
      Note that the appliance identity as defined by the ARCHAPPL_MYIDENTITY has to match the identity element of one of the appliances in the list of appliances as defined by the ARCHAPPL_APPLIANCES. Each appliance (which includes the mgmt, engine, etl and retrieval WAR's) must have a unique identity.
      If the ARCHAPPL_APPLIANCES is not set, then we look for a file called appliances.xml in the WEB-INF/classes of the current WAR using WEB-INF/classes/appliances.xml. The default build script places the site-specific appliances.xml into WEB-INF/classes/appliances.xml.
      See Also:

    • ARCHAPPL_MYIDENTITY

      static final String ARCHAPPL_MYIDENTITY
      This is an optional environment variable that determines this appliance's identity. If this is not set, the archiver appliance uses InetAddress.getLocalHost().getCanonicalHostName() to determine the FQDN of this machine. This is then used as the appliance identity to lookup the appliance info in ARCHAPPL_APPLIANCES. To use this environment variable, for example, in Linux, set the appliance's identity using export ARCHAPPL_MYIDENTITY="archiver". Each appliance (which includes the mgmt, engine, etl and retrieval WAR's) must have a unique identity. To accommodate the multi-instance unit tests, if this environment variable is not set, we check for the existence of the java system property ARCHAPPL_MYIDENTITY. Typically, the multi-instance unit tests (which are incapable of altering the environment) use the java system property method. In environments that run the unit tests, leave the environment variable ARCHAPPL_MYIDENTITY unset so that the various multi-instance unit tests have the ability to control the appliance identity.
      See Also:

    • ARCHAPPL_SITEID

      static final String ARCHAPPL_SITEID
      This is the environment variable that identifies the site (LCLS, LCLSII, slacdev, NSLSII etc) to be used when generating the war files. This is primarily a build-time property; the build.xml has various site specific hooks which let you change the appliances.xml, policies, images etc on a per site basis. The unit tests use the tests site which is also the default site if this environment variable is not specified. Files for a site are stored in the sitespecific/<site> folder.
      See Also:

    • ARCHAPPL_CONFIGSERVICE_IMPL

      static final String ARCHAPPL_CONFIGSERVICE_IMPL
      This is an optional environment variable/system property that is used to identify the config service implementation.
      See Also:

    • CONFIG_SERVICE_NAME

      static final String CONFIG_SERVICE_NAME
      This is the name used to identify the config service implementation in one of the existing singletons (like the ServletContext)
      See Also:

    • ARCHAPPL_PERSISTENCE_LAYER

      static final String ARCHAPPL_PERSISTENCE_LAYER
      This is an optional environment/system.property that is used to identity the persistence layer If this is not set, we initialize MySQLPersistence as the persistence layer; so in production environments, you can leave this unset/blank Set this to the class name of the class implementing ConfigPersistence The unit tests however will set this to use InMemoryPersistence, which is a dummy persistence layer.
      See Also:

    • ARCHAPPL_POLICIES

      static final String ARCHAPPL_POLICIES
      This is an optional environment/system.property that is used to specify the location of the policies.py policies file. If this is not set, we search for the policies.py in the servlet classpath using the path /WEB-INF/classes/policies.py
      See Also:

    • ARCHAPPL_PROPERTIES_FILENAME

      static final String ARCHAPPL_PROPERTIES_FILENAME
      The name/path of the archappl.properties file. By default, we look for archappl.properties in the webapp's classpath - this will typically resolve into WEB-INF/classes of the webapp. However, you can override this using an environment variable (or java system property) of the same name. For example, export ARCHAPPL_PROPERTIES_FILENAME=/etc/mylab_archappl.properties should force the components to load their properties from /etc/mylab_archappl.properties
      See Also:

    • DEFAULT_ARCHAPPL_PROPERTIES_FILENAME

      static final String DEFAULT_ARCHAPPL_PROPERTIES_FILENAME
      This is the name of the properties file that is looked for in the webapp's classpath if one is not specified using a environment/JVM property.
      See Also:

    • ARCHAPPL_NAMEDFLAGS_PROPERTIES_FILE_PROPERTY

      static final String ARCHAPPL_NAMEDFLAGS_PROPERTIES_FILE_PROPERTY
      See Also:

  • Method Details

    • initialize

      void initialize(jakarta.servlet.ServletContext sce) throws ConfigException
      If you have a null constructor and need a ServletContext, implement this method.
      Parameters:
      sce - ServletContext
      Throws:
      ConfigException -  

    • getWebInfFolder

      String getWebInfFolder()
      We expect to live within a servlet container. This call returns the full path to the WEB-INF folder of the webapp as it is deployed in the container. Is typically a call to the servletContext.getRealPath("WEB-INF/")
      Returns:
      String WebInfFolder  

    • postStartup

      void postStartup() throws ConfigException
      This method is called after the mgmt WAR file has started up and set up the cluster and recovered data from persistence. Each appliance's mgmt war is responsible for calling this method on the other components (engine, etl and retrieval) using BPL. Until this method is called on all the web apps, the cluster is not considered to have started up.
      Throws:
      ConfigException -  

    • getStartupState

      ConfigService.STARTUP_SEQUENCE getStartupState()
      Used for inter-appliance startup checks.
      Returns:
      STARTUP_SEQUENCE  

    • isStartupComplete

      boolean isStartupComplete()
      Have we completed all the startup steps?
      Returns:
      boolean True or False

    • getTimeOfAppserverStartup

      long getTimeOfAppserverStartup()
      Get an approximate time in epoch seconds when the appserver started up.
      Returns:
      The time this app server started up.

    • getInstallationProperties

      Properties getInstallationProperties()
      An arbitrary list of name/value pairs can be specified in a file called archappl.properties that is loaded from the classpath.
      Returns:
      Properties  

    • getAppliancesInCluster

      Iterable<ApplianceInfo> getAppliancesInCluster()
      Get all the appliances in this cluster. Much goodness is facilitated if the objects are returned in the same order (perhaps order of creation) all the time.
      Returns:
      ApplianceInfo  

    • getMyApplianceInfo

      ApplianceInfo getMyApplianceInfo()
      Get the appliance information for this appliance.
      Returns:
      ApplianceInfo  

    • getAppliance

      ApplianceInfo getAppliance(String identity)
      Given an identity of an appliance, return the appliance info for that appliance
      Parameters:
      identity - The appliance identify
      Returns:
      ApplianceInfo  

    • hasClusterFinishedInitialization

      boolean hasClusterFinishedInitialization()
      To prevent split brain side-effects, we support cetain BPL only when all the member of the cluster have finished loading their PVs into the cluster. This consists of two checks 1) Make sure all appliances listed in appliances.xml have started up and are part of the cluster 2) All appliances in the cluster have registered their PVs with the cluster. Previously, we'd allow appliances.xml to have more appliances that are actually present in the cluster. However; this is becoming increasingly hard to support. We've had to tighten this to avoid split brain issues which can happen when the networking between instances fails.
      Returns:
      boolean  

    • getAllPVs

      Collection<String> getAllPVs()
      Get an exhaustive list of all the PVs this cluster of appliances knows about Much goodness is facilitated if the objects are returned in the same order (perhaps order of creation) all the time.
      Returns:
      String AllPVs  

    • getAllExpandedNames

      void getAllExpandedNames(Consumer<String> func)
      For automated PV submission, IOC engineers could add .VAL, fields, aliases etc. This method attempts to return all possible PV's that the archiver could know about. This is a lot of names; so we take in a consumer that potentially streams a name out as quickly as possible.
      Parameters:
      func - A consumer of pvNames

    • getApplianceForPV

      ApplianceInfo getApplianceForPV(String pvName)
      Given a PV, get us the appliance that is responsible for archiving it. Note that this may be null as the assignment of PV's to appliances can take some time.
      Parameters:
      pvName - The name of PV.
      Returns:
      ApplianceInfo  

    • getPVsForAppliance

      Set<String> getPVsForAppliance(ApplianceInfo info)
      Get all PVs being archived by this appliance. Much goodness is facilitated if the objects are returned in the same order (perhaps order of creation) all the time.
      Parameters:
      info - ApplianceInfo
      Returns:
      string All PVs being archiveed by this appliance

    • getPVsForThisAppliance

      Set<String> getPVsForThisAppliance()
      Get all the PVs for this appliance. Much goodness is facilitated if the objects are returned in the same order (perhaps order of creation) all the time.
      Returns:
      String All PVs being archiveed for this appliance

    • getCachedPVCountsForThisAppliance

      ConfigService.CachedPVCounts getCachedPVCountsForThisAppliance()

    • queryPVTypeInfos

      <T> Collection<T> queryPVTypeInfos(com.hazelcast.query.Predicate<String,PVTypeInfo> predicate, com.hazelcast.projection.Projection<Map.Entry<String,PVTypeInfo>,T> projection)
      Query this cluster's pvTypeInfos using the supplied the predicate and then run the supplied projection operator. This runs using Hz's query functions and can be run from any war file For example, to quickly determine the appliances for a bunch of PV's, project the applianceIdentity and then do a stream groupby.
      Returns:

    • breakDownPVsByAppliance

      Map<String,List<String>> breakDownPVsByAppliance(List<String> pvNames)
      Prepare for batch jobs by breaking down a list of PV's into a Map that maps appliance identity to a list of PV's being archived on that appliance.
      Returns:

    • executeClusterWide

      <T> Map<String,T> executeClusterWide(ConfigService.EAABulkOperation<T> theOperation)

    • executeOnAppliance

      <T> T executeOnAppliance(ApplianceInfo applianceInfo, ConfigService.EAABulkOperation<T> theOperation)

    • getPVsForApplianceMatchingRegex

      Set<String> getPVsForApplianceMatchingRegex(String nameToMatch)
      Get the pvNames for this appliance matching the given regex.
      Parameters:
      nameToMatch -  
      Returns:
      string PVsForApplianceMatchingRegex  

    • registerPVToAppliance

      void registerPVToAppliance(String pvName, ApplianceInfo applianceInfo, PVRegistrationType registrationType) throws AlreadyRegisteredException
      Make changes in the config service to register this PV to an appliance
      Parameters:
      pvName - The name of PV.
      applianceInfo - ApplianceInfo
      registrationType - If reassigning; then an AlreadyRegisteredException is not raised
      Throws:
      AlreadyRegisteredException -  

    • registerPVToAppliance

      default void registerPVToAppliance(String pvName, ApplianceInfo applianceInfo) throws AlreadyRegisteredException
      Throws:
      AlreadyRegisteredException

    • getTypeInfoForPV

      PVTypeInfo getTypeInfoForPV(String pvName)
      Gets information about a PV's type, i.e its DBR type, graphic limits etc. This information is assumed to be somewhat static and is expected to come from a cache if possible as it is used in data retrieval.
      Parameters:
      pvName - The name of PV.
      Returns:
      PVTypeInfo  

    • updateTypeInfoForPV

      void updateTypeInfoForPV(String pvName, PVTypeInfo typeInfo)
      Update the type information about a PV; updating both ther persistent and cached versions of the information. Clients are not expected to call this method a million times a second. In general, this is expected to be called when archiving a PV for the first time, or perhaps when an appserver startups etc...
      Parameters:
      pvName - The name of PV.
      typeInfo - PVTypeInfo

    • removePVFromCluster

      void removePVFromCluster(String pvName)
      Remove the pv from all cached and persisted configuration.
      Parameters:
      pvName - The name of PV.

    • getAggregatedApplianceInfo

      ApplianceAggregateInfo getAggregatedApplianceInfo(ApplianceInfo applianceInfo) throws IOException
      Facilitates various optimizations for BPL that uses appliance wide information by caching and maintaining this information on a per appliance basis
      Parameters:
      applianceInfo - ApplianceInfo
      Returns:
      ApplianceAggregateInfo  
      Throws:
      IOException -  

    • addToArchiveRequests

      void addToArchiveRequests(String pvName, UserSpecifiedSamplingParams userSpecifiedSamplingParams)
      The workflow for requesting a PV to be archived consists of multiple steps This method adds a PV to the persisted list of PVs that are currently engaged in this workflow in addition to any user specified overrides
      Parameters:
      pvName - The name of PV.
      userSpecifiedSamplingParams - - Use a null contructor for userSpecifiedSamplingParams if no override specified.

    • updateArchiveRequest

      void updateArchiveRequest(String pvName, UserSpecifiedSamplingParams userSpecifiedSamplingParams)
      Update the archive request (mostly with aliases) if and only if we have this in our persistence.
      Parameters:
      pvName - The name of PV.
      userSpecifiedSamplingParams -  

    • getArchiveRequestsCurrentlyInWorkflow

      Set<String> getArchiveRequestsCurrentlyInWorkflow()
      Gets a list of PVs that are currently engaged in the archive PV workflow
      Returns:
      String ArchiveRequestsCurrentlyInWorkflow  

    • doesPVHaveArchiveRequestInWorkflow

      boolean doesPVHaveArchiveRequestInWorkflow(String pvname)
      Is this pv in the archive request workflow.
      Parameters:
      pvname - The name of PV.
      Returns:
      boolean True or False

    • getInitialDelayBeforeStartingArchiveRequestWorkflow

      int getInitialDelayBeforeStartingArchiveRequestWorkflow()
      In clustered environments, to give capacity planning a chance to work correctly, we want to kick off the archive PV workflow only after all the machines have started. This is an approximation for that metric; though not a very satisfactory approximation. TODO -- Think thru implications of making the appliances.xml strict...
      Returns:
      - Initial delay in seconds.

    • getUserSpecifiedSamplingParams

      UserSpecifiedSamplingParams getUserSpecifiedSamplingParams(String pvName)
      Returns any user specified parameters for the archive request.
      Parameters:
      pvName - The name of PV.
      Returns:
      UserSpecifiedSamplingParams  

    • archiveRequestWorkflowCompleted

      void archiveRequestWorkflowCompleted(String pvName)
      Mark this pv as having it archive pv request completed and pull this request out of persistent store Can be used in the case of aborting a PV archive request as well
      Parameters:
      pvName - The name of PV.

    • getExtraFields

      String[] getExtraFields()
      Get a list of extra fields that are obtained when we initially make a request for archiving. These are used in the policies to make decisions on how to archive the PV.
      Returns:
      String ExtraFields  

    • getRuntimeFields

      Set<String> getRuntimeFields()
      Get a list of fields for PVs that are monitored and maintained in the engine. These are used when displaying the PV in visualization tools like the ArchiveViewer as additional information for the PV. Some of these could be archived along with the PV but need not be. In this case, the engine simply maintains the latest copy in memory and this is served up when data from the engine in included in the stream.
      Returns:
      String RuntimeFields

    • addAlias

      void addAlias(String aliasName, String realName)
      Register an alias
      Parameters:
      aliasName -  
      realName - This is the name under which the PV will be archived under

    • removeAlias

      void removeAlias(String aliasName, String realName)
      Remove an alias for the specified realname
      Parameters:
      aliasName -  
      realName - This is the name under which the PV will be archived under

    • getAllAliases

      List<String> getAllAliases()
      Get all the aliases in the system. This is used for matching during glob requests in the UI.
      Returns:
      String AllAliases  

    • getRealNameForAlias

      String getRealNameForAlias(String aliasName)
      Gets the .NAME field for a PV if it exists. Otherwise, this returns null
      Parameters:
      aliasName -  
      Returns:
      String RealNameForAlias

    • getAliasesForRealName

      List<String> getAliasesForRealName(String realName)
      Gets all the aliases in the system that map to this real PV name
      Parameters:
      realName -  
      Returns:
      A list of aliases for this real name

    • getPolicyText

      InputStream getPolicyText() throws IOException
      Return the text of the policy for this installation. Gets you an InputStream; remember to close it.
      Returns:
      InputStream  
      Throws:
      IOException -  

    • computePolicyForPV

      PolicyConfig computePolicyForPV(String pvName, MetaInfo metaInfo, UserSpecifiedSamplingParams userSpecParams) throws IOException
      Given a pvName (for now, we should have a pv details object of some kind soon), determine the policy applicable for archiving this PV.
      Parameters:
      pvName - The name of PV.
      metaInfo - The MetaInfo of PV
      userSpecParams - UserSpecifiedSamplingParams
      Returns:
      PolicyConfig  
      Throws:
      IOException -  

    • getPoliciesInInstallation

      HashMap<String,String> getPoliciesInInstallation() throws IOException
      Return a map of name to description of all the policies in the system This is used to drive a dropdown in the UI.
      Returns:
      HashMap  
      Throws:
      IOException -  

    • getFieldsArchivedAsPartOfStream

      List<String> getFieldsArchivedAsPartOfStream() throws IOException
      This product offers the ability to archive certain fields (like HIHI, LOLO etc) as part of every PV. The data for these fields is embedded into the stream as extra fields using the FieldValues interface of events. This method lists all these fields. Requests for archiving these fields are deferred to and combined with the request for archiving the .VAL. We also assume that the data type (double/float) for these fields is the same as the .VAL.
      Returns:
      String  
      Throws:
      IOException -  

    • getArchiverTypeSystem

      TypeSystem getArchiverTypeSystem()
      Returns a TypeSystem object that is used to convert from JCA DBR's to Event's (actually, DBRTimeEvents)
      Returns:
      TypeSystem  

    • getWarFile

      ConfigService.WAR_FILE getWarFile()
      Which component is this configservice instance.
      Returns:
      WAR_FILE  

    • getRetrievalRuntimeState

      RetrievalState getRetrievalRuntimeState()
      Returns the runtime state for the retrieval app
      Returns:
      RetrievalState  

    • getETLLookup

      PBThreeTierETLPVLookup getETLLookup()
      Return the runtime state for ETL. This may eventually be moved to a RunTime class but that would still start from the configservice.
      Returns:
      PBThreeTierETLPVLookup  

    • getEngineContext

      EngineContext getEngineContext()
      Return the runtime state for the engine.
      Returns:
      EngineContext  

    • getMgmtRuntimeState

      MgmtRuntimeState getMgmtRuntimeState()
      Return the runtime state for the mgmt webapp.
      Returns:
      MgmtRuntimeStat  

    • isShuttingDown

      boolean isShuttingDown()
      Is this appliance component shutting down?
      Returns:
      boolean True or False

    • addShutdownHook

      void addShutdownHook(Runnable runnable)
      Add an appserver agnostic shutdown hook; for example, to close the CA channels on shutdown
      Parameters:
      runnable - Runnable

    • shutdownNow

      void shutdownNow()
      Call the registered shutdown hooks and shut the archive appliance down.

    • getEventBus

      com.google.common.eventbus.EventBus getEventBus()
      Get the event bus used for events within this appliance.
      Returns:
      EventBus  

    • getExternalArchiverDataServers

      Map<String,String> getExternalArchiverDataServers()
      This product has the ability to proxy data from other archiver data servers. We currently integrate with Channel Archiver XMLRPC data servers and other EPICS Archiver Appliance clusters. Get a list of external Archiver Data Servers that we know about.
      Returns:
      Map ExternalArchiverDataServers

    • addExternalArchiverDataServer

      void addExternalArchiverDataServer(String serverURL, String archivesCSV) throws IOException
      Add a external Archiver Data Server into the system.
      Parameters:
      serverURL - - For Channel Archivers, this is the URL to the XML-RPC server. For other EPICS Archiver Appliance clusters, this is the data_retrieval_url of the cluster as defined in the appliances.xml.
      archivesCSV - - For Channel Archivers, this is a comma separated list of indexes. For other EPICS Archiver Appliance clusters, this is the string pbraw.
      Throws:
      IOException -  

    • removeExternalArchiverDataServer

      void removeExternalArchiverDataServer(String serverURL, String archivesCSV) throws IOException
      Removes an entry for an external Archiver Data Server from the system Note; we may need to restart the entire cluster for this change to take effect.
      Parameters:
      serverURL - - For Channel Archivers, this is the URL to the XML-RPC server. For other EPICS Archiver Appliance clusters, this is the data_retrieval_url of the cluster as defined in the appliances.xml.
      archivesCSV - - For Channel Archivers, this is a comma separated list of indexes. For other EPICS Archiver Appliance clusters, this is the string pbraw.
      Throws:
      IOException -  

    • getChannelArchiverDataServers

      List<ChannelArchiverDataServerPVInfo> getChannelArchiverDataServers(String pvName)
      Return a list of ChannelArchiverDataServerPVInfos for a PV if one exists; otherwise return null. The servers are sorted in order of the start seconds. Note: this only applies to Channel Archiver XML RPC servers. For proxying external EPICS Archiver Appliance clusters, we do not cache the PV's that are being archived on the external system.
      Parameters:
      pvName - The name of PV.
      Returns:
      ChannelArchiverDataServerPVInfo  

    • refreshPVDataFromChannelArchiverDataServers

      void refreshPVDataFromChannelArchiverDataServers()
      For all the Channel Archiver XMLRPC data servers in the mix, update the PV info. This should help improve performance a little in proxying data from ChannelArchiver data servers that are still active. For proxying external EPICS Archiver Appliance clusters, since we do not cache the PV's that are being archived on the external system, this is a no-op.

    • getFailoverServerURLs

      Set<String> getFailoverServerURLs()
      Get a list of external archiver appliances configured for failover. Failover appliances are not used as proxies to minimize imposing the retrieval load of the this installation on a potentially less powerful appliance used principally for failover.
      Returns:

    • getFailoverApplianceURL

      String getFailoverApplianceURL(String pvName)
      Get the first external archiver appliance that also archives this PV and is configured with a mergeDuringRetrieval query parameter.
      Returns:
      - Returns null if no failover appliance.

    • resetFailoverCaches

      void resetFailoverCaches()
      Each retrieval component in a cluster caches the PV's from remote failover appliances. These caches contain one entry for each PV in this appliance indicating if the PV is being archived in the remote appliance. This information is cached using a TTL to minimize the impact on the remote failover appliance. This method manually unloads this cache.

    • getPVNameToKeyConverter

      PVNameToKeyMapping getPVNameToKeyConverter()
      Implementation for converting a PV name to something that forms the prefix of a chunk's key. See @see{PVNameToKeyMapping} for more details.
      Returns:
      PVNameToKeyMapping  

    • getPausedPVsInThisAppliance

      Set<String> getPausedPVsInThisAppliance()
      Get a set of PVs that have been paused in this appliance.
      Returns:
      String  

    • getNamedFlag

      boolean getNamedFlag(String name)
      Named flags are used to control various process in the appliance; for example, the ETL process in a PlainStoragePlugin Named flags are not persistent; each time the server starts up, all the named flags are set to false You can optionally load values for named flags from a file by specifying the ARCHAPPL_NAMEDFLAGS_PROPERTIES_FILE_PROPERTY property in archappl.properties. This method gets the value of the specified named flag. If the flag has not been defined before in the cluster, this method will return false.
      Parameters:
      name -  
      Returns:
      boolean True or False

    • setNamedFlag

      void setNamedFlag(String name, boolean value)
      Sets the value of the named flag specified by name to the specified value
      Parameters:
      name -  
      value -  

    • getNamedFlagNames

      Set<String> getNamedFlagNames()
      Return the names of all the named flags that we know about
      Returns:
      String