Package edu.stanford.slac.archiverappliance.plain

Class AppendDataStateData

java.lang.Object
edu.stanford.slac.archiverappliance.plain.AppendDataStateData
Direct Known Subclasses:
ParquetAppendDataStateData, PBAppendDataStateData
public abstract class AppendDataStateData extends Object
Companion class to PlainStoragePlugin that handles the appending of event streams in a partition-aware fashion. This is used both by the engine and by ETL.
Author:
mshankar

  • Field Details

    • writer

      protected EventFileWriter writer

    • rootFolder

      protected final String rootFolder

    • desc

      protected final String desc

    • partitionGranularity

      protected final PartitionGranularity partitionGranularity

    • pv2key

      protected final PVNameToKeyMapping pv2key

    • previousFilePath

      protected Path previousFilePath

    • currentEventsYear

      protected short currentEventsYear

    • previousYear

      protected short previousYear

    • lastKnownTimeStamp

      protected Instant lastKnownTimeStamp

  • Constructor Details

    • AppendDataStateData

      public AppendDataStateData(PartitionGranularity partitionGranularity, String rootFolder, String desc, Instant lastKnownTimestamp, PVNameToKeyMapping pv2key, PathResolver pathResolver)
      Parameters:
      partitionGranularity - partitionGranularity of the PB plugin.
      rootFolder - RootFolder of the PB plugin
      desc - Desc for logging purposes
      lastKnownTimestamp - This is probably the most important argument here. This is the last known timestamp in this storage. If null, we assume time(0) for the last known timestamp.
      pv2key - PVNameToKeyMapping

  • Method Details

    • getPathResolver

      protected PathResolver getPathResolver()

    • partitionBoundaryAwareAppendData

      public int partitionBoundaryAwareAppendData(BasicContext context, String pvName, EventStream stream, String extension, String extensionToCopyFrom) throws IOException
      Append data into PB files honoring partition boundaries switching into new partitions as we cross the boundary.
      1. We make sure timestamp monotonicity is maintained.
      2. We generate clean partitions.
      Parameters:
      context -  
      pvName - The PV name
      stream -  
      extension -  
      extensionToCopyFrom -  
      Returns:
      eventsAppended  
      Throws:
      IOException -  

    • checkStream

      protected Event checkStream(String pvName, ETLContext context, ETLBulkStream bulkStream, Class<? extends ETLBulkStream> streamType) throws IOException
      Throws:
      IOException

    • bulkAppend

      public abstract boolean bulkAppend(String pvName, ETLContext context, ETLBulkStream bulkStream, String extension, String extensionToCopyFrom) throws IOException
      Append data in bulk skipping some of the per event checks.
      Parameters:
      pvName - The PV name
      context - The ETL context
      bulkStream - The ETL bulk stream
      extension -  
      extensionToCopyFrom -  
      Returns:
      boolean  
      Throws:
      IOException -  

    • shouldISwitchPartitions

      protected void shouldISwitchPartitions(BasicContext context, String pvName, String extension, Instant ts) throws IOException
      Should we switch to a new partition? If so, return the new partition, else return the current partition.
      Parameters:
      context -  
      pvName - The PV name
      extension -  
      ts - The epoch seconds
      Throws:
      IOException -  

    • shouldISkipEventBasedOnTimeStamps

      protected boolean shouldISkipEventBasedOnTimeStamps(Event event)
      Tell appendData if we should skip this event based on the last known event, current year of the destination file etc...
      Parameters:
      event -  
      Returns:
      Boolean  

    • preparePartition

      protected Path preparePartition(String pvName, EventStream stream, BasicContext context, String extension, String extensionToCopyFrom, Instant ts, Path pvPath, PathResolver pathResolver) throws IOException
      Prepare a new partition.
      Parameters:
      pvName - The PV name
      stream -  
      context -  
      extension -  
      extensionToCopyFrom -  
      ts - The epoch seconds
      pvPath -  
      Returns:
      pvPath  
      Throws:
      IOException -  

    • createNewWriter

      protected abstract EventFileWriter createNewWriter(String pvName, Path pvPath, EventStream stream) throws IOException
      Throws:
      IOException

    • updateStateBasedOnExistingFile

      protected abstract void updateStateBasedOnExistingFile(String pvName, Path pvPath) throws IOException
      Throws:
      IOException

    • closeStreams

      public void closeStreams() throws IOException
      Throws:
      IOException