Package edu.stanford.slac.archiverappliance.PlainPB

Class AppendDataStateData

java.lang.Object

edu.stanford.slac.archiverappliance.PlainPB.AppendDataStateData

public class AppendDataStateData
extends Object

Companion class to PlainPBStoragePlugin that handles the appending of event streams in a partition-aware fashion. This is used both by the engine and by ETL.

Author:

mshankar

Field Summary

Fields

Modifier and Type	Field	Description
protected Instant	lastKnownTimeStamp
protected short	previousYear

Constructor Summary

Constructors

Constructor	Description
AppendDataStateData(PartitionGranularity partitionGranularity, String rootFolder, String desc, Instant lastKnownTimestamp, PVNameToKeyMapping pv2key, PlainPBStoragePlugin.CompressionMode compressionMode)

Method Summary

Modifier and Type	Method	Description
boolean	bulkAppend(String pvName, ETLContext context, ETLBulkStream bulkStream, String extension, String extensionToCopyFrom)	Append data in bulk skipping some of the per event checks.
void	closeStreams()
int	partitionBoundaryAwareAppendData(BasicContext context, String pvName, EventStream stream, String extension, String extensionToCopyFrom)	Append data into PB files honoring partition boundaries switching into new partitions as we cross the boundary.
protected Path	preparePartition(String pvName, EventStream stream, BasicContext context, String extension, String extensionToCopyFrom, Instant ts, Path pvPath, PlainPBStoragePlugin.CompressionMode compressionMode)	Prepare a new partition.
protected boolean	shouldISkipEventBasedOnTimeStamps(Event event)	Tell appendData if we should skip this event based on the last known event, current year of the destination file etc...
protected void	shouldISwitchPartitions(BasicContext context, String pvName, String extension, Instant ts, PlainPBStoragePlugin.CompressionMode compressionMode)	Should we switch to a new partition? If so, return the new partition, else return the current partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

previousYear

protected short previousYear

lastKnownTimeStamp

protected Instant lastKnownTimeStamp

Constructor Details

AppendDataStateData

public AppendDataStateData(PartitionGranularity partitionGranularity,
 String rootFolder,
 String desc,
 Instant lastKnownTimestamp,
 PVNameToKeyMapping pv2key,
 PlainPBStoragePlugin.CompressionMode compressionMode)

Parameters:

partitionGranularity - partitionGranularity of the PB plugin.

rootFolder - RootFolder of the PB plugin

desc - Desc for logging purposes

lastKnownTimestamp - This is probably the most important argument here. This is the last known timestamp in this storage. If null, we assume time(0) for the last known timestamp.

pv2key - PVNameToKeyMapping

Method Details

partitionBoundaryAwareAppendData

public int partitionBoundaryAwareAppendData(BasicContext context,
 String pvName,
 EventStream stream,
 String extension,
 String extensionToCopyFrom)
                                     throws IOException

Append data into PB files honoring partition boundaries switching into new partitions as we cross the boundary.

1. We make sure timestamp monotonicity is maintained.
2. We generate clean partitions.

Parameters:

context -  

pvName - The PV name

stream -  

extension -  

extensionToCopyFrom -  

Returns:

eventsAppended  

Throws:

IOException

shouldISwitchPartitions

protected void shouldISwitchPartitions(BasicContext context,
 String pvName,
 String extension,
 Instant ts,
 PlainPBStoragePlugin.CompressionMode compressionMode)
                                throws IOException

Should we switch to a new partition? If so, return the new partition, else return the current partition.

Parameters:

context -  

pvName - The PV name

extension -  

ts - The epoch seconds

Throws:

IOException -  

closeStreams

public void closeStreams()

shouldISkipEventBasedOnTimeStamps

protected boolean shouldISkipEventBasedOnTimeStamps(Event event)

Tell appendData if we should skip this event based on the last known event, current year of the destination file etc...

Parameters:

event -  

Returns:

Boolean  

preparePartition

protected Path preparePartition(String pvName,
 EventStream stream,
 BasicContext context,
 String extension,
 String extensionToCopyFrom,
 Instant ts,
 Path pvPath,
 PlainPBStoragePlugin.CompressionMode compressionMode)
                         throws IOException

Prepare a new partition.

Parameters:

pvName - The PV name

stream -  

context -  

extension -  

extensionToCopyFrom -  

ts - The epoch seconds

pvPath -  

Returns:

pvPath  

Throws:

IOException -  

bulkAppend

public boolean bulkAppend(String pvName,
 ETLContext context,
 ETLBulkStream bulkStream,
 String extension,
 String extensionToCopyFrom)
                   throws IOException

Append data in bulk skipping some of the per event checks.

Parameters:

pvName - The PV name

context - The ETL context

bulkStream - The ETL bulk stream

extension -  

extensionToCopyFrom -  

Returns:

boolean  

Throws:

IOException -