Package edu.stanford.slac.archiverappliance.plain.parquet

Class ParquetBackedPBEventFileStream

java.lang.Object

edu.stanford.slac.archiverappliance.plain.parquet.ParquetBackedPBEventFileStream

All Implemented Interfaces:

ETLParquetFilesStream, Closeable, AutoCloseable, Iterable<Event>, RemotableOverRaw, ETLBulkStream, EventStream

public class ParquetBackedPBEventFileStream
extends Object
implements ETLParquetFilesStream, RemotableOverRaw

An EventStream implementation that reads data from one or more Parquet files.

This class serves two primary purposes:

1. Data Retrieval: It can stream events from a list of Parquet files, applying time-based filters using Parquet's predicate pushdown for efficient querying.
2. Optimized ETL: It implements ETLParquetFilesStream, allowing it to act as a logical concatenation of multiple source files. The ParquetETLInfoListProcessor uses this capability to combine smaller Parquet files (e.g., hourly) into larger ones (e.g., daily) without fully deserializing and re-serializing the data, significantly improving ETL performance.

See Also:

• ParquetPlainFileHandler
• ParquetETLInfoListProcessor
• ETLParquetFilesStream

Constructor Summary

Constructors

Constructor	Description
ParquetBackedPBEventFileStream(String pvName, Path path, ArchDBRTypes type)
ParquetBackedPBEventFileStream(String pvName, Path path, ArchDBRTypes type, ParquetInfo fileInfo)
ParquetBackedPBEventFileStream(String pvName, List<Path> paths, ArchDBRTypes type, Instant startTime, Instant endTime)
ParquetBackedPBEventFileStream(String pvName, List<Path> paths, ArchDBRTypes type, Instant startTime, Instant endTime, ParquetInfo fileInfo)

Method Summary

Modifier and Type	Method	Description
void	close()
RemotableEventStreamDesc	getDescription()
Event	getFirstEvent()
Event	getFirstEvent(BasicContext context)	Get the first event in this event stream.
ParquetInfo	getFirstFileInfo()	Get parquet first file info
List<Path>	getPaths()	Get parquet file paths
Iterator<Event>	iterator()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Details

ParquetBackedPBEventFileStream

public ParquetBackedPBEventFileStream(String pvName,
 Path path,
 ArchDBRTypes type)

ParquetBackedPBEventFileStream

public ParquetBackedPBEventFileStream(String pvName,
 Path path,
 ArchDBRTypes type,
 ParquetInfo fileInfo)

ParquetBackedPBEventFileStream

public ParquetBackedPBEventFileStream(String pvName,
 List<Path> paths,
 ArchDBRTypes type,
 Instant startTime,
 Instant endTime)

ParquetBackedPBEventFileStream

public ParquetBackedPBEventFileStream(String pvName,
 List<Path> paths,
 ArchDBRTypes type,
 Instant startTime,
 Instant endTime,
 ParquetInfo fileInfo)

Method Details

getFirstFileInfo

public ParquetInfo getFirstFileInfo()

Description copied from interface: ETLParquetFilesStream

Get parquet first file info

Specified by:

getFirstFileInfo in interface ETLParquetFilesStream

Returns:

First parquet file info

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

iterator

public Iterator<Event> iterator()

Specified by:

iterator in interface Iterable<Event>

getDescription

public RemotableEventStreamDesc getDescription()

Specified by:

getDescription in interface EventStream

Specified by:

getDescription in interface RemotableOverRaw

getPaths

public List<Path> getPaths()

Description copied from interface: ETLParquetFilesStream

Get parquet file paths

Specified by:

getPaths in interface ETLParquetFilesStream

Returns:

List of paths to parquet files

getFirstEvent

public Event getFirstEvent(BasicContext context)
                    throws IOException

Description copied from interface: ETLBulkStream

Get the first event in this event stream. If there are no events in this stream, return null.

Specified by:

getFirstEvent in interface ETLBulkStream

Parameters:

context - BasicContext

Returns:

Event return the first event, or null

Throws:

IOException -  

getFirstEvent

public Event getFirstEvent()