JournalQueuingStrategy implements persistent temporary storing of SensorData into a filesystem structure. It controls contents of a selected directory. The contents consists of buffer files and an index file (journal). More...

#include <JournalQueuingStrategy.h>

Inheritance diagram for BeeeOn::JournalQueuingStrategy:

Data Structures
class	Entry
	An instance of Entry represents a single record in the FileBuffer. Such record contains a single SensorData instance. Moreover, name of the source buffer and offset after the parsed data is provided. More...

class	FileBuffer
	Representation of a persistent file buffer that contains entries holding the stored SensorData. More...

struct	FileBufferStat
	Helper struct with statistics collected during an inspection of a FileBuffer instance. More...

Public Member Functions
void	setRootDir (const std::string &path)
	Set the root directory where to create or use a storage.

Poco::Path	rootDir () const

void	setDisableGC (bool disable)
	Disable garbage-collection entirely. This is useful for debugging or in case of a serious bug in the GC code.

void	setNeverDropOldest (bool neverDrop)
	Disable dropping of oldest data. This is useful for debugging or in case of a serious bug in the dropping code.

void	setBytesLimit (int bytes)
	Set top limit for data consumed by the strategy in the filesystem. This counts all data files (also dangling buffers, locks and index). When setting to a negative value, it is treated as unlimited.

bool	overLimit (size_t bytes) const

void	setIgnoreIndexErrors (bool ignore)
	Configure behaviour of index loading. If the index contains broken entries, we can either fail quickly or ignore such errors and load as much as possible. Ignoring erros can lead to data loss in some cases but it is more probable that we would just peek & pop some already peeked data. The setting is applied only during JournalQueuingStrategy::setup().

virtual void	setup ()
	Setup the storage for the JournalQueuingStrategy. It creates new index or loads the existing one. All buffers present in the index are checked and registered for future use.

bool	empty () override
	The call might call precacheEntries() to load up to 1 entry. More...

void	push (const std::vector< SensorData > &data) override
	Push the given data into a separate buffer and update the index accordingly. In case of serious failure, the method can throw exceptions.

size_t	peek (std::vector< SensorData > &data, size_t count) override
	Peek the given count of data off the registered buffers starting from the oldest one. Calling this method is stable (returns the same results) until the pop() method is called. Index is not affected by this call. More...

void	pop (size_t count) override
	Pop the count amount of data off the registered buffers. It updates index accordingly.

Protected Types
typedef std::function< void(const std::string &name, size_t offset, Poco::Timestamp &newest)>	BrokenHandler

Protected Member Functions
Poco::Timestamp	initIndexAndScan (BrokenHandler broken)
	Performs all the necessary steps done when calling setup(). The given function is called when a broken buffer is detected. More...

void	initIndex (const Poco::Path &index)
	Initialize the journaling index either by creating a new empty one or by loading the existing one.

void	prescanBuffers (Poco::Timestamp &newest, BrokenHandler broken)
	Pre-scan all buffers in the index, check their consistency, collect some information (entries counts, errors, etc.) and update the index after any changes or fixes.

void	reportStats (const Poco::Timestamp &newest) const
	Report statistics about buffers.

void	inspectAndRegisterBuffer (const std::string &name, size_t offset, Poco::Timestamp &newest)
	Inspect buffer of the given name.

Journal::Ptr	index () const

std::string	writeData (const std::string &data, bool force=true) const
	Write data safely to a file and return its name.

bool	whipeFile (Poco::File file, bool usuallyFails=false) const
	Remove the given file if possible. If usuallyFails is true, than the method does not log errors.

void	collectReferenced (std::set< std::string > &referenced) const
	Collect all referenced buffers. Such buffers must not be deleted and should not be affected in anyway as they are suspects of peek()/pop() API.

bool	garbageCollect (const size_t bytes)
	Perform garbage collection to ensure that at least the given bytes amount of space is available for writing.

void	dropOldestBuffers (const size_t bytes)
	Drop oldest valid data to ensure that at least the given bytes amount of space is available for writing.

size_t	bytesUsed () const

size_t	bytesUsedAll () const

Poco::Path	pathTo (const std::string &name) const

size_t	precacheEntries (size_t count)
	The call ensures that there are up to count entries into the m_entryCache. If there is not enough data in the m_entryCache, more data is read from the appropriate buffer. This call helps to ensure that we do not read any data twice. Also, in case a buffer becomes unreadable for some reason, it would no be read again. The index is not affected by this call.

size_t	readEntries (std::function< void(const Entry &entry)> proc, size_t count)
	Read up to count entries sequentially from buffers. For each entry, call the given method proc. Buffers' offsets are being updated during this process. The method itself does not modify the index in anyway.

void	registerBuffer (const FileBuffer &buffer, const FileBufferStat &stat)
	Register the given buffer with the strategy. The index is not updated by this call.

Protected Member Functions inherited from BeeeOn::Loggable
void	setupLogger (Poco::Logger *logger=0) const

Poco::Logger &	logger () const

	Loggable (const ClassInfo &info)

	Loggable (const std::type_info &info)

Static Protected Member Functions
static std::string	tsString (const Poco::Timestamp &t)

Static Protected Member Functions inherited from BeeeOn::Loggable
static Poco::Logger &	forMethod (const char *name)

static Poco::Logger &	forClass (const ClassInfo &info)

static Poco::Logger &	forClass (const std::type_info &info)

template<typename T >
static Poco::Logger &	forInstance (const T *i)

static void	configureSimple (Poco::Logger &logger, const std::string &level)

static void	logException (Poco::Logger &logger, const Poco::Message::Priority priority, const Poco::Exception &e, const char *file, size_t line)

Additional Inherited Members
Public Types inherited from BeeeOn::QueuingStrategy
typedef Poco::SharedPtr < QueuingStrategy >	Ptr

Detailed Description

JournalQueuingStrategy implements persistent temporary storing of SensorData into a filesystem structure. It controls contents of a selected directory. The contents consists of buffer files and an index file (journal).

The JournalQueuingStrategy maintains 3 kinds of files:

buffers - files named after their SHA-1 checksum (Git-like) containing serialized SensorData instances in a line-oriented way with CRC32 protection per-record
index - index of buffer files and byte offsets into them implemented as a journal (mostly append only file)
locks - when writing a file at once to disk (mostly buffers), a temporary lock files are created

Writing data into the storage are controlled by the bytesLimit. If the limit is reached by all persisted files (both active or dangling), the JournalQueuingStrategy tries to garbage collect unused (dangling) files and if it does not succeed then it drops also valid data that were not peeked yet.

Member Function Documentation

size_t JournalQueuingStrategy::bytesUsed ( ) const

protected

Returns: amount of bytes occupied by active buffers and index.

size_t JournalQueuingStrategy::bytesUsedAll ( ) const

protected

Returns: amount of bytes occupied by all files related to the JournalQueuingStrategy (including lock files and dangling buffers).

bool JournalQueuingStrategy::empty ( )

overridevirtual

The call might call precacheEntries() to load up to 1 entry.

Returns: true if there is no buffer containing peekable data.

Implements BeeeOn::QueuingStrategy.

Journal::Ptr JournalQueuingStrategy::index ( ) const

protected

Returns: the underlying index.

Timestamp JournalQueuingStrategy::initIndexAndScan ( BrokenHandler broken )

protected

Performs all the necessary steps done when calling setup(). The given function is called when a broken buffer is detected.

Returns: timestamp of the newest detected buffered data.

bool JournalQueuingStrategy::overLimit ( size_t bytes ) const

Returns: true if an over limit is detected for the given amount of consumed space in bytes.

Path JournalQueuingStrategy::pathTo ( const std::string & name ) const

protected

Returns: path to the given name relative to the rootDir.

size_t JournalQueuingStrategy::peek	(	std::vector< SensorData > &	data,
		size_t	count
	)

overridevirtual

Peek the given count of data off the registered buffers starting from the oldest one. Calling this method is stable (returns the same results) until the pop() method is called. Index is not affected by this call.

It is not expected to peek large amounts of data because the peek() call caches all such data in RAM.

Implements BeeeOn::QueuingStrategy.

Path JournalQueuingStrategy::rootDir ( ) const

Returns: rootDir where the repository is located.

string JournalQueuingStrategy::tsString ( const Poco::Timestamp & t )

staticprotected

Returns: string representation of the given timestamp.

The documentation for this class was generated from the following files:

/var/lib/jenkins/workspace/beeeon-gateway-master/src/exporters/JournalQueuingStrategy.h
/var/lib/jenkins/workspace/beeeon-gateway-master/src/exporters/JournalQueuingStrategy.cpp

Data Structures

Public Member Functions

Protected Types

Protected Member Functions

Static Protected Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation