Journal implements a simple journaling principle in filesystem. A Journal instance represents an append-only persistent list of records. Appending is an atomic operation. We either append the whole record or append nothing. More...

#include <Journal.h>

Inheritance diagram for BeeeOn::Journal:

Data Structures
struct	Record
	A single record in journal. More...

Public Types
typedef Poco::SharedPtr< Journal >	Ptr

Public Member Functions
	Journal (const Poco::Path &file, double duplicatesFactor=3, size_t minimalRewritesSize=4096)

bool	createEmpty ()
	Create empty journal if it does not exists yet. More...

void	checkExisting (bool regularFile=true, bool writable=true) const
	Check for common situations that might be a symptom of an invalid setup. We check whether the underlying file is readable, (optionally) writable and (optionally) whether it is a regular file. If the file does not exists, we test the parent directory whether it is readable and (optionally) wriable. More...

void	load (bool recover=false)
	Load the journal from the underlying file. If the parameter recover is true, the method skips malformed entries (with invalid checkums). Otherwise, the method throws an exception.

void	load (std::istream &in, bool recover=false)
	Load the journal from the given stream. If the parameter recover is true, the method skips malformed entries (with invalid checkums). Otherwise, the method throws an exception. More...

void	checkConsistent () const
	Check that the RAM journal representation is equivalent to the persistent representation in the underlying file.

void	checkConsistent (std::istream &in) const
	Check that the RAM journal representation is equivalent to the representation readed from the given input stream.

void	append (const std::string &key, const std::string &value, bool flush=true)
	Append the key-value pair into the journal. More...

void	append (const Record &record, bool flush=true)
	Append the given record into the journal. If the parameter flush is true (default), the record is immediatelly persisted and flushed. Otherwise, it is written into a waiting list to be flush in a batch.

void	drop (const std::string &key, bool flush=true)
	Mark the given key as dropped. More...

void	drop (const std::set< std::string > &keys, bool flush=true)
	Mark all the given keys as dropped. The operation itself can be delayed when the parameter flush is false. Otherwise, all the keys are immediatelly persisted and flushed. Note that dropping multiple keys is NOT an atomic operation. We can end up with only few keys dropped in case of a serious failure.

void	flush ()
	Flush all records in the waiting list. If the current duplicates factor is high enough and the size of the journal is bigger than the minimal rewrite size, the deduplication is performed. In case of an I/O failure while deduplicating, it fallbacks to simple append with flush.

std::list< Record >	records () const

Poco::Nullable< std::string >	operator[] (const std::string &key) const

double	currentDuplicatesFactor () const

Protected Member Functions
void	parseStream (std::istream &in, std::list< Record > &records) const

void	parseStreamRecover (std::istream &in, std::list< Record > &records) const

void	appendDrop (const std::string &key, bool flush)

void	dropInPlace (std::list< Record > &records, const std::string &key) const

double	duplicatesFactor (const std::list< Record > &records) const

bool	overMinimalSize () const

std::list< Record >	recordsRaw () const

void	interpret (std::list< Record > &records) const

void	interpretAndFlush ()

void	appendFlush ()

void	rewriteAndFlush (const std::list< Record > &records)

std::list< Record > &	committed ()

std::list< Record > &	dirty ()

void	handleFailure (std::ostream &o) const

void	checkRecord (const Record &record) const

std::string	format (const Record &record, bool zeroSum=false) const

Record	parse (const std::string &line, size_t lineno) const

size_t	bytes (const std::list< Record > &records) const

Protected Member Functions inherited from BeeeOn::Loggable
void	setupLogger (Poco::Logger *logger=0) const

Poco::Logger &	logger () const

	Loggable (const ClassInfo &info)

	Loggable (const std::type_info &info)

Additional Inherited Members
Static Protected Member Functions inherited from BeeeOn::Loggable
static Poco::Logger &	forMethod (const char *name)

static Poco::Logger &	forClass (const ClassInfo &info)

static Poco::Logger &	forClass (const std::type_info &info)

template<typename T >
static Poco::Logger &	forInstance (const T *i)

static void	configureSimple (Poco::Logger &logger, const std::string &level)

static void	logException (Poco::Logger &logger, const Poco::Message::Priority priority, const Poco::Exception &e, const char *file, size_t line)

Detailed Description

Journal implements a simple journaling principle in filesystem. A Journal instance represents an append-only persistent list of records. Appending is an atomic operation. We either append the whole record or append nothing.

It is considered that the number of records is quite low (up to few kB). Each record consists of a key and value. The key is an identifier of some entity being changed. The value represents the change of its entity to be recorded. Appending a new value for an existing key means that the associated entity has been updated.

A journal is a sequential structure which gives it several properties:

efficient writes to a persistent storage (reduced seeking and erasing)
appending data cannot destroy the previous contents
reading from begining leads to a stable interpretation of its records

Journal provides a way how to recover after a system or power failure. Based on the journal contents, it is possible to reconstruct the most recent consistent state.

Each persisted record is protected by the CRC32 checksum to detect incomplete writes or broken underlying storage. Records with invalid checksum can be in some scenarios just skipped (leads to a data loss) in other scenarios we might need to recover in a more complex ways which are currently unsupported by the Journal class.

To avoid infinite grow of the journal, it can be internally deduplicated and thus rotated. After some time, several entities in the journal would be contained multiple times with different values. However, only the most recent value is valid for each entity. Those duplications can be avoid once upon a time or when consuming too much of storage.

There are two parameters that triggers rotations:

duplicates factor - average amount of key duplicates
minimal rewrite size - minimal size of the journal in bytes

If there more duplicates than the valud of duplicates factor and the journal is currently bigger then the minimal rewrite size, the journal automatically deduplicates itself (during the flush operation) and writes its shrinked version safely into the storage. The rewrite utilizes the SafeWriter to stay as safe as possible and prevent any or most data loss possibilities.

Member Function Documentation

void BeeeOn::Journal::append	(	const std::string &	key,
		const std::string &	value,
		bool	flush = `true`
	)

inline

Append the key-value pair into the journal.

See Also: Journal::append(const Record &, bool)

void Journal::checkExisting	(	bool	regularFile = `true`,
		bool	writable = `true`
	)		const

Check for common situations that might be a symptom of an invalid setup. We check whether the underlying file is readable, (optionally) writable and (optionally) whether it is a regular file. If the file does not exists, we test the parent directory whether it is readable and (optionally) wriable.

Exceptions

Poco::FileAccessDeniedException	- file is not readable
Poco::FileReadOnlyException	- file is not writable
Poco::InvalidArgumentException	- file is not a regular file

bool Journal::createEmpty ( )

Create empty journal if it does not exists yet.

Returns: true if created, false if it already exists

double Journal::currentDuplicatesFactor ( ) const

Computes the current duplicates factor of the journal main records (waiting records are not counted).

void BeeeOn::Journal::drop	(	const std::string &	key,
		bool	flush = `true`
	)

inline

Mark the given key as dropped.

See Also: Journal::drop(const std::set<std::string> &, bool)

void Journal::load	(	std::istream &	in,
		bool	recover = `false`
	)

Load the journal from the given stream. If the parameter recover is true, the method skips malformed entries (with invalid checkums). Otherwise, the method throws an exception.

CAUTION: This operation may lead to have an inconsistent journal state between RAM and the underlying file when loading from another (underlated) data source.

Nullable< string > Journal::operator[] ( const std::string & key ) const

Returns: value of record by the given key or null is it is not in the journal

list< Journal::Record > Journal::records ( ) const

Returns: current state of each record, only the most recent records per key are returned this way (thus we have no access to the history by this method).

The documentation for this class was generated from the following files:

/var/lib/jenkins/workspace/beeeon-gateway-master/src/util/Journal.h
/var/lib/jenkins/workspace/beeeon-gateway-master/src/util/Journal.cpp

Data Structures

Public Types

Public Member Functions

Protected Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation