As build commands run they can output extra dependency information (e.g. More...

#include <deps_log.h>

Classes
struct	Deps

Public Member Functions
void	Close ()

const std::vector< Deps * > &	deps () const

	DepsLog ()

Deps *	GetDeps (Node *node)

Node *	GetFirstReverseDepsNode (Node *node)

LoadStatus	Load (const std::string &path, State state, std::string err)

const std::vector< Node * > &	nodes () const
	Used for tests.

bool	OpenForWrite (const std::string &path, std::string *err)

bool	Recompact (const std::string &path, std::string *err)
	Rewrite the known log entries, throwing away old data.

bool	RecordDeps (Node node, TimeStamp mtime, const std::vector< Node > &nodes)

bool	RecordDeps (Node node, TimeStamp mtime, int node_count, Node const *nodes)

	~DepsLog ()

Static Public Member Functions
static bool	IsDepsEntryLiveFor (const Node *node)
	Returns if the deps entry for a node is still reachable from the manifest.

Private Member Functions
bool	OpenForWriteIfNeeded ()
	Should be called before using file_.

bool	RecordId (Node *node)

bool	UpdateDeps (int out_id, Deps *deps)

Private Attributes
std::vector< Deps * >	deps_
	Maps id -> deps of that id.

FILE *	file_

std::string	file_path_

bool	needs_recompaction_

std::vector< Node * >	nodes_
	Maps id -> Node.

Friends
struct	DepsLogTest

Detailed Description

As build commands run they can output extra dependency information (e.g.

header dependencies for C source) dynamically. DepsLog collects that information at build time and uses it for subsequent builds.

The on-disk format is based on two primary design constraints:

it must be written to as a stream (during the build, which may be interrupted);
it can be read all at once on startup. (Alternative designs, where it contains indexing information, were considered and discarded as too complicated to implement; if the file is small than reading it fully on startup is acceptable.) Here are some stats from the Windows Chrome dependency files, to help guide the design space. The total text in the files sums to 90mb so some compression is warranted to keep load-time fast. There's about 10k files worth of dependencies that reference about 40k total paths totalling 2mb of unique strings.

Based on these stats, here's the current design. The file is structured as version header followed by a sequence of records. Each record is either a path string or a dependency list. Numbering the path strings in file order gives them dense integer ids. A dependency list maps an output id to a list of input ids.

Concretely, a record is: four bytes record length, high bit indicates record type (but max record sizes are capped at 512kB) path records contain the string name of the path, followed by up to 3 padding bytes to align on 4 byte boundaries, followed by the one's complement of the expected index of the record (to detect concurrent writes of multiple ninja processes to the log). dependency records are an array of 4-byte integers [output path id, output path mtime (lower 4 bytes), output path mtime (upper 4 bytes), input path id, input path id...] (The mtime is compared against the on-disk output path mtime to verify the stored data is up-to-date.) If two records reference the same output the latter one in the file wins, allowing updates to just be appended to the file. A separate repacking step can run occasionally to remove dead records.

Definition at line 68 of file deps_log.h.

Constructor & Destructor Documentation

◆ DepsLog()

DepsLog::DepsLog ( )

inline

Definition at line 69 of file deps_log.h.

References file_, and needs_recompaction_.

Referenced by Recompact().

◆ ~DepsLog()

DepsLog::~DepsLog ( )

Definition at line 47 of file deps_log.cc.

References Close().

Member Function Documentation

◆ Close()

void DepsLog::Close ( )

Definition at line 147 of file deps_log.cc.

References file_, and OpenForWriteIfNeeded().

Referenced by Recompact(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), and ~DepsLog().

◆ deps()

const std::vector< Deps * > & DepsLog::deps ( ) const

inline

Definition at line 105 of file deps_log.h.

References deps_.

Referenced by GetFirstReverseDepsNode(), Load(), Recompact(), RecordDeps(), and UpdateDeps().

◆ GetDeps()

DepsLog::Deps * DepsLog::GetDeps ( Node * node )

Definition at line 305 of file deps_log.cc.

References deps_, and Node::id().

Referenced by RecordDeps().

◆ GetFirstReverseDepsNode()

Node * DepsLog::GetFirstReverseDepsNode ( Node * node )

Definition at line 313 of file deps_log.cc.

References deps(), deps_, and nodes_.

◆ IsDepsEntryLiveFor()

bool DepsLog::IsDepsEntryLiveFor ( const Node * node )

static

Returns if the deps entry for a node is still reachable from the manifest.

The deps log can contain deps entries for files that were built in the past but are no longer part of the manifest. This function returns if this is the case for a given node. This function is slow, don't call it from code that runs on every build.

Definition at line 379 of file deps_log.cc.

References Edge::GetBinding(), and Node::in_edge().

Referenced by Recompact().

◆ Load()

LoadStatus DepsLog::Load	(	const std::string &	path,
		State *	state,
		std::string *	err )

Definition at line 154 of file deps_log.cc.

References deps(), State::GetNode(), Node::id(), kCurrentVersion, kFileSignature, kFileSignatureSize, kMaxRecordSize, LOAD_ERROR, LOAD_NOT_FOUND, LOAD_SUCCESS, METRIC_RECORD, needs_recompaction_, nodes_, platformAwareUnlink(), Node::set_id(), Truncate(), and UpdateDeps().

Referenced by BuildTest::RebuildTarget(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), and TEST_F().

◆ nodes()

const std::vector< Node * > & DepsLog::nodes ( ) const

inline

Used for tests.

Definition at line 104 of file deps_log.h.

References nodes_.

Referenced by RecordDeps(), and RecordDeps().

◆ OpenForWrite()

bool DepsLog::OpenForWrite	(	const std::string &	path,
		std::string *	err )

Definition at line 51 of file deps_log.cc.

References file_, file_path_, needs_recompaction_, and Recompact().

Referenced by BuildTest::RebuildTarget(), Recompact(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), TEST_F(), and TEST_F().

◆ OpenForWriteIfNeeded()

bool DepsLog::OpenForWriteIfNeeded ( )

private

Should be called before using file_.

When false is returned, errno will be set.

Definition at line 434 of file deps_log.cc.

References file_, file_path_, kCurrentVersion, kFileSignature, kMaxRecordSize, and SetCloseOnExec().

Referenced by Close(), RecordDeps(), and RecordId().

◆ Recompact()

bool DepsLog::Recompact	(	const std::string &	path,
		std::string *	err )

Rewrite the known log entries, throwing away old data.

Definition at line 326 of file deps_log.cc.

References Close(), deps(), deps_, DepsLog(), IsDepsEntryLiveFor(), METRIC_RECORD, nodes_, OpenForWrite(), platformAwareUnlink(), and RecordDeps().

Referenced by OpenForWrite().

◆ RecordDeps() [1/2]

bool DepsLog::RecordDeps	(	Node *	node,
		TimeStamp	mtime,
		const std::vector< Node * > &	nodes )

References nodes().

Referenced by Recompact().

◆ RecordDeps() [2/2]

bool DepsLog::RecordDeps	(	Node *	node,
		TimeStamp	mtime,
		int	node_count,
		Node const	nodes )

Definition at line 68 of file deps_log.cc.

References deps(), file_, GetDeps(), Node::id(), kMaxRecordSize, nodes(), OpenForWriteIfNeeded(), RecordId(), and UpdateDeps().

◆ RecordId()

bool DepsLog::RecordId ( Node * node )

private

Definition at line 400 of file deps_log.cc.

References file_, kMaxRecordSize, nodes_, OpenForWriteIfNeeded(), Node::path(), and Node::set_id().

Referenced by RecordDeps().

◆ UpdateDeps()

bool DepsLog::UpdateDeps	(	int	out_id,
		Deps *	deps )

private

Definition at line 389 of file deps_log.cc.

References deps(), and deps_.

Referenced by Load(), and RecordDeps().

Friends And Related Symbol Documentation

◆ DepsLogTest

friend struct DepsLogTest

friend

Definition at line 127 of file deps_log.h.

References DepsLogTest.

Referenced by DepsLogTest.

Member Data Documentation

◆ deps_

std::vector<Deps*> DepsLog::deps_

private

Maps id -> deps of that id.

Definition at line 125 of file deps_log.h.

Referenced by deps(), GetDeps(), GetFirstReverseDepsNode(), Recompact(), and UpdateDeps().

◆ file_

FILE* DepsLog::file_

private

Definition at line 119 of file deps_log.h.

Referenced by Close(), DepsLog(), OpenForWrite(), OpenForWriteIfNeeded(), RecordDeps(), and RecordId().

◆ file_path_

std::string DepsLog::file_path_

private

Definition at line 120 of file deps_log.h.

Referenced by OpenForWrite(), and OpenForWriteIfNeeded().

◆ needs_recompaction_

bool DepsLog::needs_recompaction_

private

Definition at line 118 of file deps_log.h.

Referenced by DepsLog(), Load(), and OpenForWrite().

◆ nodes_

std::vector<Node*> DepsLog::nodes_

private

Maps id -> Node.

Definition at line 123 of file deps_log.h.

Referenced by GetFirstReverseDepsNode(), Load(), nodes(), Recompact(), and RecordId().

The documentation for this struct was generated from the following files:

Classes

Public Member Functions

Static Public Member Functions

Private Member Functions

Private Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

◆ DepsLog()

◆ ~DepsLog()

Member Function Documentation

◆ Close()

◆ deps()

◆ GetDeps()

◆ GetFirstReverseDepsNode()

◆ IsDepsEntryLiveFor()

◆ Load()

◆ nodes()

◆ OpenForWrite()

◆ OpenForWriteIfNeeded()

◆ Recompact()

◆ RecordDeps() [1/2]

◆ RecordDeps() [2/2]

◆ RecordId()

◆ UpdateDeps()

Friends And Related Symbol Documentation

◆ DepsLogTest

Member Data Documentation

◆ deps_

◆ file_

◆ file_path_

◆ needs_recompaction_

◆ nodes_