Welcome to JCache

Version: v0.9.5

Authors: Jim Rosenblum (jrosenblum@carelogistics.com).

Erlang, Distributable, In-Memory Cache

JSON-Query; Consistency Assist; and Simple, TCP Interoperability Protocol

Features

Cache entries are Map, Key, Value, [TTL], [Sequence]
Maps represent a name-space for Keys - similar to the notion of 'bucket' in other caching systems
Maps, Keys and Values can be any Erlang term
TTL is time-to-live in seconds
Consistency Assist through Sequence Numbers: An alternative API allows for a sequence-number parameter on the put/x, evict/x, match/x and remove/x operations. Operations whose sequence number is lower than the current, per-map max are disallowed thereby ensuring, for example, that stale puts do not overwrite newer ones due to the newer one beating the stale ones to the cache.
JSON Query Support
Query by JSON: When Values are JSON, evict_match/2, evict_all_match/1 and values_match/2 search or evict keys whose JSON value, at a location specificed by a java-style, dot-path string, equals the given value. That is, jc:values_match(bed, "id.type=3") would return all values for Keys in the bed 'Map' whose JSON value was an object with an "id":3 in the top-level.
Ad-hoc, Index Support: In order to support faster operations, (2-3 orders of magnitude), each map can have up to four, dot-path, strings configured for which jc will provide index support.
Auto Index Recognition - Frequently used JSON querries will be automatically detected and indexing initiated.
User Controlled Eviction
Map-level TTL: A job runs at configured intervals and removes items whose create-date is older than a map-specific, configured number of seconds.
Item-level TTL: PUTS can include a TTL which defines when the item should be evicted. Used for shorter TTLs, as an exception to a Map-level TTL, or when more precision is required than offered by the Map-level TTL.
Pub/Sub
Clients can subscribe to Map events for a specific key or for any key, and for write, delete or either operations
Clients can create and subscribe to arbitrary 'topics' and broadcast arbitrary messages under those topic names
Clients can subscribe to node-up and node-down events
Interopability: Binary string over TCP returning JSON
Bridge process that accepts messages from a client indicating cache operations, executes the cache operations and returns the results to the client. This has been used with JInterface to interoperate with CLOJURE and Java clients
Fine-grained logging via lager

Cache Functions (jc)

Create
put(Map, Key, Value, [TTLSecs]) -> {ok, Key} | {error, badarg}
put_all(Map, [{K,V},{K,V},...], [TTLSecs]) -> {ok, CountOfSuccessfulPuts} | {error, badarg}
Delete
evict(Map, Key) -> ok
evict_all_match(Criteria = "Json.Path.Match=Value") -> ok
evict_map_since(Map, Age_In_Secs) -> ok
evict_match(Map, Criteria = "Json.Path.Match=Value") -> ok
remove_items(Map, Keys) -> {ok, [{K, V}, ...]} for each {K, V} deleted.
Retrieve
get(Map, Key) -> {ok, Value} | miss
get_all(Map, [K1, K2, ...]) -> {ok, {Found=[{K1,V1},...], Misses=[K2,...]}}
key_set(Map) -> {ok, [K1, K2, ...]} for each Key in the Map
values(Map) -> {ok, [V1, V2, ...]} for each Value in the Map
values_match(Map, Criteria="JSon.Path.Match=Value") -> {ok, [{K1,V1}, {K2,V2}, ...]}
Flush
clear(Map) -> ok
flush() -> ok
flush(silent) -> ok, Does not send alerts to subscribers
Predicates
contains_key(Map, Key) -> true | false.
Meta
cache_nodes() -> {{active, [Node1,... ]}, {configured, [Node1,... ]}}
cache_size() -> {size, [{TableName, RecordCnt, Words}],...}
map_size(Map) -> {records, Count}
maps() -> {maps, [Map1, Map2,...]}
up() -> {uptime, [{up_at, String},{now, String}, {up_time, {D, {H, M, S}}}]

Consistency Supported Functions (jc_s)

Identical to the Create and Evict family of functions of the jc module (see above), except:

An additional sequence parameter, which is expected to be a monotonically incresing integer (with respect to a given Map), used to disalow "out of sequence" operations
Functions return {error, out_of_seq} if out of sequence operation is attempted
evict(Map, Key, Seq)
evict_all_match(Criteria = "Json.Path.Match=Value", Seq)
evict_match(Map, Criteria = "Json.Path.Match=Value", Seq)
put(Map, Key, Value, [TTLSecs], Seq)
put_all(Map, [{K,V},{K,V},...], [TTLSecs], Seq)
remove_items(Map, Keys, Seq)
Meta Functions
sequence() -> [{Map, Highest_Number},... ]
sequence(Map) -> Hightest_Number

Eviction Manager Functions (jc_eviction_manager)

set_max_ttl(Map, Secs) -> ok | {error, badarg}
get_max_ttls() -> [{Map, Secs}, ...]

Pub/Sub Functions (jc_psub)

map_subscribe(Pid, Map, Key|any, write|delete|any) -> ok | {error, badarg}
map_unsubscribe(Pid, Map, Key|any, write|delete|any) -> ok | {error, badarg}
client receives

{map_event, {Map, Key, delete}} or {map_event, {Map, key, write, Value} topic_subscribe(Pid, Topic, Value) -> ok | {error, badarg} topic_unsubscribe(Pid, Topic, Value) -> ok | {error, badarg} client receives: {topic_event, {Topic, Value}} topic_event(Topic, Value) -> ok * Broadcasts Value to all subscribers of Topic * topic_subscribe(Pid, jc_node_events, any) -> ok * subscribtes the user to node up and node down events:

{jc_node_events, {nodedown, DownedNode, [ActiveNodes],[ConfiguredNodes]}}

{jc_node_events, {nodeup, UppedNode, [ActiveNodes],[ConfiguredNodes]}}

Indexing Functions (jc_store)

start_indexing(Map, Path={bed,"menu.id"}) -> ok | {error, no_indexes_available} | {error, Term}
stop_indexing(Map, Path={bed,"menu","id"}) -> ok
indexes(Map) -> {indexes, [{{Map, Path}, Position},...]} for all indexes of given Map
indexes() -> {indexes, [{{Map, Path}, Position},...]} for all indexes

Interoperability: String protocol

This is a binary-encoded, string protocol used to provide socket-based interoperability with JC.

All messages to the cache system are string representations of a tuple, All messages form the caching system to the client are JSON

The protocol defines three message types: CONNECT, CLOSE and COMMAND all of which are binary strings consisting of an 8-byte size followed by the actual command messages.

Responses are also binary strings with an 8-byte size prefix.

The CONNECT command initiates a session,

M = <<"{connect,{version,\"1.0\"}}">>

Size is 25, so the CONNECT message is:

<<25:8, M/binary>> = 
<<25,40,58,99,111,110,110,101,99,116,32,123,58,118,101,
  114,115,105,111,110,32,49,46,48,125,41>>

The server will respond to a CONNECT command with either an error or the encoded version of {"version":" "1.0""}

<<15:8, <<"{\"version\":1.0}">> = 
<<15,123,34,118,101,114,115,105,111,110,34,58,49,46,48,125>>

The CLOSE command closes the socket, ending the session

M = <<"{close}">>

 Size is 7 so the CLOSE message is:
 <<7,123,99,108,111,115,101,125>>

COMMAND messages are string versions of the tuple-messages which jc_bridge uses only without the self() parameter. For example

{self(), {put, Map, Key, Value}} becomes 
{put, Map, Key, Value}

The return will be an encoded version of a JSON string. A client session might look as follows:

client:send("{put, evs, \"1\", \"{\\\"value:\\\":true}\"}")
<<"{\"ok\":1}">>

client:send("{get, evs, \"1\"}"),
    <<"{\"ok\":"{\\\"value\\\":true}\"}">>

Configuration

Application configuration is in sys.config which is heavily commented
Cookie, node-name and auto-restart of VM controlled by vm.args

Application Modules

jc_cluster
Simple, mnesia-based, cluster creation and management
jc, jc_s, jc_store, jc_eviction_manager
Caching operations, Key-specific and Map-level TTLs
jc_sequence
Singleton service enforcing strictly monotonic sequence numbers on jc_s operations
jc_analyzer
Analysis and indexing inititation of JSON query strings
jc_protocol
Socket processing of messages and Erlang -> JSON
jc_psub:
Pub / Sub of cache write and delete events
On-demand, ad-hoc topic events
Predefined, jc_node_events topic provides subscribers node-up and node-down notifications
jc_bridge
Server that acts as a proxy between an external process and jc functionality

Net Split/Join Strategy

Mnesia does not merge on its own when a node joins (returns) to a mesh of nodes. There are two situations where this is relevant:

* j_cache nodes start in a disconnected state so more than one initiates a new cluster and then, subsequently, those nodes join into one cluster; * A node drops out of the cluster due to some network glitch and then rejoins.

To handle these situations, whenever a cluster is created by a Node (node@123.45.67, for example), it creates a ClusterId - its Node name (node@123.45.67), for that cluster.

Given this ClusterId, we have the following strategy:

1. _Cluster Creation_: creates an initial ClusterId; 2. _Nodedown_: If the Node that created the cluster dissapears, a surviving Node changes the ClusterId such that ClusterId is now this new Node's name. In the case of a disconnected newtwork, one of the islands will have the original ClusterId Node dissapear, and it will create a new one as described. 3. _Nodeup_ Whenever a Node appears, an arbitary Node ensures that any Nodes that report a different ClusterId (different than the arbitrary Node's ClusterId) are killed to be restarted by the hearbeat application. If any Nodes required restarting, the entire cache is flushed or not per policy in config.sys.

Build Instructions

Ensure that Erlang 20 or higher is installed
Get the Source Code from Stash

[root@db01] git clone https://github.com/jr0senblum/jc.git

Edit the sys.config and vm.args files in ./config
- vm.args: Indicate the correct node names and cookie
- sys.config: Adjust prarameters as neccesary.
[root@db01] ./rebar3 release

or

[root@db01] ./rebar3 prod release

Documentation

[root@dbo1] ./rebar3 edoc

Generated by EDoc