DSet

DSet is a distributed data set. It is one of the central entity in Prajna. Please use the generic version of this class.

Instance members

Instance member	Description
`collectionWriteInitiated` Signature: ConcurrentDictionary<int,bool>
`ContentKey()` Signature: unit -> uint64	Set a content key for DSet that governs partition mapping, For two DSets that have the same content key, a single key will be mapped uniquely to a partition
`ContentKey()` Signature: unit -> unit	Set a content key for DSet that governs partition mapping, For two DSets that have the same content key, a single key will be mapped uniquely to a partition
`Length` Signature: int64	Get the number of key-values or blobs in DSet. .Length can be applied to either 1) source DSet (metadata is read via .LoadSource()), 2) intermediate DSet which is derived from source DSet, or 3) destination DSet after save operation has succeeded.
`Name()` Signature: unit -> string	Get or set the name of DSet
`Name()` Signature: unit -> unit	Get or set the name of DSet
`NumParallelExecution()` Signature: unit -> int	Maximum number of parallel threads that will execute the data analytic jobs in a remote container. If 0, the remote container will determine the number of parallel threads used according to its computation and memory resource available.
`NumParallelExecution()` Signature: unit -> unit	Maximum number of parallel threads that will execute the data analytic jobs in a remote container. If 0, the remote container will determine the number of parallel threads used according to its computation and memory resource available.
`SizeInBytes` Signature: int64	Get the size of all key-values or blobs in DSet

Static members

Static member	Description
`FolderRecursiveSeq(...)` Signature: (localFolderName:string * sPattern:string * sOption:SearchOption) -> seq<string * byte []> * uint64 ref * string []	Turn a local or network folder into seq to be fed into DSet.store sPattern, sOption is the search pattern and option used in Directory.GetFiles
`length(x)` Signature: x:DSet -> int64	The number of values in the DSet. This function can only be used for Source/Destination DSet that is persisted, it will return Int64.MinValue for other DSet.
`RetrieveFolderRecursive(...)` Signature: (localFolderName:string * o:seq<string * byte []>) -> int * uint64	store a seq (e.g., that is retrieved from DSet.toSeq() to a local folder.
`sizeInBytes(x)` Signature: x:DSet -> int64	The storage footprint of the DSet. This function can only be used for Source/Destination DSet that is persisted, it will return Int64.MinValue for other DSet.

Fork me on GitHub