Prajna

Extension member functions for DSet>

Generic distributed dataset. If storing the content of DSet to a persisted store, it is highly advisable that 'U used is either of .Net system type, or using a customized serializer/deserializer to serialize ('U)[]. Otherwise, the stored data may not be retrievable as the BinaryFormatter may fail to deserialize a class if the associated code has changed.

Functions for DSet<'U> when 'U represents a key-value pair with type 'K*'V

Extension member functions for DSet<'K*'V>

Generic distributed dataset. If storing the content of DSet to a persisted store, it is highly advisable that 'U used is either of .Net system type, or using a customized serializer/deserializer to serialize ('U)[]. Otherwise, the stored data may not be retrievable as the BinaryFormatter may fail to deserialize a class if the associated code has changed.

BlobMetadata used in Job Execution. The class gets exposed because of use in abstract function.

PrajnaCacheTye only kicks in for StorageType.RAM It is only used for in job DSet

ClientInfo contains information of a Prajna remote node and its daemon. Information below is exposed as F# is not capable of suppress comment on internal/private structure member.

Cluster represents a set of remote nodes in either a private cloud or a public cloud. It is the class that govern which remote nodes that services, contracts and data analytical jobs are running upon.

Verb-Noun pair that forms one Prajna message. Only advance programmer that need to write additional Prajna service should use this data structure

Noun to be used for message passing between Prajna nodes Only advance programmer that need to write additional Prajna service should use this data structure

Verb to be used for message passing between Prajna nodes Only advance programmer that need to write additional Prajna service should use this data structure

DParam, setting parameters for DSet mapping Base of PrajnaDistributed Object

DSet is a distributed data set. It is one of the central entity in Prajna. Please use the generic version of this class.

DStream is a distributed byte[] stream, a central entity in DSet.

Control the launch of container in Debug or Release mode.

A set of default parameter that controls Prajna execution behavior

Base of DStream/DSet, all exposed functions in this class is internal function to be executed during a data analytical jobs. They shoould not be used by programmer.

Represent Prajna Environment for running Prajna program

Remote execution roster, which contains a list of file (dlls, data, environment variables) that is required for the job

Enumeration class that controls how a remote container for the job/service is executed.

Enumeration class of constant of type of Load balancing algorithm supported

The configuration for a local cluster

An extension to GenericConn to process NetworkCommand GenericConn is an internal object which contains Send/Recv Components to process SocketAsyncEventArgs objects NetworkCommandQueue contains Send/Recv Components to process NetworkCommand objects The general pipeline looks like following For Recv: Network -> GenericConn (ProcessRecvGenericConn) -> NetworkCommandQueue (function from AddRecvProc) -> Application RecvAsync->queue CompRecv of GenericConn CompRecv of NetworkCommandQueue SocketAsyncEventArgs->NetworkCommand NetworkCommand->application To Queue of NetworkCommandQueue For Send: Network <- (SendAsync) GenericConn <- (ProcessSendGenericConn) NetworkCommandQueue <- Application CompSend of GenericConn CompSend of NetworkCommandQueue network<-SocketAsyncEventArgs SocketAsyncEventArgs<-NetworkCommand Application writes to NetworkCommand queue using ToSend

A Prajna CommandQueue that is resulted from accept()

A set of NetworkCommandQueue that are established via the Connect call Each connect is reference counted, so that the connect will shutdown when the reference reaches 0

Delegate that is called when some one connects in

Access state of RemoteContainer

Parameters related to Prajna remote running environment

Type of Storage used by DSet

Enumeration Class that controls the behavior of the remote container launch

Cache Service doesn't do anything itself. It only register a service with an AppDomain/Exe, so that when the client stops, the AppDomain/Exe continues to run and hold cached data.

Specify contract servers to be monitored, this class can not be serialized

Specify servers to be used to launch services, import and export contract

Govern the behavior of the default serialization to be used

For built in Prajna functions.

Infromation of distributed Function Provider To use distributed function,

Representing a single registered distributed function DistributedFunctionStore provides a central location for handling distributed functions.

Distributed function store with Async interface.

Representing a single registered distributed function

WorkerRoleEntryPoint can be extended to run a Prajna worker service. The worker service will extend the class WorkerRoleEntryPoint with the following functions: Please note that the extended class will be serialized to the remote end. It is crucial to consider the data footprint in serialization for efficient execution.

WorkerRoleInstance can be extended to run a Prajna worker service. Unlike WorkerRoleEntryPoint, WorkerRoleInstance is to be instantiated at the remote end so it is class member is not serialized.

WorkerRoleInstance can be extended to run a Prajna worker service. Unlike WorkerRoleEntryPoint, WorkerRoleInstance is to be instantiated at the remote end so it is class member is not serialized.

This class contains the parameter used to start a multithreaded WorkerRoleInstance. The class (and if you have derived from the class, any additional information) will be serialized and send over the network to be executed on Prajna platform. Please fill in all class members that are not default.

Manifested list of servers to be used to launch services, import and export contract This class is a local manifestation of ContractServersInfo that is to be used.

Specify servers to be used to launch services, import and export contract

ContractStore provides a central location for export/import contract

Infromation of distributed Function Provider To use distributed function,

Launching a PrajnaInstance on a group of remote nodes. serviceName: name of the serivce param: parameter to be called by OnStart serviceInstance: OnStart(param), OnStop(), Run(), IsRunning() call for the initiating of the service. You must derive serviceInstance from WorkerRoleInstance PrajnaInstance share similar characteristics with Prajnaservice, with characteristics below. 1. You can run Service on Azure VM or your own cluster in the data center. 2. You can host multiple worker role in a same remote program. 3. Service can interact (call each other) with each call being a function call. 4. You can perform Big data analytical tasks (map/filter/choose/collect, fold, map-reduce, sort, hash-join, cross-join, etc.) on data generated by Service 5. You can start Service with a parameter. 6. You can track debug output (log, stderr, stdout) of the remote service. But in addition, PrajnaInstance is 1. Initialied remotely (with a function that you have specified). As such, PrajnaInstance doesn't need to be serializable, and can contain heavy data element. 2. You may specify the number of thread to work on the PrajnaInstance (parameter specified in Start Parameter.

Running a empty service, when the service is running, cache data will be hold at remote container.

Manifested list of servers to be used to launch services, import and export contract This class is a local manifestation of ContractServersInfo that is to be used.

Specify servers to be used to launch services, import and export contract

ContractStore provides a central location for export/import contract

Govern the behavior of the default serialization to be used

Infromation of distributed Function Provider To use distributed function

Launching a PrajnaInstance on a group of remote nodes. serviceName: name of the serivce param: parameter to be called by OnStart serviceInstance: OnStart(param), OnStop(), Run(), IsRunning() call for the initiating of the service. You must derive serviceInstance from WorkerRoleInstance PrajnaInstance share similar characteristics with Prajnaservice, with characteristics below. 1. You can run Service on Azure VM or your own cluster in the data center. 2. You can host multiple worker role in a same remote program. 3. Service can interact (call each other) with each call being a function call. 4. You can perform Big data analytical tasks (map/filter/choose/collect, fold, map-reduce, sort, hash-join, cross-join, etc.) on data generated by Service 5. You can start Service with a parameter. 6. You can track debug output (log, stderr, stdout) of the remote service. But in addition, PrajnaInstance is 1. Initialied remotely (with a function that you have specified). As such, PrajnaInstance doesn't need to be serializable, and can contain heavy data element. 2. You may specify the number of thread to work on the PrajnaInstance (parameter specified in Start Parameter.

WorkerRoleInstance is extended to run a service. It is to be instantiated at the remote end so that it is class member does not have the constraint to be serializable.

WorkerRoleInstance is extended to run a service. It is to be instantiated at the remote end so that it is class member does not have the constraint to be serializable.

This class contains the parameter used to start a multithreaded WorkerRoleInstance. The class (and if you have derived from the class, any additional information) will be serialized and send over the network to be executed on Prajna platform. Please fill in all class members that are not default.

KV class implements a distributed key-value store.

Utility class to parse commandline argument. Each argument is in the form of -pattern param, and the programmer can supplied a default value for each parameter. Please note that the pattern is case insensitive prefix matched, so pattern -dir will match "-directory" "-DIRECT". Please design your commandline argument pattern with different prefix.

Compare byte[], note that the default comparison of byte[] is Reference Equality

Programmer will implementation CustomizedSerializerAction of Func to customarily deserialize an object

Programmer will implementation CustomizedSerializerAction of Action to customarily serialize an object

ExpandableBuffer wraps around a bytearray object that can be extended in both direction, front & back The valid bytes in the Buffer is between Head & Tail. The current implementation of expandableBuffer is not threadsafe, and should not be used in multiple thread environment.

Prajna Trace level, the lower the level, the more important of the message.

Logger

This class is created for initialization of Random(), in which the Random generator can be correlated as they have the same time signature. A salt is added to make sure that the resultant random number is different based on different salt value.

Construct a comparer that uses Object.ReferenceEquals to compare object.

Extensions to StreamBase to allow for custom serialization

Extension Methods for System.IO.Stream

Extension Methods for System.String

Construct a comparer to quick compare String*'T. The class is constructured to take a comparer of string, and a type T that supports equality, and construct a new comparer

UnitAction represent one function to be checked during the wake up of ThreadPool wait. For performance reason, it is important to only put light function in this loop. Any heavy operation should be forked to be executed on another thread/task.

Helper class for caching web object.

Provides logging interface and utility functions.

LogLevel

Provides logging interface and utility functions.

A component class provides a generic tool to build a processing pipeline using a threadpool It provides the following functionality: 1. An arbitrary object (an item of type 'T) is dequeue from a queue - a queue must have a BaseQ as a base class 2. The object is repeatedly "processed" until the processing says it is finished by arbirary piece of processing code The processing code returns a tuple of (bool, ManualResetEvent): (complete, event) When complete == true, then item has completed and new item is to be dequeued When event <> null, then processing cannot continue and must wait for event to fire before trying again There may be cases when complete is true, but event is not null, in these cases, processing waits for event to fire However, if complete is false, event must be non-null The processing may occur on its own thread or on a threadpool If event <> null, - If threadpool is being used, the component processing is removed from the threadpool work item queue and gets requeued once event fires - If processing is occuring on own thread, then that thread blocks for event to fire There is also support for multiple processing steps to take place in a single item

A class capable of reading/writing a buffer asynchronously using an underlying connection

A generic connection which processes SocketAsyncEventArgs

base class which maintains multiple GenericConn connections which - share a common pool of SocketAsyncEventArgs which are processed by a superclass along with - share a threadpool for receiving and sending

A class capable of reading/writing value types asynchronously using an underlying connection

A generic interface for a connection

DNS helper functions. Cannot instantiate class, only contains static members.

Abstract Class with static members for network utility functions

maintains multiple network connections

The base class for various concurrent queue structures with and without flow control logic all enqueue/dequeue methods return (success, waithandle) in all classes 1. GrowQ: No limit on queue growth 2. FixedLenQ: Growth limited by queue length 3. FixedSizeQ: Growth limited by queue "size", where "size" is sum of all element sizes

Helper class to build HTML page based on template for WCF services.

Helper class to form a HTML page

A Customized Tab Control

A Grid with both horizontal and vertical scroll bar

A Grid with Scrollbar, in which each element and column head of the Grid is a string

A WPF window that display a large text file

The collection of UI include a set of tabbed window and a trace log window

A window UI with a menu