salsa  0.4.0
 All Classes Functions Variables Typedefs Enumerations Pages
README.md
1 # SALSA - Scalable Adaptive Large Structures Analysis
2 
3 The goal of the project is to develop framework, that will distribute large job between workers and process it in parallel.
4 Framework should be able to adapt to different conditions in order to analyse data as effectively as possible.
5 
6 # Run via docker/podman
7 
8 > Note: One can subsitute command `podman` by `docker`
9 
10 ## Simple cluster using UDP discovery
11 
12 ### Cluster
13 
14 ```shell
15 podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa
16 ```
17 
18 ### Start monitoring (optional)
19 
20 ```shell
21 podman run -it --rm --net=host --entrypoint salsa-mon registry.openbrain.sk/salsa/salsa -u "tcp://*:5555"
22 ```
23 
24 ### Submit job
25 
26 ```shell
27 podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"
28 ```
29 
30 ## Simple cluster using GOSSIP discovery
31 
32 ### Discovery
33 
34 ```shell
35 podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n disovery
36 ```
37 
38 ### Redirector
39 
40 ```shell
41 podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr
42 ```
43 
44 ### Worker
45 
46 ```shell
47 podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n wk
48 ```
49 
50 ### Submit job
51 
52 ```shell
53 podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"
54 ```
55 
56 ### Redirector and worker
57 
58 ```shell
59 podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr+wk
60 ```
61 
62 ### Only redirector with debug message and publisher set
63 
64 ```shell
65 podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr --debug
66 ```
67 
68 ## Extra option as cmd parameter
69 
70 ```shell
71 podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr{ip:192.168.1.20,port:35000}
72 ```
73 
74 # Installation via yum/dnf
75 
76 Install `OpenBrain` repository
77 
78 ```shell
79 yum install yum-plugin-copr -y
80 yum copr enable obl/stable -y
81 ```
82 
83 Install salsa package
84 
85 ```shell
86 yum install salsa -y
87 ```
88 
89 # Develop
90 
91 ## Install all dependencies
92 
93 ```shell
94 yum install redhat-lsb cmake gcc-c++ zyre-devel spdlog-devel protobuf-devel jsoncpp-devel yaml-cpp-devel doxygen ncurses-devel ndm-devel help2man
95 ```
96 
97 ## Compile project
98 
99 ```shell
100 cd <project-dir>
101 scripts/make.sh
102 ```
103 
104 ### scripts/make.sh options
105 
106 | Option | Description |
107 | :-------- | :------------------------------------------------------------------------------------------- |
108 | `install` | Install after compilation. |
109 | `clean` | Destroy build folder before building. |
110 | `clang` | Use Clang compiler instead of system default. |
111 | `strict` | Enable all warnings (requires `clang`). Not `-Werror`! |
112 | `ninja` | Use Ninja build system instead of system default. |
113 | `doc` | Build documentation (via Doxygen) |
114 | `release` | Build as release. (This is used in pipeline.) Infers `doc` and `strict`. Warnings as errors. |
115 
116 `strict` option was created for development purposes, so you can compile with all warnings, while not failing build (no `-Werror`).
117 If you are contributing, make sure that you pass compilation with at least `release` option. Even better if you pass with `release clang` (thanks to Clang's `-Weverything`).
118 
119 ## Example running master-worker arch from developemnt
120 
121 ```shell
122 $SALSA_ROOT/build/src/salsa -c $SALSA_ROOT/etc/config.json -n rdr,wk --debug # Runs redirector and 1 worker. Beware! Every worker gets all cores of machine!
123 $SALSA_ROOT/build/src/salsa-feeder -u <submitter-url> -f <path/to/list> # Command list is one command per line. (<submitter-url> : tcp://localhost:41000)
124 $SALSA_ROOT/build/src/salsa-feeder -u <submitter-url> -t "cmd arg1 arg2:N" # Using template switch executable is "cmd arg1 arg2" and it is done "N" times
125 ```
126 
127 ## Publishing info about jobs
128 
129 Run before (on rdr)
130 
131 ```shell
132 export SALSA_PUB_URL=">tcp://localhost:5555"
133 ```
134 
135 ## Kill job with specific JOB ID
136 ```
137 salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL:<job_id>"
138 ```
139 Next command will work for `JOB_ID= E25633E7362F43FF8DFE32E3C525E223`
140 ```
141 salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL:E25633E7362F43FF8DFE32E3C525E223"
142 ```
143 
144 ## Kill all jobs
145 
146 ```
147 salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL_ALL"
148 ```