Re-ops

Functional Live-Infrastructure as Code

Created by Ronen Narkis / @narkisr settings

Operations

The process of managing computerized resources

by manipulating their state

Status quo

Complex

Slow to react

Hard to composes

Data is second class

Re-ops

Live environment for managing servers

Composed from re-core, re-mote, re-gent

No magic, functions and a dash of macros!

Re-mote

Remote automation

Data collection, publishing

Scheduled execution

Hosts

(def sandbox (Hosts. {:user "ronen"} ["host-a" ,,,,]))

(defprotocol Shell
  (ls [this target flags]))

(extend-type Hosts
   Shell
  (ls [this target flags]
   ; implementation
  ))

(defn listing [hs]
  (run (ls hs "/" "-la") | (pretty)))
	    

Pipelines

Series of functions (operations)

Data flows from one function to the next

Intermediate functions operate on data

Final function output, publish, block/run async etc..

We can compose pipelines (just functions)

Pipeline deconstruction

                            filtering             output
                                ^                   ^
(defn listing [hs]              |                   |
  (run (ls hs "/" "-la") | (pick successful) | (pretty)))
   |                     |
   |                     V
   |       [this, {:successful [], :failure []}
   V
expands to: (| (| (ls ..) (pick ..)) (pretty))

; pipe expansion
(let [[this' res] (ls hs)]
  (pick this' res))

	    

Publishing

; free uses free-script
(defn #^{:category :stats} ram-publish
  "RAM free and used percentage collection and publishing"
  [hs]
  (run (free hs)
     | (collect)
     | (publish (stock "Free RAM" :timeseries :free))
     | (publish (stock "Used RAM" :timeseries :used))))
	  

Dashboard

Stevedore

; expands into
; # base.clj:190
; ls / -la
(let [target "/" flags "-la"]
  (script ("ls" ~target ~flags)))

(defn free-script []
  (script
   (set! R @("free" "-m"))
   (if (not (= $? 0)) ("exit" 1))
   (pipe
     ((println (quoted "${R}")))
     ("awk" "'NR==2 { print $2 \" \" $3 \" \" $4 }'"))))
	  

Re-mote demo

Scheduling

; every day on 10 AM
(defn apt-jobs [hs]
  (watch :update (every-day 10) (fn [] (aptdate hs))))

; every 5 seconds
(defn stats-jobs [hs]
  (watch :cpu (seconds 5) (fn [] (cpu-publish hs))))

; Saturday 4 AM
(defn zfs-jobs [hs]
  (watch :scrub (at-day DateTimeConstants/SATURDAY 4)
    (fn [] (run (scrub hs "tank") | (email "scrub done")))))
	    

Time sequences

(defn seconds
  ([n f]
    (periodic-seq
       (t/plus (local-now) (t/seconds f)) (-> n t/seconds)))
  ([n]
    (periodic-seq (local-now) (-> n t/seconds))))
	    

Re-core

Manage VMs (AWS, DigitalOcean, KVM)

Persistent (Redis/ES)

Queue (Redis)

Pipeline/Protocol patterns

Systems

(def systems (Systems.))

(defn list
  "List available instances:
    (list) ; list all
    (list ip) ; list all with ip (running)
  "
  ([]
   (list identity))
  ([f]
   (run (ls systems) | (filter-by f) | (pretty))))
                     ^
                     |
                  [this, {:systems [...]}]
	    

System model

{
  :machine {
    :hostname "reops" :user "re-ops" :domain "local"
    :os :ubuntu-16.04 :cpu 2 :ram 1024
  }

  :kvm {
    :node :remote
  }

  :type "reops"
}
	    

Re-core Demo

Re-gent

Persistent connections (Zeromq)

Secure (CurveZMQ)

Distributed Clojure functions

Easy to deploy

; using serializable.fn
(def ^{:doc "adding one"} plus-one
  (s/fn [x] (+ 1 x)))

; run the function on the hosts
(call plus-one [1] hs)

	    

Re-gent Demo

Metrics

; using oshi
(get-in (read-metrics) [:operatingSystem :processes])

{"7695d7ecbd60484281ff2cd0af979cbe" 
  [{:group "re-ops",
    :groupID "1000",
    :kernelTime 420,
    :name "java",
    :parentProcessID 1,
    :path "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java",
    ...
   }]}
	    

Reloaded workflow

(ns user
  [clojure.tools.namespace.repl :refer (refresh refresh-all)]
  ....
 )

; create environment from scratch
(go)

; halt the environment
(stop)

; stop and go (when re-defining does not work)
(reset)

Reset

clojure.tools.namespace.repl refresh refresh-all

Works across threads

Reset all VARS

Atoms state lost

Re-defining preferred (protocols)

Future

Hosts queries

Re-gent functions in pipelines

Stream into riemann

FreeBSD support

Exposing pipeline end points (http)

Questions ?

Check https://github.com/re-ops

@narkisr

Settings

Select theme:
Black (default) - White - League - Sky - Beige - Simple
Serif - Blood - Night - Moon - Solarized