Building a real world, end user, open source application in Clojure

Moritz Heidkamp @DerGuteMoritz

Moritz Ulrich @the_kenny

bevuta.svg

euroclojure.png

about-us.svg

http://www.bevuta.com/

technical.png

library.jpg

Figure 4: lincoln's inn library by mariusz kluzniak / CC BY-NC-ND 2.0

enterprise.jpg

Figure 5: Just Another Office Building by Daniel Foster / CC BY-NC-SA 2.0

philosophy1.jpg

Figure 6: Philosophers Club by Todd Lappin / CC BY-NC 2.0

?

pepa.svg

real world

end user

open source application

built with Clojure and ClojureScript

What is Pepa?

paperwork.jpg

Figure 7: 27B-stroke-6! Bloody paperwork! by TheeErink / CC BY-NC-ND 2.0

What is Pepa?

screenshot-dashboard.png

What is Pepa?

screenshot-document.png

Architecture

pepa-architecture.svg

Architecture

com.stuartsierra/component

Architecture

pepa.core 

(defn make-system []
  (-> (component/system-map
       :config (config/make-component)
       :bus (bus/make-component)
       :db (component/using
             (db/make-component)
             [:config :bus])
       :file-page-extractor (component/using
                              (fpe/make-component)
                              [:config :db :bus])
       :page-renderer (component/using
                        (page-renderer/make-component)
                        [:config :db :bus])
       :page-ocr (component/using
                   (page-ocr/make-component)
                   [:config :db :bus])
       :web (component/using
              (web/make-component)
              [:config :db :bus])
       :smtp (component/using
               (smtp/make-component)
               [:config :db])
       :lpd (component/using
              (printing/make-lpd-component)
              [:config :db])
       :zeroconf (component/using
                   (zeroconf/make-component)
                   [:config]))
      (wrap-logging)))

Configuration

config.clj 

{:db {:host "10.233.1.2"
      :user "pepa"
      :dbname "pepa"}
 :web {:port 4035
       :host "localhost"
       :log-requests? false
       :default-page-dpi 150
       :poll {:timeout 30}}
 :rendering {:png {:dpi #{50 150}}}
 ...}

Configuration

pepa.config 

(defrecord Config []
  component/Lifecycle
  (start [component]
    (let [file (or (System/getenv "PEPA_CONFIG")
                   "config.clj")
          settings (load-file file)]
      (into component settings)))
  (stop [component]
    (->Config)))

Database

postgresql.svg

Metadata

Document contents

Full text search

Communication

Queue

Schema

schema.png

pepa.db

(defrecord Database [config datasource]
  component/Lifecycle
  (start [component]
    (log/info component "Starting database")
    (let [spec (:db config)
          cpds (doto (ComboPooledDataSource.)
                 ...)]
      (assoc component :datasource cpds)))

  (stop [component]
    (log/info component "Stopping database")
    (when datasource
      (log/info component "Closing DB connection")
      (.close datasource))
    (assoc component :datasource nil)))

pepa.db

Extends and re-exports parts of clojure.java.jdbc 

user> (go)
#<SystemMap>
user> (require '[pepa.db :as db])
nil


user> (db/query (:db system) "SELECT id FROM documents")
({:id 1} {:id 2})


user> (db/insert! (:db system) :documents {:title "Some document"})
{:id 3,
 :title "Some document",
 :modified nil,
 :document_date nil,
 :created #inst "2015-06-21T20:34:54.142-00:00",
 :notes nil,
 :file nil,
 :state_seq 6}

pepa.model

user> (require '[pepa.model :as m])
nil


user> (m/query-documents
       (:db system)
       '(and (tag "new") (like title "Invoice%")))

({:id 2, :title "Invoice 20150393"})


user> (m/add-tags! (:db system) 2 ["foo" "bar"])
true

pepa.model

user> (m/get-document (:db system) 2)
{:id 2,
 :title "Invoice 20150393",
 :created #inst "2015-06-15T11:20:16.454296000-00:00",
 :modified #inst "2015-06-15T11:34:49.607352000-00:00",
 :notes nil,
 :pages
 [{:id 1,
   :rotation 0,
   :render-status :processing-status/processed,
   :dpi #{50 150}}
  {:id 2,
   :rotation 0,
   :render-status :processing-status/processed,
   :dpi #{50 150}}],
 :tags ["new" "origin/web" "bar" "foo"],
 :document-date nil}

Storing files …

pepa.model

(defn store-file! [db attrs]
  (db/with-transaction [db db]
    (db/insert! db :files attrs)
    (db/notify! db :files/new)))

pepa.db

(defn notify!
  ([db topic]
   (notify! db topic nil))
  ([db topic data]
   (advisory-xact-lock! db topic)
   (bus/notify! (:bus db) topic data)))

… and processing them

file-page-extractor

page-renderer

page-ocr

pepa.processor.file-page-extractor

(defrecord FilePageExtractor [config db processor]
  IProcessor
  (next-item [component]
    "SELECT id, content_type, data, origin FROM files
     WHERE status = 'pending' ORDER BY id LIMIT 1")

  (process-item [component file]
    ;; Extract pages, broadcast corresponding :pages/new notifications
    ;; via pepa.bus and update status.
    ...)

  component/Lifecycle
  (start [component]
    (log/info component "Starting file page extractor")
    (assoc component
           :processor (processor/start component :files/new)))

  (stop [component]
    (log/info component "Stopping file page extractor")
    (when-let [processor (:processor component)]
      (processor/stop processor))
    (assoc component
           :processor nil)))

pepa.processor

(defn next-item* [component processor]
  (db/with-transaction [db (:db component)]
    (db/advisory-xact-lock! db (:notify-topic processor)))
  (db/query (:db component) (next-item component) :result-set-fn first))

(defn process-next [component processor]
  (let [{:keys [notify-chan control-chan]} processor]
    (if-let [item (next-item* component processor)]
      (do (process-item component item)
          (async/alts!! [control-chan] :default :continue))
      (async/alts!! [notify-chan control-chan]))))

(defn start [component notify-topic]
  (let [notify-chan (bus/subscribe (:bus component)
                                   notify-topic
                                   (async/sliding-buffer 1))
        control-chan (async/chan)
        processor (->Processor notify-topic notify-chan control-chan nil)
        worker (async/thread
                 (loop []
                   (case (first (process-next component processor))
                     (:stop nil) :done
                     (recur))))]
    (assoc processor
           :worker worker)))

Logging

(defprotocol ILogger
  (-log
    [logger component level throwable message]
    [logger component level message]))

(defrecord CTLLogger [config]
  ILogger
  (-log
    ([logger component level throwable message]
     (clojure.tools.logging/log
      (component-name component) level throwable message))
    ...))

Logging

user> (require '[pepa.log :as log])
nil
user> (log/info (:db system) "Testing Logging")
nil
user> (log/warn (:page-renderer system) "Hello Guys!")
nil

Output

13:15:02.714 INFO  [Database] Testing Logging
13:15:04.436 WARN  [PageRenderer] Hello Guys!

Logging

core.clj

(defn make-system []
  (-> (component/system-map ...)
      (wrap-logging)))

log.clj

(defn wrap-logging
  ([system logger]
   (let [all-components (remove (set (keys logger))
                                (keys system))]
     (-> system
         (component/system-using
          (zipmap all-components (repeat [::logger])))
         (assoc ::logger (component/using logger [:config])))))
  ([system]
   (wrap-logging system (make-ctl-logger))))

Printing

Virtual Network Printer

LPD and IPP

Minimal, taylored to Pepa

Open Source, EPL

HTTP API

Immutant

Compojure

Liberator

Transit

Frontend

ClojureScript

Om

Styling: Garden

Routing: Secretary

Development: Figwheel

Data Model

data.cljs

(defonce state
  (atom {:documents {}
         :navigation {:route :dashboard
                      :query-params {}}
         :tags {}
         :seqs {}}))

Data Model

cljs.user=> (:documents @pepa.data/state)

{2 {:id 2
    :title "Invoice 20150393"
    :created #inst "2015-06-15T11:20:16.454296000-00:00"
    :modified #inst "2015-06-15T11:34:49.607352000-00:00"
    :notes nil
    :pages [{:id 1
             :rotation 0
             :render-status :processing-status/processed
             :dpi #{50 150}}
            {:id 2
             :rotation 0
             :render-status :processing-status/processed
             :dpi #{50 150}}]
    :tags ["new" "origin/web" "bar" "foo"]
    :document-date nil}
 ...}

Routing

data.cljs

(defonce state
  (atom {..
         :navigation {:route :dashboard
                      :query-params {}}
         ..}))

navigation.cljs

(defn navigation-ref []
  (-> data/state
      (om/root-cursor)
      :navigation
      (om/ref-cursor)))

(secretary/defroute document-route "/document/:id" [id query-params]
  (om/update! (navigation-ref)
              {:route [:document (js/parseInt id)]
               :query-params query-params}))

Routing

components/root.cljs

(ui/defcomponent root-component [state owner]
  (render-state [_ _]
    ...
    [:main
     (match [(get-in state [:navigation :route])]
       [:dashboard] (om/build dashboard/dashboard state)
       [[:document id]] (om/build document/document id)
       ...)]
    ...))

More Stuff To Look Into

Long Polling

SMTP

Zeroconf

DB Schema Generation

Authentication + Authorization

Meta

Source: https://github.com/bevuta/pepa

License: AGPL

Copyright © 2015 bevuta IT GmbH

bevuta.svg

Thank you!

/

← →

Moritz Heidkamp - Building a real world, end user, open source application in Clojure