Hacker News

The program is the database is the interface

2025-03-0814:3120461www.scattered-thoughts.net

I do my accounts each year with a simple script. Something like this: (ns accounts (:require [clojure.string :as str] [clojure.pprint :as pp])) ;; converted from statement.csv (def txs [{:date #inst…

I do my accounts each year with a simple script. Something like this:

(ns 
accounts

 (
:require 
 [clojure.string 
:as 
str]

 [clojure.pprint 
:as 
pp]))

;; converted from statement.csv

(def 
txs

 [{
:date #inst "2022-05-13T11:01:56.532-00:00"

:amount -3.30

:text 
"Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"
}

 {
:date #inst "2022-05-12T10:41:56.843-00:00"

:amount -3.30

:text 
"Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"
}

 {
:date #inst "2022-05-12T00:01:03.264-00:00"

:amount -72.79

:text 
"Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"
}

 {
:date #inst "2022-05-10T10:33:04.011-00:00"

:amount -20.00

:text 
"e-Transfer to: John Smith"
}

 {
:date #inst "2022-05-11T17:12:43.098-00:00"

:amount -90.00

:text 
"Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"
}])

(def 
date->tag

 {
#inst "2022-05-12T00:01:03.264-00:00" :things
})

(def 
text->tag 
 {
"Coffee" 
:eating-out

"Range Physio" 
:medical
})

(defn 
tx->tag 
[tx]

 (or

 (date->tag (
:date 
tx))

 (first 
 (for [[text tag] text->tag

:when 
(str/includes? (
:text 
tx) text)]

 tag))))

(def 
txs-with-tags

 (vec 
 (for [tx txs]

 (assoc tx 
:tag 
(tx->tag tx)))))

(def 
total-per-tag

 (reduce 
 (fn [totals tx]

 (update-in totals [(
:tag 
tx)] #(+ (
:amount 
tx) (or % 
0
))))

 {}

 txs-with-tags))

(def 
untagged 
 (vec

 (for [tx txs-with-tags

:when 
(nil? (
:tag 
tx))]

 tx)))

(pp/pprint 
 [[
:untagged 
untagged]

 [
:total-per-tag 
total-per-tag]])

There are many things about this which are nice.

  • It's just a single file - easy to backup and version control.
  • It's composable - I can easily write code to answer unexpected questions (eg how much did I spend on currency conversions this year) or use the computed data in other calculations (eg runway projections).
  • It's easy - I just pretty-printed the data-structures I was already using instead of having to build a UI.

But the workflow isn't always great. When I run the code above, I see:

> clj accounts.clj

[[
:untagged

 [{
:date #inst "2022-05-10T10:33:04.011-00:00"
,

:amount -20.0
,

:text 
"e-Transfer to: John Smith"
,

:tag nil
}]]

 [
:total-per-tag

 {
:eating-out -3.3
, 
:things -72.79
, 
:medical -90.0
, 
nil -20.0
}]]

That transfer to John Smith isn't covered by any of the tagging rules. So I select the date, switch to the editor window and paste the date into the date->tag definition:

(def 
date->tag

 {
#inst "2022-05-12T00:01:03.264-00:00" :things

#inst "2022-05-10T10:33:04.011-00:00" :eating-out
})

Now I see:

> clj accounts.clj

[[
:untagged 
[]]

 [
:total-per-tag

 {
:eating-out -23.3
, 
:things -72.79
, 
:medical -90.0
}]]

Multiply this by a thousand transactions and it becomes tedious.

Pretty-printing also makes it difficult to decide the amount of detail I should print. Sometimes I want to see which transactions contribute to each tag total. But if I always print them all then it's hard to see the totals themselves without a lot of scrolling.

It's also hard to share this workflow with someone non-technical. I have to setup and maintain the correct environment on their machine, teach them how to use a text editor to change the tagging rules, how to interpret syntax errors, how to use git to share changes etc.

I could solve these kinds of problems by writing a web app and storing txs, date->tag and text->tag in a database.

Then I could put controls on the transaction itself that allow changing the tag in place. And I could add an expandable section next to each total, so that it's easy to see the transactions for that tag when I want to but they don't take up space by default.

Plus sharing becomes trivial - everyone has a web browser.

But this is a big leap in effort:

  • A database introduces a different data model - I have to translate between database data-types and my programming languages native data-types.
  • A database introduces a different language, or at least a new api that differs from the way I access native data-structures.
  • Rather than using a familiar text editor to read and change data, I can only access it via some query language or api.
  • I have to shuffle data back and forth between database and program at the correct times.
  • It isn't easy to version control the contents of the database.
  • Before I can add interactive elements, I have to map my existing data-structures to gui elements (eg to html and css) instead of just printing them.
  • I have to add an extra layer of state management to manage the state of the gui itself.
  • All of this work has to be repeated whenever I have a new question I want to answer.

None of this is insurmountable, but it's definitely much more work than the original script.

So a simple script is low-effort but produces a low-quality experience. And a custom app can produce a high-quality experience but is high-effort.

So I made a thing.

It is very hacky and easy to break. But it will hang together just long enough to convey the idea.

It's kind of like a notebook. There are cells where you can write code and the resulting values will be nicely rendered:

;; converted from statement.csv

(def txs

 [{:date #inst "2022-05-13T11:01:56.532-00:00"

 :amount -3.30

 :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}

 {:date #inst "2022-05-12T10:41:56.843-00:00"

 :amount -3.30

 :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}

 {:date #inst "2022-05-12T00:01:03.264-00:00"

 :amount -72.79

 :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"}

 {:date #inst "2022-05-10T10:33:04.011-00:00"

 :amount -20.00

 :text "e-Transfer to: John Smith"}

 {:date #inst "2022-05-11T17:12:43.098-00:00"

 :amount -90.00

 :text "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"}])

(defs date->tag

 {#inst "2022-05-12T00:01:03.264-00:00" :things})

(defs keyword->tag

 {"Coffee" :eating-out

 "Range Physio" :medical})

Funtions render a little differently.

(defn text->tag [text]

 (first

 (for [[keyword tag] keyword->tag

 :when (clojure.string/includes? text keyword)]

 tag)))

If you type "Joe's Coffee Hut" (including the "!) into the textbox above and hit the text->tag button, you'll see the result of running the text->tag function on that input.

Functions aren't limited to just returning values though. They can modify the values stored in other cells:

(def txs-with-tags

 (vec 
 (for [tx txs]

 (assoc tx 
 :tag (or

 (date->tag (:date tx))

 (text->tag (:text tx)))

 :actions [(fn ignore []

 (edit! 'date->tag assoc (:date tx) :ignore))

 (fn tag [tag] 
 (edit! 'date->tag assoc (:date tx) tag))]))))

If you type :eating-out into the one of the textboxes above and then hit the tag button, it will change the date->tag cell to contain an entry for the date of that transaction with the tag :eating-out.

And then any downstream cells will update, so you'll see the tag for that transaction change to :eating-out.

These actions are just values so they can be passed around like any other value. For example, if I make a list of untagged transactions then I'll still have access to the same actions:

(def untagged

 (vec

 (for [tx txs-with-tags

 :when (nil? (:tag tx))]

 tx)))

We can also attach metadata to values to control how they render:

(def hidden-vec

 (with-meta

 [1 2 3]

 {:preimp/hidden true}))

If you click on the + above it will reveal the contents of the vec. This is useful for controlling the default level of detail.

(defn hidden [v]

 (with-meta v {:preimp/hidden true}))

(def total-per-tag

 (reduce 
 (fn [totals tx]

 (if (= :ignore (:tag tx))

 totals

 (update-in totals [(:tag tx)] 
 (fn [total+txs]

 (let [[total txs] (or total+txs [0 (hidden [])])]

 [(+ total (:amount tx))

 (conj txs (update-in tx [:actions] hidden))])))))

 {}

 txs-with-tags))

You can click on a + above to reveal the transactions for that tag.

The demo on this page is ephemeral - you could do something similar in many notebook environments using mutable data-structures.

But the preimp repo contains a server which persists the entire history of the notebook to disk, and also syncs changes between different clients to allow (coarse-grained) collaborative editing.

The server also allows reading and writing cell values over http. Here's the script that I use to upload my bank statements:

(ns 
wise

 (
:require 
[clj-http.client 
:as 
client]

 [clojure.data.json 
:as 
json]))

(def 
endpoints 
{

;; FILL ME IN

})

(defn 
api-get 
[user path]

 (let [domain (get-in endpoints [user 
:wise-domain
])

 url (str domain 
"/" 
path)

 response (client/get url {
:headers 
{
"Authorization" 
(str 
"Bearer " 
(get-in endpoints [user 
:wise-token
]))}})]

 (assert (= 
200 
(
:status 
response)))

 (json/read-str (
:body 
response))))

(def 
now 
(java.time.Instant/now))

(defn 
get-transactions 
[user]

 (into []

 (for [profile (api-get user 
"v2/profiles"
)

:let 
[profile-id (get profile 
"id"
)]

 balance (api-get user (str 
"v4/profiles/" 
profile-id 
"/balances?types=STANDARD"
))

:let 
[balance-id (get balance 
"id"
)

 statement (api-get user (str 
"/v1/profiles/" 
profile-id 
"/balance-statements/" 
balance-id 
"/statement.json?intervalStart=2022-01-01T00:00:00.000Z&intervalEnd=" 
now 
"&type=COMPACT"
))]

 transaction (get statement 
"transactions"
)]

 transaction)))

(defn 
update-preimp 
[user]

 (let [transactions (get-transactions user)

 cell-name (symbol (str (name user) 
"-wise-transactions"
))

 cell-value (pr-str 
`
(
~'
defs 
~
cell-name 
~
transactions))

 body (json/write-str

 {
:cell-id 
(get-in endpoints [user 
:cell-id
])

:value 
cell-value})]

 (client/put

 (get-in endpoints [user 
:preimp-domain
])

 (merge (get-in endpoints [user 
:preimp-headers
]) {
:body 
body}))))

(defn 
update-preimp-dev 
[_]

 (update-preimp 
:sandbox
))

(defn 
update-preimp-prod 
[_]

 (update-preimp 
:jamie
)

 (update-preimp 
:cynthia
))

Finally, you can export a preimp notebook to produce a perfectly valid clojurescript program.

You can run this program in the repl:

> clj -M -m cljs.main -i ./exported.cljs --repl --repl-env node 
preimp.exported=> (require 
'
clojure.pprint)

nil

preimp.exported=> (clojure.pprint/pprint txs-with-tags)

[{
:date #inst "2022-05-13T11:01:56.532-00:00"
,

:amount -3.3
,

:text

"Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"
,

:tag :coffee
,

:actions

 [
#object
[preimp$exported$iter__897_$_ignore]

#object
[preimp$exported$iter__897_$_tag]]}

 {
:date #inst "2022-05-12T10:41:56.843-00:00"
,

:amount -3.3
,

:text

"Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"
,

:tag :coffee
,

:actions

 [
#object
[preimp$exported$iter__897_$_ignore]

#object
[preimp$exported$iter__897_$_tag]]}

 {
:date #inst "2022-05-12T00:01:03.264-00:00"
,

:amount -72.79
,

:text 
"Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"
,

:tag :things
,

:actions

 [
#object
[preimp$exported$iter__897_$_ignore]

#object
[preimp$exported$iter__897_$_tag]]}

 {
:date #inst "2022-05-10T10:33:04.011-00:00"
,

:amount -20
,

:text 
"e-Transfer to: John Smith"
,

:tag :eating-out
,

:actions

 [
#object
[preimp$exported$iter__897_$_ignore]

#object
[preimp$exported$iter__897_$_tag]]}

 {
:date #inst "2022-05-11T17:12:43.098-00:00"
,

:amount -90
,

:text

"Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"
,

:tag :medical
,

:actions

 [
#object
[preimp$exported$iter__897_$_ignore]

#object
[preimp$exported$iter__897_$_tag]]}]

nil

This tiny extension to the notebook model allows writing simple ugly crud apps with very little effort. You simply provide the logic and preimp gives you storage, sharing and UI for free. But the result is not an opaque image, nor is it tied to the preimp environment - you can export everything into a regular script and run it in the repl.

TODO rest of this section

required: rich data model, which can be roundtripped through text some way of attaching metadata to values, to control their rendering without interfering with execution in the repl

needed: no declaration order no mutable environment no side-effects (other than edit!) integrity constraints perf (compiler latency) (thoughput only needs to be better than spreadsheet)

extensions: much more ui options selection (+ actions on the side) dropdowns understand types / destructuring render values as copyable text (cf ?) undo/vc (have whole history in db) live repl, sandboxing distribution - single binary, single-file db, optional server (like fossil)

research: finer-grained collaboration (what data model?) bidirectional editing / provenance

We have to do the other demo too. You know the one.

(defs next-id 2)

(defs todos 
 {0 {:text "make a cool demo" :status :done}

 1 {:text "step 2: ???" :status :todo}})

(defn new-todo [text] 
 (edit! 'todos assoc next-id {:text text :status :todo})

 (edit! 'next-id inc))

(defn named [name fn]

 (with-meta fn {:preimp/named name}))

(defn status-toggle [id]

 (let [status (get-in todos [id :status])

 other-status (case status

 :done :todo

 :todo :done)]

 (named (name status)

 (fn [] (edit! 'todos assoc-in [id :status] other-status)))))

(defs filter :all)

(def toggle-filter

 (named (str "viewing " (name filter))

 (fn [] (edit! 'filter #(case filter 
 :all :todo

 :todo :done

 :done :all)))))

(def filtered-todos

 (vec

 (for [[id todo] todos

 :when (#{:all (:status todo)} filter)]

 [(:text todo)

 (hidden [(fn set-text [text]

 (edit! 'todos assoc-in [:id :text] text))

 (status-toggle id)

 (fn delete []

 (edit! 'todos dissoc id))])])))

(def app

 (vec

 (apply list

 new-todo

 toggle-filter

 filtered-todos)))

TODO spreadsheets don't have state problem, but not interactive (at least in same way)


Read the original article

tosh

Karma: 173187
...
 @Hacker__News
@hacker._news

Comments

  • By hilti 2025-03-0818:199 reply

    What the author demonstrates here is a powerful principle that dates back to LISP's origins but remains revolutionary today: the collapse of artificial boundaries between program, data, and interface creates a more direct connection to the problem domain.

    This example elegantly shows how a few dozen lines of Clojure can replace an entire accounting application. The transactions live directly in the code, the categorization rules are simple pattern matchers, and the "interface" is just printed output of the transformed data. No SQL, no UI framework, no MVC architecture - yet it solves the actual problem perfectly.

    The power comes from removing indirection. In a conventional app, you have: - Data model (to represent the domain) - Storage layer (to persist the model) - Business logic (to manipulate the model) - UI (to visualize and interact)

    Each boundary introduces translation costs, impedance mismatches, and maintenance burden.

    In the LISP approach shown here, those boundaries disappear. The representation is the storage is the computation is the interface. And that direct connection to your problem is surprisingly empowering - it's why REPLs and notebooks have become so important for data work.

    Of course, there are tradeoffs. This works beautifully for personal tools and small-team scenarios. It doesn't scale to massive collaborative systems where you need rigid interfaces between components. But I suspect many of us are solving problems that don't actually need that complexity.

    I'm reminded of Greenspun's Tenth Rule: "Any sufficiently complicated program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." The irony is that by embracing LISP principles directly, you can often avoid building those complicated programs in the first place.

    Rich Hickey's "Simple Made Easy" talk explores this distinction perfectly - what the industry calls "easy" (familiar tools and patterns) often creates accidental complexity. The approach shown here prioritizes simplicity over easiness, and the result speaks for itself.

    • By jahewson 2025-03-0820:00

      Over time I’ve come to see LISP less as the natural collapse of artificial boundaries but the artificial collapse of natural ones. Where and how data is stored is a real concern, but where and how the program is stored isn’t. Security boundaries around data and executable code are of paramount importance. Data storage concerns don’t benefit from being mixed with programming language concerns but from computer and storage architecture concerns (eg column stores).

      In toy programs, such as this one, those concerns can all be discarded, so LISP is a good fit. But in serious systems it’s soon discovered that what is offered is in fact “simplistic made easy”. That’s not to say that traditional systems don’t suffer from all the ills Hickey diagnoses in them, but that we differ on what the cure is.

    • By jamii 2025-03-0820:42

      > it solves the actual problem perfectly

      The whole post was about how that doesn't solve the problem perfectly - there is no way to interactively edit the output.

      > by embracing LISP principles directly

      This could just as easily have been javascript+json or erlang+bert. There's no lisp magic. The core idea in the post was just finding a way for code to edit it's own constants so that I don't need a separate datastore.

      Eventually I couldn't get this working the way I wanted with clojure and I had to write a simple language from scratch to embed provenance in values - https://news.ycombinator.com/item?id=43303314.

    • By cogman10 2025-03-0820:221 reply

      > It doesn't scale to massive collaborative systems where you need rigid interfaces between components. But I suspect many of us are solving problems that don't actually need that complexity.

      Here's the issue. Starting out you almost certainly don't need that rigid interface. However, the longer the app grows the more that interface starts to matter and the more costly retrofitting it becomes.

      The company I currently worked at started out with a "just get it done" approach which lead to things like any app reaching into any database directly just to get what it needs. That has created a large maintenance issue that to this day we are still trying to deal with. Modifying the legacy database schema in any way takes multiple months of effort due to how it might break the 20 systems that reach into it.

      • By crq-yml 2025-03-0822:52

        My take on what the issue is, is primarily in the ramifications of Conway's law and how our social structures map to systems.

        When the system is small, it makes a great deal of sense to be an artisan and design simple automations that work for exactly that task, which for the most common things is always supported by any production-oriented programming environment - there's a lot of ways in which you can't go wrong because the problem is so small relative to the tools that any approach will crush it. "Just get it done" works because no consequence is felt, and on the time scale of "most businesses fail within five years", it might never be.

        When it's large, everyone would prefer to defer to a common format and standard tools. The problems are now complex, have to be discussed and handled by many people, they need documentation and clear boundaries on roles and responsibilities. But common formats and standards are a pyramid of scope creep - eventually it has to support everyone - and along the way, monopolistic organizations vie for control over it in hopes of selling the shovels and pickaxes for the next gold rush. So we end up with a lot of ugly compatibility issues.

        In effect, the industry is always on this treadmill of hacking together a simple thing, blowing out the complexity, then picking up the pieces and reassembling them into another, slightly cleaner iteration.

        Maintenance can be done successfully - there are always examples of teams and organizations that succeed - but like with a lot of infrastructure, there's an investment bias towards new builds.

    • By phkahler 2025-03-0822:45

      >> No SQL, no UI framework, no MVC architecture - yet it solves the actual problem perfectly.

      No SQL but in it's place is some code. The point of SQL was to standardize a language for querying data. This is just using a language other than the standard. A UI is a way for people to avoid writing code.

      Sure doing your own custom thing results in something easy for the programmer. Nothing new about that.

    • By rlupi 2025-03-094:371 reply

      > Of course, there are tradeoffs. This works beautifully for personal tools and small-team scenarios. It doesn't scale to massive collaborative systems where you need rigid interfaces between components. But I suspect many of us are solving problems that don't actually need that complexity.

      Spreadsheets. If you squint the right way, they embody the lisp principles for the non-programmers' world.

      • By kazinator 2025-03-0918:53

        MS Excel has IF(this, then, else), AND(expr, expr, ...), OR(expr, expr, ...) and more recently LAMBDA.

    • By deterministic 2025-03-143:18

      > "Any sufficiently complicated program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."

      I have never seen that in practice (30+ years of industry experience working on very large applications).

      I think it is one of those statements that fans of Lisp love to quote (a lot) without having any empirical data to back it up.

      And yes I am sure that there are examples out there. You can probably find examples of anything if you look hard enough. But that doesn't make it a general rule.

    • By wruza 2025-03-095:29

      This makes no sense to me. CL is not an astral technology. It’s just a parentheses-rich language that is worse than almost any other until you get to macros, continuations and psychotic polymorphism. Which are cool to talk about at hacker parties, no /s, but don’t do much business-wise.

    • By fuzzfactor 2025-03-0918:26

      >the collapse of artificial boundaries between program, data, and interface creates a more direct connection to the problem domain.

      I always figured that was one of the reasons that Excel can tackle so many different problems.

    • By lgrapenthin 2025-03-0821:28

      Was this written by a LLM?

  • By jamii 2025-03-0820:39

    This is an old prototype. I ended up making a language for it from scratch so that I could attach provenance metadata to values, making them directly editable even when far removed from their original source.

    https://www.scattered-thoughts.net/log/0027#preimp

    https://x.com/sc13ts/status/1564759255198351360/video/1

    I never wrote up most of that work. I still like the ideas though.

    Also if I had ever finished editing this I wouldn't have buried the lede quite so much.

  • By jim_lawless 2025-03-0816:411 reply

    This reminds me of home computing in the late 70's when we used to keep our "database" info in DATA statements embedded in a given BASIC program.

    https://jimlawless.net/images/basic_data.png

    • By julesallen 2025-03-0822:221 reply

      Glad I'm not the only lunatic who would do something like this. At least until I got my paws on dBASE.

      I worked with somebody with your name in the early 90s on a Sequent/Dynix system, that wasn't you by chance was it?

HackerNews

About