I've been programming in Clojure professionally for roughly ten years now. It's a great language and extremely effective if you're building products both on the JVM and in the browser.
I think I've written a handful of macros in that time, there is one in my current codebase in a test class, and it could probably be removed.
Macros are for extending the language. Programmers who are new to Clojure tend to pick macros up and get a bit excited and create lots of DSLs and other cute little trinkets, and then they put macros down again because they realise they don't compose.
The power of Clojure is in simple datastructures, the core functions that operate on them, composition, and host interop.
The most useful thing that I made with macros was a library for opinionated exception handling for Clojure core.async, at the end of which I realised I was using core.async a bit too much and ended up removing a bunch of it.
The product I work on today is an Enterprise toolkit for Apache Kafka (https://kpow.io) that according to Github is 97% Clojure. Data-oriented, great delivery cadence, and terrific fun to work on. Having a single language on the JVM and in the Browser is enormous leverage, certainly we feel the benefit every day.
It’s been several years since I’ve worked in Clojure. When I started to dive into the language, I was warned not to write macros without a good justification (and heeded those warnings).
One of my most memorable projects[1] relied heavily on macros. I knew at the time that I was using the tools I had to automate a performance optimization technique. What I didn’t know at the time was that my former (and once again future) ecosystem, JavaScript (now TypeScript), was going through a tooling revolution (of sorts, no value attached to that observation). Babel, and then countless other tools, were transforming more and more JS. Now that sort of thing is pretty much the default.
All of that is to say: macros are not a silver bullet, they’re not trivial to write/read/maintain… but goodness dealing with similar use cases in JS environments routinely makes me long for them.
The amount of esoteric knowledge required to build something similar to my 150 line library in a JS context is absolutely bonkers. Versus just using built in language features and calling runtime functionality as needed at build time.
I don’t want to overly romanticize either macros generally or Clojure specifically. But I sincerely believe a large part of the frustration with the JS ecosystem from developers’ perspective would be significantly less frustrating if manipulating the language and sharing build/runtime logic were part of the language itself.
There is one macro defined in that library. It looks like it uses Garden’s “defstyles” macro as a key component as well.
I don’t intend to contradict what you wrote by pointing this out; it was just funny to read “relied heavily on macros” only to find exactly one “defmacro” in the codebase! This further illustrates your point: macros are a great tool to have in your toolbox, and when it’s worth writing one, it can be VERY much worth writing one.
This is a good clarification! By “relied heavily” I meant that’s how its core functionality is implemented, not that there’s a large corpus of macro code.
Thanks for sharing - you've spotted the hole in my experience.
I tend to lead more on the JVM and I have talented team-mates who drive the architecture on the JS side. I love being able to contribute both front and back end feature-wise but I don't have too much depth in understanding the bigger issues and advantages/drawbacks of Clojure (and macros) in JS land.
How do you use lisp style macros when you have 500 developers working on a single product?
If you have 500 developers on a java project, you tell the new person which version of java you’re using, you mention whatever libraries or frameworks you lean on heavily and you give them a walk through the subsystems, their key components and essential abstractions and then you spend time showing how the build system is setup and how to create an environment for the system. Then they start knocking out features. There’s a bit more than I’ve said but overall It’s pretty straightforward, any java-for-hire person knows the score and everyone just cracks on with it.
How do you do this when the language itself is programmable? Clojure probably avoids my concern by two facts - it embraces immutability and it only has macros, it doesnt have reader macros.
Common Lisp has neither of those - it might be common to write in an immutable style but in Common Lisp you can (setf ) til your heart’s content. You can also define reader macros - i recently followed an example to implement json parsing in the reader. I’m still equal parts impressed, amazed and horrified.
So. You can probably tell I’ve fallen for lisp in a big way, but how do you scale up to a big dev team? Much as i love lisp in my short journey so far, it seems like a super power that could only be leveraged by up to a “two pizza team”.
>How do you use lisp style macros when you have 500 developers working on a single product?
In general, you don't, or not at the same rate as functions.
In big orgs, macros have three worthy uses:
- cleanly separated libraries for the org, providing a common underpinning
- smaller projects with in the org
- making your language nicer
That last one shouldn't be underestimated. I can count on one hand the # of times I've written macros for work, but I use them all the time, as in calling macros others have written.
You can very much build a big company on lisp (NuBank the most available-to-mind example). But it's true that at larger team sizes there are extra coordination obstacles that must be surmounted before you get to macro heaven.
i think the crux is that it's hard to find 500 people who are all at such high level of capability that could make use of macros in lisp - esp. if they all need to follow a similar style of programming (iirc, lisp macros tend to follow idiosyncrasies of the person writing it).
It's almost as though you're asking for 500 Michael Angelo to help paint the sistine chapel in order to do it faster.
I'm sure there is some nonsense reason why everyone understands templates just fine and the web would literally not exist without them because html is so tedious but "macros" are this mythical complicated impossible to reason about thing??!
Seriously there is a limit to how much you can trust all the secondhand garbage "knowledge" you get from reading opinions of people who claim to have tried something.
Setting the record straight:
- common lisp is fast
- common lisp is stable
- common lisp is useful! (and interactive! even impatients can get their instant gratification)
- common lisp has non-trivial libraries for various useful things
- common lisp is simple and has excellent educational resources
- common lisp is crufty but the community has such a high average education that actually the resources have aged well and hides at least as much insight and wisdom gems as the type theoretical FP world that we now put on the pedistal we used to keep lisp on.
- common lisp is huge. No need to learn it all just hack on what you need and magically you'll be able to make software that works.
If macros are such a problem then restrict their use to seniors or "tool devs" (people who go around improving dev ergonomics by extending the language is synonymous to any other DSTool)... This way you have maximum flexibility and by restricting the words you allow you can have this flexibility while you need it while also programmatically restricting your monkeys to a short enough rope that it'll be hard to hang anyone. You can stay in some Java comparible subset (Java because lispers hate on Java like vimers do emacs).
Yes I Markov chained a bunch of sentences in this comment... I am too lazy to fully engage with the "someone is wrong on the internet"-thing, anecdotally it has been my feeling that it is no longer a good way to learn something to be "wrong on the internet" nobody is cleaning up anymore... These days everyone has a job; extract from the commons and go rave while Rome burns.
I think the lack of reader macros really help here. Macros in Clojure are much more "functions that run at compile time" rather than "look at this fun DSL I threw together".
What I think it comes down to though is adherence to convention. A preference hierarchy has grown together with Clojure and it goes:
1. Can it be expressed using data structures, do that. For example a map/dictionary lookup.
2. If that fails, express your more complex idea using a function.
3. If you absolutely must, write a macro.
You usually end up with very few macros except for those in the standard library, and those you end up with are more commonly of the convenience kind. One example would be the standard web routing library of the early 10's called Compojure that relied heavily on macros where my current favourite Reitit uses just data.
The last time “macros in Clojure” came up, I searched through some of the libraries I’ve run across to see how often they’re used and, as you suggest, it ain’t much.
Re-frame has just one, and it looks like it’s used for testing. Clara-rules has eight that are used in the library, which implements a forward-chaining Rete rule engine. Core.logic has a handful, and that’s essentially “prolog inside Clojure.” So clearly those libraries get a lot of mileage out of data and functions, with just a dash of macro.
I have yet to reach for them myself, and I can’t imagine a situation where I’d need a macro instead of a function. I think they’re probably great for library developers who are smarter than me, and I’m happy to leave them to it.
I think they come in handy when you've got some functionality that has to happen in a context that you can't easily move between function calls... Then you benefit from the ability to template into the description of that context some actions to perform _before_ you enter into the "black box".
That's pretty abstract but honestly its not really such an earthshattering feature. I'd argue it's pretty much necessary for any "final" programming language to be infinitely metaprogrammable (i.e. a lisp-3) but we don't have a mature language like that yet. Probably it will arrive from laskell/hisp kinda synthesis.
Highly recommend checking out materials from Nubank, the Brazilian banking startup that now owns Cognitect (and therefore Clojure). They have several hundred engineers on the back end working almost entirely in Clojure. Believed to be the largest Clojure plant, though many other large orgs now have sizable Clojure staffing.
I saw some of the Nubank folks speak in person in NYC before the pandemic (and before the acquisition of Cognitect) and it was hands down some of the most impressive work I've encountered. Little in the way of macros, just functions and data. What they have done to scale a Datomic infrastructure is astounding.
They have quite a lot of materials online. Worth some time.
Now that the Cognitect team is essentially embedded and more resources are going into core, I imagine at scale tooling will see some enhancements in the coming years.
Most enterprise software nowadays is broken up in services or micro-services anyways, and monoliths spanning more than 6 to 15 developers are becoming a lot more rare. If you've got a monolith with 500 developers touching the code base you're in trouble even if you used Haskell.
Otherwise, macros are actually not super hard to figure out, Lisp macros are easy to write and that means also easy to read and understand, and you've got an interactive development environment so you can quickly explore and try things out to learn how a macro behaves.
Like you said, Clojure also doesn't have reader macros, so the syntax is standard across the board, so macro usage is limited in scope to particular call sites.
Finally, it's not like you're going to have a myriad of custom macros, and generally when you have them they make things simpler and easier, not harder. So you quickly can ramp up on the common macros of a code base and become familiar with them and start to benefit from their usage as well. They'll also likely have a doc string that documents them, so it's not like you have nothing to go with to understand what they do.
Fair, in this case my feelings are they are also easy to read and understand, compared to alternatives.
Similar behavior implemented with text parsing, or annotation, or pre-processor, or inside the compiler, the code to implement them is a lot less tractable.
Once you know how to write macros, it becomes fairly easy to also look at the implementation of a macro and figure out what it does. Macros tend to be very self-contained and often relatively short in implementation.
can you explain why this is such a big concern? as far as i can tell, you can set development guidelines and employ people who will stick to the rules. it's not like you can sneak in macros and present them as functions
What happens when you scale java development to a team of that size? Short version: every optimisation you can dream of gets implemented.
Clever use of Off heap datastructures aren’t going to be enough to impress. Forget thinking your custom class loader with cooperating javaAgent that manipulates the bytes of classes to dynamically add or remove instrumentation is going to impress. These things are just taken for granted.
Even a bot that identifies dead code after traversing the AST, then re-generates valid source code from the (now pruned) AST, runs the test suite then checks in the change all by itself is seen as meh after a while.
No matter how out-there an idea is, the play area is constrained by some immutable platform rules (the java memory model). So no matter how funky things get, any other java dev will still be able to reason about any code in the system.
What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?
> What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?
i think lisp is very nicely structured so that you only allow a certain subset to be used according to seniority. if you have good development paractices at hand, then even if you have a very bespoke program it will be well documented and anything ecentric is explained
lisp is indeed very powerful, but so is c++ for example. the difference is that the syntax of lisp allows you to be neater where you want power and more conservative where you don't, far more easily
This is the best answer so far. Bank Python ain’t so bad. I’m going to stop worrying so much about lisp macro abuse. And I’m going to finally start reading let over lambda.
I work in an organization with thousands of engineers; the largest projects have perhaps 10 contributors. Coordination with huge numbers of developers is done through gRPC; doesn’t require compromising on service internals.
That’s a way to reduce the output of your developers - keep them feeling busy but don’t get features to customers any faster.
If someone proposed some new dogma to live by:
1. Don’t use method parameters - write some IDL instead, add some steps to your build process to auto gen some bindings from the IDL. Introducing a dependency (with its own security patch lifecycle) in the process.
2. Don’t call a method in a library - instead setup mutual authentication and transport layer security, open a port (probably add load balancing and failover routing), set a reminder to maintain ssl certs before they expire, add distributed logging collectors, add endpoint instrumentation, add service cataloguing and lookup, slow each call down by serialising then deserialising the request then repeat that serdes overhead for the response.
…
You’d rightly point out their suggestions add complexity and slow things down, what was so bad about calling a library method in the process’s address space in the first place.
Macros are linguistic abstractions; you describe them as part of your briefing on "key components and essential abstractions". In practice, most macros tend to look the same, e.g. with-<foo> macros only differ in how to acquire and release some <foo>.
FAQ: But what are macros for and why not functions?
A: As you can see in the linked article, they do lots of things but macros are really useful for creating new control structures. Things like threading macros (->), (and), (for), (time), etc.
It is difficult to create new control structures in languages that don't have good macros. Try implementing new flow control using functions and appreciate how hard it is.
This is why a lisp, like Clojure, can add new control structures using a library. Many languages struggle to do that and rely on some committee somewhere deciding it is a good idea.
I'm surprised the `gensym` machinery is so "naive" as in it could actually be possible to craft a symbol that can refer to a `gensym`ed symbol.
I am not very familiar with other languages that implement (un)hygienic macros, is that a common problem in things like Scheme, Lisp, or even Rust, or do they all circumvent that issue with actually private/un-nameable symbols?
Right, avoid using symbols that look like gensym stuff. It would be pretty silly to do something like that. If I saw someone using "G__1234" then that would be a pretty big eyebrow raiser.
clojure macros are quite retro and minimalistic relative to racket's. this is partly an intentional decision; clojure and racket are basically opposite extremes of lisps when it comes to how big a role macro authorship is supposed to play in the ecosystem. personally i'd say that while clojure's decision to de-emphasize macros is, given the rest of the language design, a good one, it's slightly unfortunate that clojure macros don't incorporate some of safety and ergonomic features from scheme/racket, e.g. hygiene by-default and more able declarative templates aka racket/syntax-parse.
I think I've written a handful of macros in that time, there is one in my current codebase in a test class, and it could probably be removed.
Macros are for extending the language. Programmers who are new to Clojure tend to pick macros up and get a bit excited and create lots of DSLs and other cute little trinkets, and then they put macros down again because they realise they don't compose.
The power of Clojure is in simple datastructures, the core functions that operate on them, composition, and host interop.
The most useful thing that I made with macros was a library for opinionated exception handling for Clojure core.async, at the end of which I realised I was using core.async a bit too much and ended up removing a bunch of it.
The product I work on today is an Enterprise toolkit for Apache Kafka (https://kpow.io) that according to Github is 97% Clojure. Data-oriented, great delivery cadence, and terrific fun to work on. Having a single language on the JVM and in the Browser is enormous leverage, certainly we feel the benefit every day.