Code is Data, Data is Code

Rewriting API Documentation with Markdown and ClojureScript

We recently decided to explore rewriting the FullContact API documentation. One of the primary goals of this project was to see if we could accomplish this using our existing component library, which would allow us to incorporate any kind of functionality we want beyond the standard, static HTML documentation we have had in the past. An additional requirement of the project was that the documentation would be generated via markdown, so that changes wouldn’t have to be made directly by the engineering team.

The Foundation

The current front-end stack at FullContact relies heavily on ClojureScript, and our component library is written in reagent (ClojureScript interface to react), so this became the centerpiece of our project. We started looking around at options for converting markdown into something more familiar to the ClojureScript world. We found a library that converts markdown into hiccup data structures, which is exactly what reagent uses.

Initially, we used this library to parse the markdown on the client and generate our reagent components, but we ran into a couple of issues. First, we use inline styles in our components that are generated from ClojureScript maps, but the library was leaving the styles in their string format. In order to have the styles from our markdown in a data structure that we could work within our components, we wrote the following code to add style maps to components:

(defn string->tokens [style]

 (->> (clojure.string/split style #";")

      (mapcat #(clojure.string/split % #":"))

      (map clojure.string/trim)))


(defn tokens->map [tokens]

 (zipmap (keep-indexed #(if (even? %1) %2) tokens)

         (keep-indexed #(if (odd? %1) %2) tokens)))


(defn style->map [style]

 (tokens->map (string->tokens style)))


(defn convert-styles [i]

 (if (vector? i)

   (let [[tag & xs] i

         fi (first xs)

         new-fi (if (and (map? fi) (:style fi))

                  (assoc fi :style (style->map (:style fi)))

                  fi)]

     (into [tag new-fi] (map convert-styles (rest xs))))

   i))

This allowed us to merge custom styles into our component’s default styles. Next, there were obvious speed issues with handling all of this on the client. The nice thing about the library we’re using (and Clojure) is that it’s written in .cljc files, meaning the functions can be used in both ClojureScript and Clojure. As a result, we moved the generation of our hiccup from markdown into Clojure and added it as a step for building our code for deployment. While this sped things up significantly, there were still speed issues (more to come on this later).

Making Use of Custom Componentry

At this point, we have successfully converted our markdown into reagent components with the ability to provide custom styling, but it’s still limited to the basic markdown components and custom HTML. We want to make all of our custom components available for use. In exploring solutions for this, we were reminded of one of the fundamental (and amazing) concepts in Clojure – code is data, data is code. Clojure is in the Lisp family of programming languages. Lisps are homoiconic, meaning all code written in these languages is encoded as data structures, and hiccup is no exception. Hiccup just uses vectors to denote elements and maps to represent an element’s attributes. This makes it incredibly easy to modify via standard data transformation functions (since the code is just data). As a result, we were able to recursively loop over our markdown-converted hiccup and map custom elements to our known custom reagent component counterparts via the following simple function:

(defn hiccup-with-components [component-map comp]

 (if (vector? comp)

   (let [[tag & xs] comp

         new-cmp (get component-map tag)]

     (into [(or new-cmp tag)] (map #(hiccup-with-components component-map %) xs)))

   comp))

Where component-map is just a map of custom element tag name (as a keyword) to reagent component definition, like:

 {:my-custom-cmp some.ns/my-custom-cmp

...}

and comp is our markdown-converted hiccup. This means we can make the following transformation:

// markdown
<my-custom-cmp data-title=”Title” style=”color:#545454”></my-custom-cmp>

into:

;; hiccup

[:my-custom-cmp {:data-title “Title” :style {:color “#545454”}}]

which gets mapped to something like:

;; reagent component

(defn my-custom-cmp [{:keys [data-title style]}]

 [:div {:style    (merge {:background-color "#fff"

                          :color            "#000"

                          :display          :flex

                          :align-items      :center

                          :justify-content  :center

                          :padding          "10px 20px"}

                         style)

        :on-click (fn [] ...)}

  data-title])

Now all we have to do to allow a new component to be used in markdown is add it to our component-map (components for generating an API key, creating an account, playing around with APIs, etc).

Building a Visual Component Library

In order to provide visibility into available components to be used in our markdown files, we generated a visual component library via our component-map that provides visuals for each component with default properties, as well as the code that can be used to generate that component in a markdown file. The components look like this:

and clicking the <> icon exposes the code to be copied and modified:

Next Steps

At this point, we’ve managed to build a framework for our API documentation that utilizes our existing component library, which means all of our components look and act the same across both our applications and our documentation. Additionally, it was built in such a way that it can be maintained by team members outside of engineering. This means that we have accomplished our initial set of goals, however, there’s still plenty of room for enhancements. 

As mentioned before, one of the biggest areas needing improvement is speed. We currently load our hiccup via Extensible Data Notation (.edn) files, but our end goal is to not have to load anything at all on the client. Therefore, we’re seeing if we can make use of Clojure macros to generate all reagent components from markdown during code compilation (instead of having to load the hiccup and generate them on the client). This will speed our API documentation up quite a bit, and we look forward to seeing the results.