Latest update on Lisp Interface Builder Thing

Moving, so not as much time to code these days. The software (interface builder thing) is coming along well. Practicing reductionism and watching code disappear. New features are coming into place. Am able to make progress so long as I stay away from design and structure (note to self: code first, map second.) Every time I introduced structure from paper, I've learned how to reduce it and have had to go in and carve out chunks of code. What a waste of time.

The code to build apps/dialogs/menus and other elements is comically small (nascent, but functional.) Still squeezing it down. Lisp makes it easy to make code look like data and vice-versa. It's sorta surreal.

Using it while building it and that's getting fun because I'm finding that I'm using it more and coding it less.

Don't yet have enough for a demo, but can see one coming very soon. For me, that is. I tend to move kind of slow at times. Only because I try to be thorough. Or, I'm just slow. Suppose that depends on your perspective.

Short notes on an interface builder

Went to Pennsylvania for a few days for a wedding. Nice state they got there. Knew I was back in New York when the potholes threatened to swallow the car - the worst of the potholes are deep with square edges and will rattle your teeth - not to mention flatten tires and bend rims.

Making good progress on the interface builder. Moving away from coding elements by hand to creating and modifying elements via UI. Will have the UI connected to the editor tonight. Once connected, it'll be easy to call up the editor for the active UI component of the current app.

As it is, it's pretty fun to be modifying parts of a running app as it becomes apparent that a change is necessary. The lowest level is coming together and as soon as a few more elements are in and working, will sit down and review the code. Reviewed code on the train Friday, but already a lot of changes to review. Mostly have to move a handful of functions to the right files. Also had to clear out prototype functions.

Pretty floored at how fast this is coming together. But, also finding that it can be confusing to build a new UI to edit a UI while using it at the same time. To alleviate some confusion it'll be necessary to write docs on the interactions so there's a clear map to get from point A to point B. Wouldn't hurt to ship the code with docs.

More within the next couple of days.

Building Interfaces in Common Lisp

Well, this is officially starting to get fun.

Coding up live interfaces to interact with data. Mix of low-level support and debugging, along with developing the actual interface. But, can see how fast it'll be to construct interfaces to create, update, and delete instances, while building interfaces to present and interact with a set of records. Cribbing CRM app interfaces from various sources to see how quickly it'll be to construct apps that are more than system utilities or development tools.

Wasted more time thinking that it was time to design/spec/organize the low-level objects. After realizing it wasn't time, went back to top-down, hand-coding and things started moving again. Thinking that my years in OOP probably did more harm than good. It really is easier in Lisp to code, factor and test than it is to sit down and spec out a system before writing any code. In the end, the structures created by interacting with Lisp are more stable and flexible. Besides, it's just more fun to build working code. There's less angst over defects or design issues, too.

Connected the plumbing to access the editor from a running application. Cool! Will connect the editing interface soon. Want to look over the code so far and extract the forms needed to build the interfaces (that are right now hand-coded.)

You know it's getting fun when eating and sleeping become low-priority events. Coffee's food. Right?

Update on Building Interfaces

So far, so good. Spent a chunk of Saturday thinking I understood the low-level structure only to realize I didn't. So I decided to listen to Paul Graham (again!) and just use a list until the structure became clear. After that, I went back to working top down since that's where I first saw the patterns.

Built the interfaces for the web utility that will generate site map files, paserve publish commands and the functions that create category and summary pages. It was much easier to see the patterns at the high-level. Sketched out the interactions between the low and high-levels and started hand-coding some of the core elements.

Will continue to whittle the code down at the top while bringing the low-level up to meet it. At the whiteboard last night sketched out the design to make sure the editor can be called up from a running app. It can - I felt relieved.

Went through the Hemlock manual on Sunday to visualize how to open up Hemlock by making some tools and interfaces for configuration. Have some tools already, so will use them and other new interfaces to test the interface builder.

More to come soon - probably have another update middle-end of the week.

Peculiar Circularities in Common Lisp

I just realized a peculiar circularity of Lisp. I'm building an application that modifies itself while it's running. It'll have a sister app that'll run the apps while making it easy to connect to the editor and modify the component that's running (at least that's how I'm looking at it while it's being designed. It might look different when it's done.)

A defined interface can be up and running and I'll be able to say, "Hey! This widget needs to be modified." And, it'll be a snap to open the editor for that widget and start making changes while the app's running. I don't know if the app is going to be klunky or smooth, useful or useless, but I get that part (editing a live app) and can make it work.

I know it's been done before. And I know that Lisp is the "programmable programming language". But I guess today, it's a little more awe-inspiring than it was yesterday.

Just wanted to share.

One Step Up and a Tangent

While building the interface for the web site code, I saw duplication in the layout and the functional core. So, I stepped back, squinted and saw a pattern emerge.

It was a familiar pattern, one that I'd made an attempt to address a while ago. It has to do with being able to build, test and release interfaces easily and quickly.

But the first few attempts I made at building an interface to build interfaces ran into walls. Either I was too immersed in some other problem to fully comprehend the scope or didn't give it the time it required to spec it out. More likely, I just didn't get it yet.

Whichever it was, this time I'm pretty sure I've simplified everything just enough to build a good interface builder for my way of working with Hemlock. I can't say graphically what it is or how it will work yet. Sat down and worked out the core on paper and will be putting it together starting today. Will post an update this weekend.

In most of these posts, I've been pushing projects onto a stack. What I've been looking for is a way to work faster so the projects don't stack up, but flow out. My hope and (pretty firm) belief is that if the interface builder works, then I'll be able to start popping projects off the stack. And that would be quite a relief.

I'm not sure if anybody else would even consider working the way I do - it's odd. But, if it makes it possible for me to release code, I guess that's a good thing in it's own right.

Web dev

After manually updating three files after any addition or update to web-related code, went and wrote a couple of scripts to do it automatically. Got tired of forgetting to add a publish directive, update a site map file or add a function to render categories to a page. The generated files are:

1. pub.lisp - list of publish commands for paserve
2. categories.lisp - list of functions associated with a url that present summary detail for a category (see desc in #3).
   
3. doc.lisp - functions that generate summary pages for categories (like Google News with the category heading followed by a short list of article links followed by a short summary.)
4. sitemap.xml - site map file for search engines.
   

Might also be another file that will generate the index page and nav bar elements based on the categories installed for the site.

Putting an interface on top of it in Hemlock, so adding, modifying, and deleting categories, web pages, article summaries, site maps and content is a little easier. The code saves a set of structure instances containing enough detail to reconstruct an element. These instances are available via a set of interfaces that manage load and save. When generating output, the site-specific detail is loaded and a set of macros generate output, which is then written to a site-specific directory. Pics as soon as the interface comes together.

So far, it generated most of one test-site's framework in a relatively short script. Putting an interface on it should be pretty painless.

Want to be able to construct a site's framework with as little typing as possible and make it easy to automatically update relevant parts when details are modified. Also want to specify layouts through an easy to operate interface that provides for customization. And if I never have to manually modify another site map file, I'll be one happy camper!

Later - should be able to automatically SSH to a remote server, upload files, attach to a running Lisp and run ASDF to update the site(s).

Learning Lisp - Packages and Build

The other day, while being stoked that I'd incorporated ASDF into my arsenal, I realized that Lisp was, from the build perspective, just another programming language. The thought kind of struck me, because it's obvious Lisp is just another programming language. A pretty amazing "just another programming language". But from the perspective of code organization, documentation, build and test, writing a good Lisp program has all the requirements of a good program in any other language.

If I Were Starting to Learn Lisp Today

As a programmer who started coding in the early eighties, I wondered what I was doing for the last 9 years of coding Lisp that prevented me from being as organized with my Lisp code as I'd been with my C, C++, Java, SQL, and Perl code?

The best answer I could come up with was that I was writing Lisp code for the interpreter. I didn't create packages with defpackage because I wrote code in the :CL-USER or :HEMLOCK packages. The goal was to get code into the REPL (read-eval-print loop) and start modifying or extending whatever the problem du jour was. Whenever a load issue presented itself, I just kind of hacked around the problem with scripts that were mostly sufficient for the application - it's so easy to "roll your own" in Lisp, that it becomes possible to become insular to the extreme. I was sort of aware that ASDF was out there, but my scripts were fine - especially since there wasn't occasion to load often.

Most of what I've been writing in Lisp over the years has been what I'd consider pure research projects. I've studied a new way to do object-oriented programming and have invested in finding ways to improve productivity for developers and general computer users. So, much of my time has been spent inside of Hemlock (and before Hemlock, Emacs for tool development.) When I had to restart Hemlock, loading files was automatic via .cmucl-init and .hemlock-init. Aside from core files, I'd simply open the files I was working on, start a slave-lisp, load the files and start working. No need for ASDF and except for declaring code (in-package :HEMLOCK), I didn't even worry about packages. I wasn't in "production mode" - getting code ready to ship.

Worse, when library code became stable, then I'd build a new core. Simply start Lisp and the init files would load the various library files. Then a save-lisp call and bada-bing, bada-boom, no more load issues because now my libraries became a part of Lisp. How cool is that?!

Big Mistake

And in the previous paragraphs lie the two big mistakes I made when I learned Lisp. It's so powerful and seductive, I just hopped in and started driving without organizing my activity. With Lisp, one writes code, and immediately sees if it works. If it does, then you move to the next problem. It's easy to modify a running Lisp or add to it. Connect to the REPL, even on a remote system, submit code and "Voila! It works!".

Basically, it was so easy to just write code in Lisp and see it work, I lost control of my build process. Exceptionally difficult for me to realize, since I'd worked on a number of very large commercial software products where CMM and organization and stable builds were the norm. I just got lost in the fog with Lisp - or, more to the point, finally had the opportunity to explore concepts at a depth previously inconceivable using other languages.

Out of this realization comes my advice to new Lisp programmers - especially those coming from another language:

1. Learn packages. Buy any Lisp book and immediately jump to the chapter on packages. Learn about them while you're learning about lists. Ask questions about them on comp.lang.lisp. Master them before you master anything else. Whether you use them or not, knowing about them makes it possible to concern yourself with architectural issues and code modularity.
2. Learn ASDF. I worked with C for about a week before I started learning about Make files. Same with C++ and Java. But Lisp, and this my own fault, I didn't learn about the Lisp equivalent of MAKE until after coding in Lisp for years. It's not in any of my books and most of the tutorials on Lisp concentrate on the language.
   

I wasn't shipping my code, so configuring it for release was neglected. Anybody who's ever shipped a product knows this is just crazy - and I've worked on teams that have shipped major commercial products, so I knew better.

Regardless of whether you're writing your own tools or building the next big commercial app, your software should always be ready for release. If for no other reason than it establishes in your mind the need to consistently build professional quality code. I'd been building code that epitomized "Quality with a K", as we used to say at one software house I worked at when it was obvious one or all of us had been hacking mercilessly on a problem.

Regardless of whether your code is going public, you need to have control over the build. In the old days, the build was once a day and it absolutely had to work. Check in by night, build in the morning. Every project I'd ever worked on was stringent in this requirement. The Golden Rule was, "Don't Break The Build!" Nothing worse than having someone in your office telling you your last checkin hosed the build.

Lisp is different because it's actually pretty hard to break anything for any length of time. Even completely factoring a project isn't generally too painful.

In a nutshell - learn about packages and learn about ASDF. Right away. With your first hello-world program. Put the code for that single function app into a file that has a (cl:defpackage ...) package specifier and put that file into an asdf specification file.

A Little About Packages

Whether you realize it or not, when you start coding Lisp, you are coding inside the :CL-USER package. The :CL-USER package :uses the :COMMON-LISP package. The :COMMON-LISP package :exports a ton of functions that you can use in :CL-USER without specifically telling the REPL where to find the functions.

Packages are a powerful feature of Lisp. But they're not the secret sauce. Learn them, use them and make them your first step into Lisp - especially if you're coming from another language that requires a similar concept - like Java or C#. They're a familiar concept that provides one with the awareness that Lisp is similar to other languages with respect to code modularity.

A Little More About ASDF

If you're just coming to Lisp from another language, learning ASDF will allow you to see how easy it is to MAKE your project. After making mods to code during an interactive session with the REPL, you can verify your system builds by running ASDF - it manages dependencies and is a breeze to configure and use.

Conclusion

Coming to Lisp can be a daunting challenge. It's big, like many other languages these days. But, by leveraging familiar aspects of the language, learning Lisp becomes fun while providing a smooth learning curve into the language. Learning to modularize your code using packages while taking advantage of ASDF for building projects will help you step smoothly into Lisp. You'll also learn that Lisp shares many similarities to other programming languages, providing control over packages/modules and building code while managing dependencies.

With the added bonus that Lisp offers exceptionally powerful constructs that make coding in Lisp exceptionally fluid. That's the Lisp secret sauce - being able to turn your thoughts into working models as quickly as you conceive of them.

ASDF - Time Saving Build Tool for Lisp

Imagine working with arguably the most fun programming language in the world. Now imagine that whenever you had to build some program from scratch, you used hacked, buggy, fragile build scripts that you'd concocted - usually at some unholy hour of the morning after Hemlock took the system down and the build was broken. Now imagine, instead of cruising along, building your latest widget, that you've just poured another hour of unproductive time down a giant sink-hole created by your very own, poorly written load-scripts only to get back to a blank editor with the question, "What was I working on again...?"

"What a Maroon, what an Embezzle" - Bugs Bunny

You said it, Bugs.

Now imagine working with arguably most fun programming language in the world. Only now after Hemlock hoarks up a hairball, you simply type:

(asdf:operate 'asdf:load-op 'go-hemlock-go)

and your software compiles and loads according to dependencies. Everything. Files compile in the right order. Dependent packages build in order. "Sweet Holy Lambda!", It's like a Lisp programmers dream come true.

I'd read about ASDF for a long time, but was too busy doing nose-down coding (plowing, as it were) to look up and find a tool to simplify life. Wasn't til I downloaded some of Zach Beane's libraries that I saw how easy it was to configure ASDF to build a project. I try not to self-flagellate, but I did smack myself in the head when I realized that I'd been wasting a lot of my time.

I may be slow to the party, but I can find the keg once I get there

It was glaringly obvious as I started working with web software that my custom loaders were weak. Before the web, I wrestled with my own load scripts once in a Blue Moon - Hemlock doesn't crash often at all - usually FreeBSD dumped first.

But when it did, and it broke on load in one of my stable files. Well, then, there's nothing worse than wishing somebody else had written the mess you were looking at - code you knew you'd crafted with your own hands to solve a problem you probably didn't want to solve in the first place, but were too focussed on your own code to look for a Better Way (that link's to the lyrics of "Cover of the Rolling Stone", this is to a YouTube version. Funny how a couple of words trigger memories - my first album when I was 9 was Dr. Hook and the Medicine Show's "Sloppy Seconds", brand-spankin' new in 1972. I badgered the snot out of every adult in my family until they finally gave in and bought it for me. Was rare that they gave in - military Dad, Spartan life - but they felt bad because my Dad was in the hospital and they wanted me to feel better. Relatives had a send-off party when we left California to go back to Seattle and we played that song a dozen times - everybody singing it and laughin' - what a riot. Like the last golden sunset before darkness. I'd have worn a groove in that song if the record hadn't been lost in the move. Wow, what a tangential trip in the wayback machine...)

Where was I...oh yeah. But putting up websites and making minor tweaks quickly highlighted the error in my ways. So I started configuring the various web projects to use ASDF. It took very little time. So very little time. It's easy to configure and use. Embarrassingly easy to use. Kind of makes me wonder what I'd been smokin' all these years that I'd overlook it.

Smokin'? Bah, probably not drinking enough coffee...

Making Web 1.0 Sites in a Web 2.0 World

Lately, it's been all web, all the time. Been developing some websites - for my company and a couple dozen of my hobbies.

So far, so good. I'm still rolling my own, but am having success. Been putting a process in place to make it easy to go from concept to implementation.

So far, I'm up to six live websites and about six more pending. They're pretty vanilla and it's been fun to see how my time has shifted from coding to content authoring. At first, it was about 80% coding. Now it's about 80% content development. The code keeps shrinking, too, so that's a good sign. The first sites are simple, old-school sites hosting articles and maybe a little more.

Once the old-school websites are up, will put up a dating site and a couple of political sites, too. They could see higher traffic than the other sites, so I'm taking the time to make sure the platforms are solid before I put them up. May actually try to ask for help building them - we'll see. I'd actually been hoping to make the shift to KPAX after watching the movie "Reddit in 100 Lines of Lisp". We'll see if I can make the switch. Then there's Arc, which looks pretty enticing, too.

Have a few forms that are embeddable in web pages. Have been kind of curious about turning them into Google Gadgets. Will run a couple of tests to see how they work out, but need more content to back them up.

It's been a lot of fun writing web code in Lisp (ASDF, PAserve, SBCL). Developing a suite of sites has made it easy to see duplication in the code, so the libraries keep growing while the site-specific code gets smaller. Wherever I see duplicate code, I break it out into a function or macro and keep moving. It never gets old to see code get smaller while providing greater utility.

My last web experience was with Java and Perl. My blood pressure was a lot higher back then because the tools kept forcing me to look at them when I wanted to look at the problem I was solving. Now, I barely notice the editor or the language and it's easy and fast to make edits and see them work (or not, sometimes.)

Overall, it's been fun to build the sites and break code into widget/gadget units, but the next step is going to be making interactive sites - it's fun to shoot around, but not nearly as fun as playing a game.

More later. One of the sites I'm working on has been for my company. Once I feel comfortable with the quality of my libs, then I'll put them and the site up. And I'll put up some URLs of the sites I'm building. Right now, they're too scrawny to feel good about showing.

Buy Lisp Stuff at the Black Grit CafePress Store

About a year ago, I put together a small set of graphic designs for Lisp and Scheme. I uploaded the graphics to CafePress, set up a store with a bunch of products, then promptly moved on to something else. Got busy with something or thought they'd sell themselves. I dunno.

Today, while scanning through email I found a message from CafePress talking about a sale. Curious, I went to their site, searched for Lisp and saw some designs. I thought, "Hey, those are kinda cool. I think I'll buy a couple." Then I realized they looked familiar. A mind truly is a thing to waste, er, a terrible thing to waste.

At the time, I wanted a way to show off my abiding love of Lisp. Now, you can show off yours, too!

Click here for the Black Grit CafePress storefront.

Back, and Badder than Ever

Well, so far the only things we've lost at the hospital were a set of teeth, a couple of toys, a heart blockage and subdural hematoma. Thankfully, everybody managed to come home. A little worse for wear, but we're all happy because the patient is happy. Pretty stressful couple of months though. The two medical problems are wildly contrasting. One requires blood thinners, the other is exacerbated by them. Hopefully, things will improve in spite of the fact that it seems a lot like walking a tightrope regarding the meds.

On a lighter note, there's coding and Lisp. Last few nights I've been getting back in the saddle with some web coding. I've got a web server set up running portable Allegroserve and SBCL, which almost matches my development environment of paserve/CMUCL.

I switched to SBCL because I was having problems loading KPAX and it loaded out-of-the box into SBCL. Haven't had a chance yet to figure out why CMUCL wasn't happy, but I'll debug it locally. I've probably missed something in the load or config.

For now, I'm just building basic web sites. Going to take a look at KPAX as it looks like it might be a lot of fun to use. I'll also take a look at CL-Prevalence as a data store. I've been using Postgres with Eric Marsden's nice pg-dot-lisp interface for a long time. While pg-dot-lisp makes it easy to interface Lisp with Postgres, it'll be nice to take a look at other storage options that don't involve SQL.

The other thing I started the other night was writing an interface to Amazon Web Services. It's kinda rough, but taking shape. Over the next few weeks (crossing fingers), I'll be finishing up my charting code and putting it on my website for download with some chart generation pages.