A better GTD and CRM flow for emacs org-mode

This was my attempt to remix org-mode with my Taskpaper flow to try to get all the benefits of org-mode for note taking, tracking and being date-aware.

While I still owe Taskpaper a huge debt for making GTD effective for me, I’ve talked before about how it not being date-aware, having repeating tasks, and lack of customization, as well as it not being great for tracking over time, led me to try org-mode.

While org-mode itself was amazing for tracking, it’s default setups, even with heavy customization, made it less than ideal for managing my weekly and monthly schedules (not to mention lack of hierarchical todos) and made it difficult to lose the forest for the trees.

Let’s talk about what I wanted to accomplish with modifying org-mode in this way, so I had a one-stop view for my week and what needed to be done on any particular day.

1. Have a date-sensitive agenda that can remind me of items with dates attached

   For me, these were both deadlines (key, as I often have to chase people I’ve given deadlines to), scheduled items such as meetings (generally non-work), days blocked off for specific things coming up in my calendar (say, a trip or similar), and items like birthdays and anniversaries of friends or say, epic milestones (my recent 20 years since I moved to Paris for instance (yes, I do keep track of things like this), or long-running reminders, for example, like when I need to get vaccine boosters. For me this is what I equate with “tickler” functionality a la Getting Things Done by David Allen.

2. Manage and prioritize the ToDos or non-date-sensitive items

   which I generally assign to specific weeks or future weeks to work on in that particular week. I manage my weeks rather than days as so this was a matter of how to deal with the firehose of things that need to get done and how to prioritize more effectively.)

3. Be a better friend and business associate

   I often find that’s about maintaining contact with people, so a personal, lightweight CRM. I wanted a system that would let me know when I should be pinging people to see how they were doing and make sure I am maintaining those connections, especially for the large number of people I have living on other continents, out of timezone, or similar.

4. Manage calendar events outside of using GCal

   Strangely I quite managing calendar events in text and having an agenda view that looks at my calendar the entire week. In general this is a month-headed list that keeps particular high level events as well as provides a code mechanism to pull birthdays and anniversaries out of contacts.org for people or milestone events

5. In-place capture of daily notes, subjects, lists, and todos

   My workflow in meetings capturing the essentials in a heading for the meeting, or summarizing what happened afterwards. I capture Todos inline, assign tags or dates to them and then process them in org-agenda.

The Setup

So, before going into file setups, let’s talk about how this all fits together in practice. Basically, the two major items used are the excellent Deft for emacs, which is a notational velocity clone which quickly allows filtering or creation of note files. Basically, all files in this system are under the relevant Deft directory and that allows rapid filtering and searching for items and Deft itself is hotkeyed to Ctrl-d in emacs for effortless switching.

My basic daily morning routine consists of getting into Deft and then using another hotkey Ctrl-j to create a new daily file. I’ve got a snippet in org-mode format that I then paste into that file which looks like this:

Log 2019-03-11 Mon             :logs:

Where: 
Tags: @logs

*** The  Day

Workouts:
Sleep:
Weight:
Mood:

Breakie:
Lunch:
Dinner:

Log:

- 

*** The Skinny

*** Gratitude/Savoured

1.
2.
3.


** Perso                       :perso:

** Traveloka                    :loka:

I’m on OSX, so I use the Alfred’s snippet feature to paste it in via hotkey and typing in “log”, but effectively, it gives me some standard daily tracking for the day, and then hierarchically separates my day into personal and work areas (in fact, the lack of a decent global clipboard and snippet manager in Linux is currently an issue for me as I’ve come to depend on this feature heavily.).

The nice thing about the work section is that I simply create headings for each meeting I have during the day, and then as I take notes on the things that happen in the day (or create high level todos if they’re not attached to meetings). I slot todos triggered by meetings directly into the meeting section which gives them context when in agenda view. Also, because of the way agenda mode works, and the fact they are attached to a particular yyyy-mm-dd titled org-mode file, I also get an instant idea if I need to of how long those items have been around for (though my TODO snippet uses a :PROPERTIES: drawer with a :CREATED: date on it as well.

During the day, I’ll mostly switch between my crafted Agenda view, which combines both Date sensitive items and Todos for a particular week (I’ll outline if code-wise below), and the day’s yyyy-mm-dd note buffer with the standard emacs switch buffer command, CTRL-x-b.

From Agenda view if I need more context, hitting Enter on a particular heading item jumps to a sparse tree of the item in its actual file which is super handy if I need more context due to needing more info on the item in context, or just to review information that might have spawned the todo in the first place.

The Agenda view takes care of collecting up all the daily TODOs into one interface. I’ve crafted a custom view for the Agenda which gets triggered by the ‘a’ key in Agenda mode (so ctrl-t a) and this combines both date sensitive agenda items for the day (so, deadlines, scheduled todos or meetings (generally personal, I use GCal for work related events), and dates like birthdays, anniversaries, or personal milestones I track.

I use two files to take care of the non-TODO items like meetings or date blocks I want to be reminded of, contacts.org which takes advantage of the org-contact package and cal.org which is a simple hierarchical list by month of personal things I want to have show up in the Agenda.

contacts.org I currently structure like this:

* Contacts                                                             :ping:

** Peeps

*** Samurai Jack                                                        :w17:
:PROPERTIES:
:EMAIL: some_random@email.com
:BIRTHDAY: 1982-12-12
:END:

** Fam

** Neo Peeps

** Loka Peeps 

** Aussie Peeps

...


* Anniversaries

** Milestones

*** Landed in Singers
:PROPERTIES:
:ANNIVERSARY: 2013-11-06
:EMAIL: a@a.com
:END:

** Friend's and Family Anniversaries

*** Mom and Dad's Wedding Anniversary
:PROPERTIES:
:ANNIVERSARY: 1968-11-23
:EMAIL: a@a.com
:END:

Org-contact seems to trigger of an :EMAIL: field under properties denoting what it believes is a contact. While I started off using it for being reminded of peoples’ birthdays, I found it also worked if I have it an ANNIVERSARY field and then configured cal.org to pick that up (details on that coming up, though for some reason, it lists it as a Birthday in the Agenda view though I imagine that’s a simple code change and Pull Request I need to make to that package.).

The other key thing here to note is the top level :ping: tag at the top of the Contacts hierarchy. Having this here allows me to include things in the Agenda view according to the other tag you see here :w17:, which is used in the custom Agenda view to tell me I need, on week 17, to contact Samurai Jack. This is all driven through the magic of tags and the customized agenda view.

The way it works on the week in question is that Samurai Jack shows up in a list called “Pings”. When I decide I am going to make the contact, usually either a call or an email, in the agenda view I use the super handy “Add Note” function via CTRL-c-z and type away, which adds an entry to the :LOGBOOK: drawer when I close it. Currently, I haven’t really found a good method to then manage TODOs which are linked to the contact item or logbook entry from this, but the lightweight nature of this CRM functionality works pretty well at the moment (though I’d love it if someone added a way to link TODOs and other things via a contact somehow without resorting to a tag, which is how I manage todos and such for direct reports and key stakeholders I work with now.).

The cal.org file is a lot simpler. It is a simple hierarchical list of months and a tiny bit of code to trigger reads of the org-contact file. You can see an example of the file below which demonstrates the high points. I also abuse the tag system here utilizing the tags :bday: for birthdays, :ann: for anniversaries, and :hols: for public holidays. Note the code evaluations under birthday, anniversaries, and holidays. The first two trigger org-contacts to look for the birthdays and a anniversary properties under valid contacts. The final code snippet triggers the holidays functionality in org-calendar. I’m still trying to suss out what holidays it does and does not give you, since they seem US-centric, so just to be sure, I add in holidays that would be important for my current Singapore and Indonesia based staff, as well as things I’d hate to miss such as Mother’s Day and Father’s Day. The Agenda view then triggers off the tags and filters and brings up the various date related events.

* Cal

** Mar 2019

*** Dinner with DSci Candidate @ Bakalaki
SCHEDULED: <2019-03-21 20:00-23:00>
:LOGBOOK:
:END:

** Feb 2019
** Jan 2019

* BDays                                                                :bday:

%%(org-contacts-anniversaries "BIRTHDAY")

* Anniversaries                                                         :ann:

%%(org-contacts-anniversaries "ANNIVERSARY")

* Holidays                                                             :hols:

%%(org-calendar-holiday)

** SG/ID Good Friday
<2019-04-19 Fri>
** SG/ID Labour Day
<2019-05-01 Wed>
** Canada - Mother's Day
<2019-05-12 Sun>

...

That’s about all the kind of setup you need to do file-wise in emacs and org-mode. The heavy lifting gets done by an custom Agenda view which utilizes the (excellent) super-org-agenda package. You then simply craft a formula to give you the view you want. As outlined above, my view is split into sub-categories of Dated tasks which I focus on daily and generalized tasks as todos or pings that I need to accomplish. The super-agenda view looks like this (and again, gets triggered through the standard CTRL-t a key combo. This gives you a “Today” time grid view which lists anything for today’s date which is tagged with bday, ann, hols, cal, or today or scheduled items for the day, and below that anything in the todo state “WIP”.

Under that there is a number of hierarchical lists Perso (for my own stuff), items that are Overdue, Scheduled items which did not happen which I need to Reschedule, and things that are Due Soon so I have a spidey sense on upcoming things I need to have ready soon.

After that, I move on to non-dated items. The todo list lists everything under a header “ToDos Week 12” (for anything not dated tagged :w12: and then splits those items into three lists, Perso, Work and Ping (which is the CRM contact list for the week.).

(use-package org-super-agenda
  :ensure t
  :defer t
  :config
  (let ((org-agenda-custom-commands
       '(("u" "Super view"
          ((agenda "" ( (org-agenda-span 1)
		       (org-super-agenda-groups
                        '((:name "Today"
				 :tag ("bday" "ann" "hols" "cal" "today")
  				 :time-grid t
  				 :todo ("WIP")
  				 :deadline today)
			  (:name "Perso"
				 :tag "perso")
			  (:name "Overdue"
				 :deadline past)
			  (:name "Reschedule"
				 :scheduled past)
			  (:name "Due Soon"
				 :deadline future)
			  ))))
           (tags (concat "w" (format-time-string "%V")) ((org-agenda-overriding-header  (concat "ToDos Week " (format-time-string "%V")))
	   		(org-super-agenda-groups
	   		 '((:discard (:deadline t))
			   (:name "Perso"
				  :tag "perso")
			   (:name "Loka"
				  :tag "loka")
	   		   (:name "Ping"
	   			  :tag "ping")
                           ))))
	   )))))
    (org-agenda nil "u"))
  )

Note that when you initially load your emacs, I still have to fire up org-super-agenda-mode with an M-x and typing in org-super-agenda-mode but after that everything works as advertised. Hit refresh (‘r’) in Agenda mode and you should have everything split out into nice, comprehensive sections, which make it much easier to separate out your work, personal life, and CRM functionality.

The system does depend on you maintaining tags somewhat, however. For example, when I actually am in the ToDos under the Ping heading, I will generally make contact with someone, write some sort of note about them with the Add Note functionality (CTRL-c z) and then use change tags to a future follow up week for something not date critical but that I want a tickler on (CTRL-c q) or create a new Todo or Chase for them in my regular Todos for the day.

Being tag-dependent, on Sundays or Mondays, I generally have to retag anything from the previous week (say w12 if it it is now week 13 with a w13) but that is something I could easily automate with a script (much like my “archive all daily files that have all Todos closed in them” which keeps my org-mode directory clean and fast.

For those who are curious, these are the setting in my init.el controlling Todos functions.

(setq org-agenda-skip-deadline-if-done t)
  (setq org-agenda-skip-scheduled-if-done t)
  (setq org-deadline-warning-days 5)

  ;; I personally prefer getting advanced warnings of some 
  ;; things coming up that are scheduled, especially if 
  ;; holidays or birthdays or such so I have some planning 
  ;; time. You can un-comment the below if that isn't you.
  ;; (setq org-agenda-skip-deadline-prewarning-if-scheduled t)

  (setq org-agenda-todo-list-sublevels t)

  (setq org-agenda-todo-ignore-scheduled 'all)
  (setq org-agenda-todo-ignore-deadlines 'all)
  (setq org-agenda-todo-ignore-with-date 'all)
  (setq org-agenda-tags-todo-honor-ignore-options t)

  (setq org-todo-keywords '((type "TODO(t@/!)" "WIP(w@/!)" "CHASE(c@/!)" "GAVE(g@/!)" "|" "DONE(d@/!)" "KILL(k@/!)")))
  (setq org-log-into-drawer t)
  (setq org-log-done t)
  (setq org-log-reschedule t)
  (setq org-log-redeadline t)
  (setq org-enable-priority-commands nil)

  (setq org-agenda-deadline-leaders '("DUE:       " "In %3d d.: " "%2d d. ago: "))
  (setq org-agenda-scheduled-leaders '("DO: " "Sched.%2dx: "))

I try to have a streamlined Todo keywords setup, which cycle through these:

I don’t use a NEXT keyword, technically breaking Dave Allen GTD canon because I find I generally have to move between things too rapidly though that certainly might be an addition (but even if it was, it feels more like a tag than a keyword.).

Overall, that’s about the entirety of the system though. I tweak it every so often and there are still a few bugs here and there (for example, sometimes if I look at other views, the Agenda view does not re-populate when I head back to the normalized agenda view so I have to sometimes kick out of emacs and back in to get it to work. Definitely a bug in my scripting or lack of understanding of org-mode, I’m sure.). But it is definitely working a whole site better for me though. A lot of minor tweaks I’d like to put in (previously I used the gcal package to one-way sync my personal calendar down to my emacs as well, but took that out while I was sorting out this view.).

Would love to get feedback on this. As mentioned, it seems to be working very well for me, but I’m sure there is much to be improved. While it seems to be great at catching all the things I need to be doing, I am a little worried at times about things falling through the cracks (especially if I mistag things), which is a constant fear juggling a real life and 125+ person team.