timetracker

tblancher's CLI-based time tracking system (see demo videos of the timetracker setup, macro setup, and Clarizen process)

INSTALLATION

See INSTALL.md in this repository.

OVERVIEW

This framework is what I use to keep track of my time, as I work. Only one task is allowed concurrently, so if I switch to another task, I must end the current task before proceeding. This is tracked in a log file with the following format:

log file format

2017-12-08 08:21:27:  Begin [Internal] email triage
2017-12-08 08:59:05:  End   [Internal] email triage
2017-12-08 08:59:06:  Begin [Internal] internal task 1
2017-12-08 09:10:55:  End   [Internal] internal task 1
2017-12-08 09:10:56:  Begin [Internal] internal task 1
2017-12-08 09:15:14:  End   [Internal] internal task 1
2017-12-08 09:15:14:  Begin [Cloud] Customer 1 task 1
2017-12-08 09:19:52:  End   [Cloud] Customer 1 task 1
2017-12-08 09:19:52:  Begin [Cloud] Customer 1 PDROP-0000000 case-related task 1
2017-12-08 09:27:37:  End   [Cloud] Customer 1 PDROP-0000000 case-related task 1
2017-12-08 09:27:38:  Begin [Cloud] Customer 2 task 2
2017-12-08 11:00:39:  End   [Cloud] Customer 2 task 2
2017-12-08 11:00:40:  Begin [On Prem] Customer 1 PDROP-0000000 case-related task 2
2017-12-08 11:07:24:  End   [On Prem] Customer 1 PDROP-0000000 case-related task 2
2017-12-08 11:08:56:  Begin [Internal] internal task 2
2017-12-08 11:25:43:  End   [Internal] internal task 2
2017-12-08 12:47:50:  Begin [Cloud] Customer 1 PDROP-0000000 case-related task 1
2017-12-08 13:31:52:  End   [Cloud] Customer 1 PDROP-0000000 case-related task 1
2017-12-08 13:31:52:  Begin [On Prem] Customer 3 external meeting
2017-12-08 14:11:13:  End   [On Prem] Customer 3 external meeting
2017-12-09 14:11:14:  Begin [Internal] documentation creation
2017-12-09 17:01:47:  End   [Internal] documentation creation

The timestamp is naive, it is assumed to be in your local timezone. It is also in most-to-least significant units, though any valid date and naive time should work (however, any other format is untested. BEWARE!, also, the Rust programs will not work with any other format). The separator between the timestamp and the Begin or End keywoards is : , literally a colon followed by two spaces. Anything else will break timetracker.py (and the Rust version), and the rest of it will fall down. Each task must have a Begin and End line, or else the output of timetracker will not be correct. Too many begins or ends should be detected by timetracker.py, but this will likely cause a panic in the Rust programs. The name of the task is free form, and must be identical for the Begin and End (otherwise timetracker.py will think they're different tasks).

Certain strings have special meaning to timetracker. The category name in square brackets specifies that the task is related to a specific customer, or should be tracked on a certain item in Clarizen, to make transferring to Clarizen more straightforward. These can be any string, even with spaces, but beware of using shell special characters, or special regular expression characters. The names can be anything, so things like [BB&T], [Citi], [Wells Fargo], or [Internal] are perfectly valid. For the author's most common categories, macros are set up with text string triggers in Keyboard Maestro (more on that below). If you decide to use timetracker.py (the Python 3 script), the category (in square-brackets) is optional, but won't have a separate section in do_process.sh (though it will reflect in the grand total at the end). If you decide to use the Rust programs, the category is required, or else the Rust programs will panic and error out. It is highly recommended to use the Rust programs, for speed if nothing else.

Note that the aggregate programs (do_process.sh, doprocess [Rust], chug.sh, chug [Rust]) expect the log file to be named by the ISO-8601 date format with the .log filename extension (e.g. 2017-06-09.log, 2022-07-12.log, 2022-12-19.log), so it makes sense to keep only a single day's logs in one log file (since we're expected to load these times into Clarizen on a daily basis). However, there is nothing else in the code that makes this requirement (either Rust or Python/Bash scripts). In that case the aggregate programs will cover more than a day's worth of logs, which might make it difficult to transfer to Clarizen. Thus it is highly recommended to keep only a day's worth of logs in one log file. I recommend that if a task extends past midnight the next day, to End it at midnight in yesterday's log, and Begin it at midnight the next day's log.

An important point about this timetracker system: it automatically rounds the duration to the nearest quarter hour (nearest 15 minutes). If you document a task, but it takes less than 7.5 minutes from Begin to End, it's set to 0.08 hours (4 mintues, 48 seconds/288 seconds). Otherwise it would round down to 0.00hrs. If you don't want to count tasks that take so little time, you may want to remove them from the log. Previous versions of this system would round up to the nearest quarter hour, but I feared that was incorrectly inflating the amount of time I was spending during my day.

timetracker in Rust

Again, see the INSTALL.md guide in this repository for instructions on building the Rust programs. It is highly recommended to use the Rust programs instead of the older Python and Bash scripts below, if only because the Rust programs are much, much faster to execute than the older scripts. The usage is straightforward:

timetracker usage

Pass the log name as the only argument to timetracker. Alternatively, you can execute a command pipeline that generates valid log output in the format above, piping it into timetracker:

timetracker 2022-12-19.log
grep 'Customer Name' 2022-12-19.log | timetracker

This will generate output similar to the following (using the above log file example as input):

[Cloud] Customer 1 PDROP-0000000 case-related task 1                          0.75hrs
[Cloud] Customer 1 task 1                                                     0.08hrs
[Cloud] Customer 2 task 2                                                     1.50hrs
[Internal] documentation creation                                             2.75hrs
[Internal] email triage                                                       0.75hrs
[Internal] internal task 1                                                    0.25hrs
[Internal] internal task 2                                                    0.25hrs
[On Prem] Customer 1 PDROP-0000000 case-related task 2                        0.08hrs
[On Prem] Customer 3 external meeting                                         0.75hrs

Grand total: 7.16

This is the only Rust program that does not necessarily assume the log will be named in the ISO-8601 date format with .log filename extension (e.g. 2022-11-30.log, 2022-12-19.log, etc.). You can run this periodically throughout the day as you build the log file to see how much time you've put in thus far.

doprocess usage

doprocess relies on the same internal Rust functions as timetracker, with the added output feature of separating the various categories into their own sections, for even easier transfer to Clarizen. With the new Clarizen categories, this makes entry into Clarizen take five minutes or less. You may want to sort your categories in Clarizen, to match the sort out of doprocess. Each section has its own total, with a running subtotal you can use to confirm you haven't missed anything as you transfer them to Clarizen. Its output looks like this:

[Cloud] Customer 1 PDROP-0000000 case-related task 1                          0.75hrs
[Cloud] Customer 1 task 1                                                     0.08hrs
[Cloud] Customer 2 task 2                                                     1.50hrs

Section total:      2.33hrs
Subtotal:           2.33hrs


[Internal] documentation creation                                             2.75hrs
[Internal] email triage                                                       0.75hrs
[Internal] internal task 1                                                    0.25hrs
[Internal] internal task 2                                                    0.25hrs

Section total:      4.00hrs
Subtotal:           6.33hrs


[On Prem] Customer 1 PDROP-0000000 case-related task 2                        0.08hrs
[On Prem] Customer 3 external meeting                                         0.75hrs

Section total:      0.83hrs
Subtotal:           7.16hrs



Grand total: 7.16

For everything but the [Internal] category, you should be able to copy the entire section with Section total to a single line item in Clarizen. The Section total and Subtotal for the [Internal] category can be input as separate line items for each entry, and these totals help ensure you haven't missed one or input its duration improperly.

chug usage

chug does not take a filename, or a log in the above format. Instead, it looks at the current week, and calculates the grand total for each day. If any day is missing, because the log file doesn't exist, it will print 0.00 for that day, like so:

2022-12-19.log
8.38

2022-12-20.log
0.00

2022-12-21.log
0.00

2022-12-22.log
0.00

2022-12-23.log
0.00

2022-12-24.log
0.00

2022-12-25.log
0.00

Grand total: 8.38

You can use this to review Clarizen, and make sure all time is input correctly. If you'd like to look at a previous week, you can pass the optional argument 1 or +1, n or +n where n is an unsigned integer for even further back. If you will be out of office on extended leave for the next week, and wish to run chug for it, pass a negative integer (e.g. -1, or -n). The formula for the week chug will look at is as follows:

<last Monday> - <week offset>
# where if offset is positive (either with or without a leading '+', it will look back).  If the offset is negative it will look forward
# <last Monday> will be TODAY if today is Monday

Caveats about Rust programs

The Rust programs have similar designs to the older Python and Bash scripts, but Rust is less forgiving regarding errors in the supplied logs (see format above). Namely, if you have log entries without a category, or try to use more than one category ([In Square Brackets]) in a single entry, Rust timetracker, doprocess, and chug programs will panic and error out. They should give a hint as to why they panic, but there is automatic Rust output that may not make it obvious what the problem is. Due to a recent update to the timelogging library code, too many Begin or End timestamps should be correctly identified. The programs still panic, but they give you an indication on what the problem is. For instance, with too many Begin timestamps, you'll see an error like the following:

thread 'main' panicked at 'ERROR:  Missing more than one End', timelogging/src/lib.rs:97:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Similarly, if you have too many End timestamps, you'll get an error that looks like this:

thread 'main' panicked at 'ERROR:  Missing a Begin', timelogging/src/lib.rs:101:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

For correct operation, there should be an equal number of Begin and End timestamps. If you're in the middle of a task, the last End is optional, if it's the first entry of that task it will be the minimum duration (0.08 hours).

Older Python and Bash scripts, for comparison

timetracker.py usage

The timetracker.py script is the basis, it tallies the time for each task and then outputs a report on how much each task takes to complete. It takes as its argument one or more text files, the log files in the format above. It should only report one line item per task, regardless of how many times it appears in the log. Each task should have a category/tag in square brackets, the behavior of do_process.sh (see below) is now undefined if the tag is left out.

Note that the time gets rounded to the nearest quarter hour. If it would round to zero hours (0.00hrs), the script replaces it 0.08hrs, which corresponds to 288 seconds. If you don't want a task to be set to 0.08hrs, simply leave it out of the log. Prior to the merge of nearest-quarter-hour, everything was rounded UP to the nearest quarter hour. That meant that if a task took 30 minutes and one second, it would be rounded UP to 0.75hrs (45 minutes). On longer days this had the effect of inflating the daily total, such that 10 hours of actual work was being inflated to 12 or even 13 hours on some occasions. Now that it rounds up or down to the nearest quarter hour, timetracker.py is much more accurate. There still is a bit of inflation for the short tasks (anything taking less than 288 seconds), but it shouldn't be as bad as adding an extra 15 minutes to tasks that are close to quarter hour boundaries.

The output looks like this:

timetracker output

[Internal] documentation creation                    3.00hrs
[Internal] internal task 1                           0.25hrs
[Internal] internal task 2                           0.50hrs
[Cloud] Customer 1 PDROP-0000000 case-related task 1         1.00hrs
[Cloud] Customer 1 PDROP-0000000 case-related task 2         0.25hrs
[Cloud] Customer 1 task 1                                    0.08hrs
[Cloud] Customer 2 task 2                                    1.75hrs
[Cloud] Customer 3 external meeting                          0.75hrs
[Internal] email triage                 0.75hrs
[Internal] internal task 1              0.08hrs

Section total:          8.41hrs

You may have noticed, the output of timetracker.py is in alphabetical order by category, then by task. This follows for ./do_process.sh and ./chug.sh below.

do_process.sh

This script filters the output of timetracker.py, giving each category (in square brackets) its own section, with its own tallies. The header is the list of categories and the log file basename (below this is example.log, but would normally be a date such as 2017-12-08.log). After the header, each line of output and sections are designed to be directly transferred to Clarizen manually, and the tally used to verify the daily and weekly totals in Clarizen. It calculates the time total for each organization/category. It also prints the subtotal thus far, to be sure no items are missed when adding them to Clarizen. At the end it prints a grand total for the day, which should be used along with the subtotals to cross-verify in Clarizen.

[Internal]
[Cloud] Customer 1
[Cloud] Customer 2
[Cloud] Customer 3
[Internal]
--
example



[Internal] documentation creation                    3.00hrs
[Internal] internal task 1                           0.25hrs
[Internal] internal task 2                           0.50hrs

Section total:          3.75hrs
Subtotal:  3.75hrs

[Cloud] Customer 1 PDROP-0000000 case-related task 1         1.00hrs
[Cloud] Customer 1 PDROP-0000000 case-related task 2         0.25hrs
[Cloud] Customer 1 task 1                                    0.08hrs

Section total:          1.33hrs
Subtotal:  5.08hrs

[Cloud] Customer 2 task 2                                    1.75hrs

Section total:          1.75hrs
Subtotal:  6.83

[Cloud] Customer 3 external meeting                          0.75hrs

Section total:          0.75hrs
Subtotal:  7.58

[Internal] email triage                 0.75hrs
[Internal] internal task 1              0.08hrs

Section total:          0.83hrs
Subtotal:  8.41hrs


Grand total:          8.41hrs

It takes as its argument a filename with the current date log (date +%F format, such as ./do_process.sh 2020-03-12.log), or it assumes the current date log file. Also, arbitrary filenames can be passed, so example.log becomes ./do_process.sh example.log.

Projects in Clarizen are listed alphabetically, so to transfer data from the ./do_process.sh output you simply copy the data lines from the ouput, and paste it into the notes section of the Clarizen entry. Enter the duration of the task at the top of the Clarizen entry, and select the category and subcategory of the entry.

chug.sh

This script is designed to be run on Mondays, after the previous week of log files have been generated and closed out. The standard Monday usage takes no arguments, it expects all log files to be processed to be in the current directory. It runs do_process.sh once for each day of the previous week, cleanly skipping any log files which do not exist. It pauses after each day report is output, allowing the user to transfer the times manually to Clarizen.

./chug.sh takes a single optional argument, a week offset (in case ./chug.sh is executed for log files further back than last week). This uses the GNU date functionality of calculating "last Monday." On Monday this will be "today - 7 days", but on the following Tuesday this will evaluate to "yesterday". If Monday is a holiday and you're entering your timesheets on Tuesday you can enter ./chug.sh 1 and it should do the right thing. If it's the first Monday of the month and you need to process the previous four weeks of logs, use ./chug.sh 4. The output of ./do_process.sh prints the date that is being processed at the top, if your incantation of ./chug.sh is wrong, you can quit and adjust accordingly. ./chug.sh with no arguments is equivalent to ./chug.sh 0.

The output is the output of ./do_process.sh piped to less for each day, pausing so the user can go through that day's output and transfer the items to Clarizen. If no time was logged for a given day (the file does not exist), ./chug.sh prints the missing date, but otherwise silently skips it. All seven days of the week are processed, Monday through Sunday.

If you're only interested in the weekly summary (and don't want the output of ./do_process.sh <date>.log | less), you can pass the -i option to ./chug.sh, like so: ./chug -i, or if you need a prior week, e.g. ./chug -i 2. The output will look similar to the following:

2020-03-16
9.05

2020-03-17
10.90

2020-03-18
11.96

2020-03-19
7.50

2020-03-20
9.38

2020-03-21
0

2020-03-22
0

Weekly Total:  48.79

month-pack.sh

timetracker.py and the related do_process.sh and chug.sh scripts are designed to have each day with its own YYYY-MM-DD.log file in the current, timetracker directory. Over time, the log files in this directory can become quite numerous and unwieldy. To help combat this, month-pack.sh takes all the log files from the previous month, adds them to a compressed tarball, and deletes them from the directory. It is designed to be run once all of the log files for the previous month have been processed into Clarizen.

year-pack.sh

In the same vein as month-pack.sh, year-pack.sh tars up all the monthly tarballs (named YYYY-MM.tar.xz), and puts them into a single YYYY.tar file. It is designed to be run in January when all of the previous December log files have been processed. All of the YYYY-MM.tar.xz files will be deleted once the YYYY.tar file is created.

CAVEATS

Tallies

Since I (the author, Trey Blancher) have been using this system to keep track of time, I've noticed that sometimes either Clarizen or these scripts get slightly off. Usually it's no more than 0.25 hours off in the tallies, but it gets time consuming trying to track down where the tally went wrong. If I do find the culprit, it's usually because I've entered the wrong time for a specific task in Clarizen (i.e., entering '0.08' instead of '0.25' for some tasks), or I've entered the time in the wrong cell.

2020-09-16 UPDATE

As of the institution of the running subtotals, the tallies being off in Clarizen is a MUCH less frequent problem. Usually if Clarizen is off it's because I missed an entry, or Clarizen didn't absorb an entry properly (that happens from time to time).

vim

The vim-specific files in this repository are tailored for my tastes. One big item of note, I've disabled vim cursor navigation with the arrow keys (Up, Down, Left, Right), to force me to get into the habit of using h, j, k, l for cursor navigation, and only navigate in normal mode (not insert mode). You will probably want to delete the following lines from .vimrc if you're not interested in the true Vim way®:

" Force me to stop using arrow keys
inoremap <Left> <Nop>
nnoremap <Left> <Nop>
inoremap <Right> <Nop>
nnoremap <Right> <Nop>
inoremap <Up> <Nop>
nnoremap <Up> <Nop>
inoremap <Down> <Nop>
nnoremap <Down> <Nop>

My Keyboard Maestro macros for timetracker make use of specific vim macros, and certain keybindings within .vimrc. The file timetracker.vim contains a couple of alternate macros for this purpose.

Several of the vim shortcuts I've programmed assist in managing the log text file. Invariably there will come a time where I forget to document a few tasks right away, so I piece together the logs from Slack and email, and edit the logs so it matches the format. One notable "normal" mode mapping is 'Y', which copies from the cursor position to the end of the line ('yanks' from the current position to the EOL, similar to 'C' or 'D' for changing or deleting to the EOL). It is equivalent to the action 'y$', but is only one keystroke instead of two. The Keyboard Maestro macro for 'End' tasks uses this mapping heavily.

Keyboard Maestro

Much of creating the log entries depends on specific Keyboard Maestro macros the author has set up. The file timetracker.kmmacros contains all of the macros used for this. The same text string trigger (Command + Shift + D, but you can assign it to any trigger) brings up a menu (known as a "conflict dialog" in KM), where I select 'Begin' or 'End'. The 'Begin' macro prints a log line using the current timestamp, like so:

2020-03-11 19:14:42:  Begin [Internal] timetracker doc

The 'End' macro copies the category and task name from the previous 'Begin' macro, to cut down on the amount of typing (and eliminate the possibility of typos). It then prints another 'Begin' line, ready for the next task. The output using the line above would look something like this:

2020-03-11 19:14:42:  Begin [Internal] timetracker doc
2020-03-11 19:30:36:  End   [Internal] timetracker doc
2020-03-11 19:30:36:  Begin 

I also have Keyboard Maestro text string triggers for common categories, to minimize typing. For instance, the text string triggers [css becomes [Internal] , [pro becomes [Internal] , [mm or [MM becomes [Mass Mutual] , [vzw becomes [Verizon Wireless] , [pin becomes [Company] (to match the Pindrop related items in Clarizen), etc. These are included in timetracker.kmmacros for your convenience.

See INSTALL.md in this repository for a detailed explanation of how to use the timetracker.kmmacros and set up the vim macros.