In April of 2010, I built this simple visualization using The Echo Nest’s analyze API data. The version below is modified to work without audio, as the original included an MP3. I believe this visualization was heavily inspired by something I’d seen at the time, but unfortunately I don’t remember what to include credit.

I am not a fan of the colors.

I saved a recording of it running in the iPad simulator:

In 2010, the MBTA added GPS devices to their buses, and made the data available in real time.

I collected and stored a lot of this data, with plans to make some sort of useful app. That never happened, but it was good excuse to make some simple visualizations. I took a day’s worth (June 15th, 2010) of the data and used the canvas API to draw it with varying styles.

Try it here: http://static.ryanberdeen.com/projects/bus/viz-demo-0.1/play.html. Clicking on the canvas allows you to choose the style. This is the result of “cumulative”:

The source is on GitHub: https://github.com/also/busviz

I’ve saved a couple of videos in case it ever stops working.

A full week of the data I started with is saved at http://static.ryanberdeen.com/projects/bus/data/vehicleLocations.csv (150 MB).

timestamp,id,routeTag,dirTag,lon,lat,secsSinceReport,heading,speedKmHr,predictable
1276573357828,"1117",null,null,-71.0644939,42.3318597,26,-4,0,false
1276573357828,"0699","23","out",-71.0807281,42.3055323,58,158,0,true
1276573357828,"6003","751","out",-71.0743703,42.338341,25,227,0,true
1276573357828,"0697","28","28_28_0063v0_1",-71.09268059999999,42.2672352,58,217,0,true

I’m not sure about the data quality. I made a heatmap of the position of the 77, and it seems to spend a lot of time off its route.

With links to:

• http://portfolio.ryanberdeen.com/ (dead)
• http://things.ryanberdeen.com/
• https://github.com/also/

Elsewhere, I had

iTerm2 now has the ability to copy with styles. When showing an example of running terminal commands, I’ve often wished there was an easy way to include the original color, and that’s now almost easy.

This is the result of what I’ve come up with:

The copying part is easy, but the pasting is another story. For things like Keynote presentations, it works great, but for pasting into a Markdown file there’s no easy path–pasting there will just result in plain text.

Apple has a simple example application, ClipboardViewer that shows the contents of the clipboard. When copying something, the application can provide many different types of content to be used in different situations, so Keynote gets rich text, while in text editor gets plain text.

I’m guessing that Keynote is getting RTF. public.rtf looks promising, but it’s still not going to be possible to embed it in a Markdown file. Converting it to HTML seems the way to go.

The sample code also gives a good example of the APIs needed to get at the data, NSPasteboard. This would probably be a good opportunity for me to learn Swift, but that seems pretty involved. Instead, I decided to try the NodObjC library, which makes it easy to call Objective-C APIs from Node.js. It’s probably just as easy to use one of the equivalent libraries in Python or Ruby.

A bit of trial and error with the API allows me to get at the pasteboard:

var $ = require('nodobjc');
$.framework('AppKit');

// the data I want will be on the "general" pasteboard
var pboard = $.NSPasteboard('generalPasteboard');

// there can be multiple items, but I probably want the first
var item = pboard('pasteboardItems')('objectAtIndex', 0);

// get the array of type names
var types = item('types');

// loop over the types and log them
for (var i = 0, len = types('count'); i < len; i++) {
  console.log(types('objectAtIndex', i));
}

Running this gives the list of available types, like ClipboardViewer:

$ node list.js
public.rtf
public.utf16-external-plain-text
public.utf8-plain-text
dyn.ah62d4rv4gk81n65yru
com.apple.traditional-mac-plain-text
dyn.ah62d4rv4gk81g7d3ru

I’ll try getting the public.utf8-plain-text value first, since that will be easier to deal with:

var data = pboard('dataForType', 'public.utf8-plain-text');

Whoops:

TypeError: error setting argument 2 - writePointer: Buffer instance expected as third argument

The exceptions from NodObjC can be a bit confusing. As it turns out, it doesn’t automatically convert JS strings to NSStrings.

var data = pboard('stringForType', $('public.utf8-plain-text'));
console.log(data);

This works! It’s a primitive version of pbpaste:

$ node pbpaste.js
Lorem ipsum dolor sit amet, consectetur adipiscing elit...
$ pbpaste
Lorem ipsum dolor sit amet, consectetur adipiscing elit...$

A simple change gets us access to the RTF data:

$ node pbpaste-rtf.js
{\rtf1\ansi\ansicpg1252\cocoartf1404\cocoasubrtf340
{\fonttbl\f0\fnil\fcharset0 Monaco;}
{\colortbl;\red255\green255\blue255;\red0\green187\blue0;\red255\green255\blue255;\red187\green187\blue0;
\red0\green0\blue187;}
\pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\partightenfactor0

\f0\fs24 \cf2 \cb3 \ulnone Lorem ipsum \cf4 dolor sit amet, \cf5 consectetur adipiscing elit...}

I can redirect this to a file, and it opens in TextEdit:

Now, how to convert this to HTML? This node-rtf-reader library looks promising, but it has no documentation and didn’t immediately work when I tried it.

TextEdit has the ability to save HTML, and I guessed that this was based on some API. It didn’t take too much digging to discover NSAttributedString AppKit Additions

The Application Kit extends Foundation’s NSAttributedString class by adding support for RTF, RTFD, and HTML…

I can create a NSAttributedString using initWithRTF:documentAttributes:, which takes NSData. I’ll need to get that instead of the string content of the pasteboard.

var rtfData = pboard('dataForType', $('public.rtf'));

var attrString = $.NSAttributedString('alloc')(
  'initWithRTF', rtfData,
  'documentAttributes', null);

To get HTML out, I can use dataFromRange:documentAttributes:error:, which again returns NSData.

var toHtmlOpts = $.NSDictionary(
  'dictionaryWithObject', $.NSHTMLTextDocumentType,
  'forKey', $.NSDocumentTypeDocumentAttribute);

var htmlData = attrString(
  'dataFromRange', $.NSMakeRange(0, attrString('length')),
  'documentAttributes', toHtmlOpts,
  'error', null).toString();

Now I just need to get the HTML string. NSString has a method to do that:

var html = $.NSString('alloc')(
    'initWithData', htmlData,
    'encoding', $.NSUTF8StringEncoding);
console.log(html);

Finally!

$ node pbpaste-html.js
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1404.34">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #0225c7}
</style>
</head>
<body>
<p class="p1">Lorem ipsum dolor sit amet, consectetur adipiscing elit...</p>
</body>
</html>

This HTML has a bit more to it than I want. It’s not easily included in an existing HTML or Markdown document, since it includes <html> and <body> tags, and those CSS classes seem likely to conflict with existing ones. I’ll clean that up.

I’ll use cheerio to extract the HTML I want, and PostCSS Nested to add a class to all the selectors in the stylesheet. The class name will be generated using shortid so it is less likely to conflict.

var cheerio = require('cheerio');
var postcss = require('postcss');
var shortid = require('shortid');

var doc = cheerio.load(html);

var wrapperClass = 'rtf-' + shortid.generate();

postcss([require('postcss-nested')])
  .process('.' + wrapperClass + ' {' + doc('style').text() + '}')
  .then(function (nestedCss) {
    console.log('<style>\n' + nestedCss + '\n</style>');
    console.log('<div class="' + wrapperClass + '">' + doc('body').html() + '</div>');
  });

This is looking promising:

<style>
.rtf-NkMhqhLal p.p1 {
    margin: 0.0px 0.0px 0.0px 0.0px;
    font: 12.0px Monaco;
    color: #0225c7
}
.rtf-NkMhqhLal span.s1 {
    color: #00c200
}
.rtf-NkMhqhLal span.s2 {
    color: #c7c400
}
</style>
<div class="rtf-NkMhqhLal">
<p class="p1"><span class="s1">Lorem ipsum </span><span class="s2">dolor sit amet, </span>consectetur adipiscing elit...</p>
</div>

There are a few ways the CSS or HTML could be improved, but I’m hoping this will be quite good enough for most cases.

The code could also use a few improvements–it wouldn’t be suitable for using for anything other than a CLI, since it doesn’t bother to release any of the objects it allocates.

Finally, while this was an interesting exercise, it’s a pretty roundabout way of getting the formatted text. It seems like a better solution will be to copy the text with the ANSI escape codes directly and convert that to HTML.

Anyway, here’s the code for pbpaste-html.js, formatted with rougify --formatter-opts theme=base16, then run through pbpaste-html.js:

Flow has some helpful behavior for object literals:

/* @flow */
type T = {x: number};
const o = {};
o.x = 1;
(o : T);

Without the o.x = 1; line, the cast to T will fail:

src/flow-test.js:5
  5: (o : T);
          ^ property `x`. Property not found in
  5: (o : T);
      ^ object literal

Because the object literal is empty, it is “unsealed”. If it was created as const o = {y: 2};, Flow would fail on o.x = 1;

src/flow-test.js:4
  4: o.x = 1;
       ^ property `x`. Property not found in
  4: o.x = 1;
     ^ object literal

This same Flow behavior comes into play with Object.assign().

/* @flow */
Object.assign({x: 1}, {y: 2});

{x: 1} is sealed, so the property y can’t be added.

src/flow-test.js:2
  2: Object.assign({x: 1}, {y: 2});
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ property `y` of object literal. Property not found in
  2: Object.assign({x: 1}, {y: 2});
                   ^^^^^^ object literal

Object.assign({}, {x: 1}, {y: 2}); will work, because {} is unsealed. Flow knows the shape of the result. Running flow suggest:

src/flow-test.js
--- old
+++ new
@@ -1,2 +1,2 @@
 /* @flow */
-const o = Object.assign({}, {x: 1}, {y: 2});
+const o: {x: number, y: number} = Object.assign({}, {x: 1}, {y: 2});

o is unsealed, which I don’t think you can tell using type-at-pos. This has some unfortunate consequences in a less contrived example:

/* @flow */
type T1 = {x : number};
type T2 = {y : number};
type T3 = T1 & T2;

function merge(t1 : T1, t2 : T2) : T3 {
  const result = Object.assign({}, t1, t2);
  console.log(result.info.toString()); // at one time, T1 contained info, but it was removed
  return result;
}

This typechecks, but because result is unsealed, Flow won’t prevent the access of the possibly undefined property result.info. Flow still suggests the {x: number, y: number} type, even though this would cause the code not to typecheck.

I’m impressed by the inferences Flow is able to make, and I’d like it to be stricter about what it allows. There are very few locations where I’d make use of unsealed objects, so I’d like them to be optional–either globally, set in .flowconfig, or as a type (UnsealedObject instead of Object?).

In Flow, a class may be defined using ES6 syntax extended with field declarations.

— http://flowtype.org/docs/classes.html

In Babel 5, this worked with the default configuration:

/* @flow */
class C {
  s : string;
  constructor(s) {
    this.s = s;
  }

  split() {
    return this.s.split('');
  }
}

The s : string; line is required by Flow; without it Flow will complain:

src/flow-test.js:5
  5:     this.s = s;
         ^^^^^^ assignment of property `s`
  5:     this.s = s;
              ^ property `s`. Property not found in
  2: class C {
           ^ C

In Babel 6, most Flow annotations continue to work as expected, but this class annotation now causes an error:

SyntaxError: src/flow-test.js: Missing class properties transform.
  1 | class C {
> 2 |   s : string;
    |   ^

The class properties transform the error is referring to is transform-class-properties, for “Class Property Declarations”, currently a stage 1 proposal.

Flow doesn’t depend on future ECMA262 proposals, so this seems like a mistake.