Category Archives: Uncategorized

Thinking before Typing

As I get older, I spend less time typing than I used to.  It has increased over time, but I used to dive right in and code a solution. 

This struck me today as I struggled with solving a problem that seemed simple enough.  Here’s the setup:

We’ve got a system that continually ingests tweets.  That’s not quite right, it is supposed to continually ingest.  Nothing ever goes as planned, though.  It crashes. We stop it for upgrades.  In either case, when it isn’t running we’re losing content.  Thankfully, we have the ability to “replay” a period of time.  

To date this is done manually, by noting the times when it stopped and started, and then punching those times into the replay.  It’s tedious, error prone, and might lose content without notice.

A long time ago I worked on an eBay style marketplace for industrial equipment in Europe.  I had to implement the billing generator.  The bill rate was broken into time chunks, the first day was most expensive, the next 2 days less, and so on.  The listing could be stopped and started, but pausing the billing in a similar manner.  It was complex, and I shudder to think of what my solution might have been.  I know that dealing with dates and times is tricky. 

In a nutshell, I’ve got a continuous stream of data that I want to capture completely.  I can replay holes in the timeline by specifying a start and end timestamp.  

So how do you go about solving this robustly?  The stream of data isn’t very identifiable, so overlapping time periods is hazardous.

I spent a couple hours noodling on it, not typing a thing.  In the end I was happy with the solution I came up with.

Of course, it’s also possible that I just think slower now.