TED Electricity graph: 28 Feb 08 Power

eastpole    says:

I am assuming that the regular spikes are due to our kitchen fridge coming on. Does that make sense?
Posted 22 months ago. ( permalink )

eastpole    says:

Interesting fact about gnuplot: the program can output, but not input, times that include an AM or PM indicator. Inputs must be 24-hour. Unfortunately, TED puts out lines like this:
"28/02/2008 12:13:37 AM","0.54","0.05"," 119.3"

Note that times are not 0-padded and always include AM or PM.
So, I'm trying to ramp up my perl skills to write a filter that does this. It's a very easy task but I don't know enough yet (i.e. I know vanishingly little). So wish me luck -- I'll make something available if I get it working.

That's the explanation for the odd X axis, by the way.
Posted 22 months ago. ( permalink )

eastpole    says:

I notice that TED doesn't actually manage to get a kW reading each
and every second. Who can blame it? That's a lot to ask.

If it did, there would be 86400 lines (24*60*60) in a log file that
records the reading from each second.

storyteller:/Volumes/archivist$wc -l TED*
83151 TED-s-01Mar08.csv
84218 TED-s-02Mar08.csv
83494 TED-s-03Mar08.csv
80590 TED-s-04Mar08.csv
84970 TED-s-05Mar08.csv
82967 TED-s-06Mar08.csv
83221 TED-s-07Mar08.csv
86167 TED-s-27Feb08.csv
85229 TED-s-28Feb08.csv
84970 TED-s-29Feb08.csv

For each day, the number shows the number of lines (hence seconds)
recorded. So my TED is capturing between 93% and 99% of the per-second values.

I'm pretty pleased with that.
Posted 22 months ago. ( permalink )

eastpole    says:

I was thinking I might be able to fish CSV out of the database myself. Maybe not:

tai@scribe:/mnt/archivist$ file TED.db
TED.db: SQLite 3.x database

Looks good! But no:

tai@scribe:/mnt/archivist$ sqlite TED.db
Unable to open database "TED.db": file is encrypted or is not a database

Would they encrypt it?

tai@scribe:/mnt/archivist$ strings TED.db | head -5
SQLite format 3
KWPROJMAX500
KWNOWSCALE0.5
KWNOWMAX10
KWMTDSCALE20

If so, that's not very well encrypted... So, what could be wrong?

EDIT LATER:
tai@scribe:/mnt/archivist$ sqlite -version
2.8.17

Versions could be a problem.
In investigating, I learned that sqlite tries to make the database files compatible across all kinds of architectures and OS.
Posted 22 months ago. ( permalink )

eastpole    says:

Good progress. The TED.db file has a bunch of tables:

LP_EVENT_HISTORY RDU_DAY_DATA RDU_MONTH_DATA
LP_EVENT_LOG RDU_HOUR_DATA RDU_PROPERTIES
LP_EVENT_TYPES RDU_MINUTE_DATA RDU_SECOND_DATA

And the data tables look like this:

0633396629400000000|0.3473334|0.0|119.1317
0633396630000000000|0.3473334|0.0|119.1034
0633396630600000000|0.3433898|0.0|118.7322
0633396631200000000|0.3347458|0.0|118.7677
0633396631800000000|0.3854547|0.0|118.8036
0633396632400000000|0.3760002|0.0|118.7666
0633396633000000000|0.3711865|0.0|118.6509
0633396633600000000|0.3623334|0.0|118.66
0633396634200000000|0.3695|0.0|118.6634

That looks helpful, I like it a lot.
Posted 22 months ago. ( permalink )

eastpole    says:

I am wondering what the heck

0633396633600000000

39505.405 (decimal days since 30 Dec 1899).

1055661053

eastpole    says:

I can say for sure that there is some zaniness out there.

A file time is a 64-bit value that represents the number of 100-nanosecond intervals that have elapsed since 12:00 A.M. January 1, 1601 Coordinated Universal Time (UTC).

128485252450000000

eastpole    says:

In the good news category about the TED, I can now confirm that the database keeps growing (i.e. TEDService runs) even without anyone logging in to the Windows box. This is good because it means that a power failure or reboot doesn't lead to a long data loss.

Unless the reboot hangs somehow, which they almost always do in Windows. Ah, Windows.

Also, the cygwin ssh server makes me happy. :)
Posted 22 months ago. ( permalink )

eastpole    says:

The time format does also not appear to be either hex or dec TAI64.

tai@scribe:~/bin$ date +"16o2 62^%s+2 32^*p" | dc
4000000047DE9F3A00000000

tai@scribe:~/bin$ date +"62^%s+2 32^*p" | dc
dc: stack empty
5178752010807148544

I'm quite impressed with the many different time formats that Perl modules can help you deal with. But I need to figure out which one it is! I have emailed Energy Inc. to see if they'll tell me.
Posted 22 months ago. ( permalink )

eastpole    says:

I've written a tool to fix the CSV dumps so that they read in something like a standard date format. It's fairly fast:

tai@scribe:/mnt/archivist$ time perl fixted.pl >FTED-s-28Feb08.csv

real 0m9.378s
user 0m5.304s
sys 0m0.096s

tai@scribe:/mnt/archivist$ wc -l *28Feb08*
85229 FTED-s-28Feb08.csv
85229 TED-s-28Feb08.csv

So, 9 seconds for 24-hours of data taken at 1 sec frequency? Not bad. Thanks to Eric J. Roode who wrote Time::Normalize.

I am now posting the new version graph of this same data with meaningful times of day.
Posted 21 months ago. ( permalink )

datsun2liter    says:

How are you opening the database file?

I'd like to do it with Visual Basic..

we need to start a message board somewhere on this...

you're lucky, i run about 3 kw/hr. Right now i'm running 11...

TED does make you aware of power usage..
Posted 20 months ago. ( permalink )

eastpole    says:

3 kW all the time?
Florida sounds like it needs a couple nuclear plants to keep cool. :)

Further chat in email, datsun2liter.
Posted 20 months ago. ( permalink )

eastpole    says:

I wonder about this!

Microsoft .NET C#epoch = (DateTime.Now.ToUniversalTime().Ticks - 621355968000000000) / 10000000;

I read: "Time values are measured in 100-nanosecond units called ticks, and a particular date is the number of ticks since 12:00 midnight, January 1, 0001 A.D."

The DateTime value type represents dates and times with values ranging from 12:00:00 midnight, January 1, 0001 Anno Domini (Common Era) through 11:59:59 P.M., December 31, 9999 A.D. (C.E.)

Time values are measured in 100-nanosecond units called ticks, and a particular date is the number of ticks since 12:00 midnight, January 1, 0001 A.D. (C.E.) in the GregorianCalendar calendar. For example, a ticks value of 31241376000000000L represents the date, Friday, January 01, 0100 12:00:00 midnight. A DateTime value is always expressed in the context of an explicit or default calendar.
Posted 15 months ago. ( permalink )

eastpole    says:

Confirmed!
Posted 15 months ago. ( permalink )

torben812    says:

0633775104000000000

so is there an explicit calendar or offset used in TED?
Posted 8 months ago. ( permalink )

eastpole    says:

torben,

It looks to me like like the TED doesn't care about the time; the unit doesn't send any timestamp. The software on the PC side stores data with a timestamp in the Microsoft .NET C# epoch show above.

The number you give, for instance, is the number of 100-nanosecond ticks since midnight on 1 January 1 AD.

msdn.microsoft.com/en-us/library/system.datet ime.aspx

with some division, subtraction, and multiplication, and a calendar, I make your #,

0633775104000000000

out to be on the 129th day of 2008, so, I guess, the early afternoon of May 8th, 2008. Am I right?
Posted 8 months ago. ( permalink )

eastpole    says:

torben, do you need me to show the arithmetic? It starts by dividing the big number by 10000000, to get seconds since 1 AD instead of 100-nanosecond ticks. Should I go on?
Posted 8 months ago. ( permalink )

eastpole    says:

Note there are now a couple very simple "listeners" available for Linux and Mac OS X, so now I am logging my per-second usage without needing to leave a Windows box up all the time (and freezing, and rebooting, and ...)

The man to see for this stuff is Micah Dowty -- he's awesome.
Posted 7 months ago. ( permalink )