Laboratory Notes

This is where I keep lists, notes, ramblings, and other ideas that don't fit the "blog post" style.

Suggested HL7 changes

new value type: ratio

add value type “Ratio” as in “1:64” ( which I received from LabCorp for 31147-2 ) R(something) RAT? RIO?

1
OBX|1|RATIO|006464^RPR, Quant.^labcorp.com^31147-2^Reagin ab^LOINC||1:64||NonRea<1:1|H||N|F|||201304081328|05

Reference Range segment

Add a reference range segment ( because labs like to include all possible reference ranges for some reason)
high, low, gender, age, start_date, end_date, phase, preconditions, analysis, severity

example:
1
2
  OBX|14|TX|015156^Basos^labcorp.com^706-2^Basophils/100 leukocytes^LOINC||Comment|%|0-3|||N|F|20010402||201305150937|02
  REF|0|3|female|21-30|||menopause||H|Critical
or alternately something more open to change:
1
2
3
  REF|2|3
  RIF|gender|male   # note you would obviously need some additional meta-data
  RIF|age|<20         # note you would obviously need some additional meta-data

another example found in the wild:

1
2
3
4
5
6
7
8
9
10
11
OBX|1|NM|602548^M005-IgE Candida albicans^XYZ^6059-0^Candida albicans Ab.IgE^LOINC||56.12|kU/L|Class V|A||N|F|||201404376543|01
NTE|1|L|    Levels of Specific IgE       Class  Description of Class
NTE|2|L|    ===================================
NTE|3|L|                   < 0.10         0         Negative
NTE|4|L|           0.10 -    0.31         0/I       Equivocal/Low
NTE|5|L|           0.32 -    0.55         I         Low
NTE|6|L|           0.56 -    1.40         II        Moderate
NTE|7|L|           1.41 -    3.90         III       High
NTE|8|L|           3.91 -   19.00         IV        Very High
NTE|9|L|          19.01 -  100.00         V         Very High
NTE|10|L|                  >100.00         VI        Very High

Documentation Links

1
2
3
4
5
6
7
8
9
10
NTE|3|| Reference Range: < 5.7% Decreased risk of diabetes
NTE|4|| 5.7-6.0% Increased risk of diabetes
NTE|5|| 6.1-6.4% Higher risk of diabetes
NTE|6|| > OR = 6.5% Consistent with diabetes
NTE|7|| 
NTE|8|| These Reference Intervals are supported by the 
NTE|9|| current "Standards of Medical Care in Diabetes"
NTE|10|| published in January of the current year in
NTE|11|| Diabetes Care, the Journal of the American
NTE|12|| Diabetes Association.

I want to go back and look at some examples of how these things are cited in footnotes before I move forward, though my preference would be to use a URI like this:

1
http://care.diabetesjournals.org/content/37/Supplement_1/S14.extract

or maybe there’s a ‘book’ URI or [URN]:https://en.wikipedia.org/wiki/Uniform_resource_name
, something like

urn:isbn:0451450523

Specifically prohibit ‘L’ or anything referencing the ‘local’ code set. CE3 and CE6 coding systems should be intended to be unique.

So if you’re sending LOINC codes, you should use the coding system LOINC
if you work at LabCorp then you should use LabCorp.com or some similar value you think is unique.

compendium schema/message, compendium request schema/message

example:

1
2
3
4
5
6
MSH|^~\\&|LAB|LAB||95507|20131113135322||COMPENDIUM
NOU|015156^Basos^labcorp.com^706-2^Basophils/100 leukocytes^LOINC|
REF|0|3|female|21-30|||menopause||H|Critical
REF|2|7|female|31-50|||follicular||H|OK
REF|0|20|male|21-30|||menopause||H|OK

better notes

  • result status messages
    • Test Pending ( see example below )
    • Quantity Not Sufficient
    • not indicated by other results
  • operational messages
    • tests will be deprecated
    • panel will be deprecated
    • reference ranges will / have changed
  • Mismatched Data
    • gender is male but you supplied a LMP

“pending” example:

1
OBX|2|ST|2862-1^ALBUMIN^LOINC^28621^ALBUMIN^questdiagnostics.com||Pending

instead of this:

1
2
3
4
5
6
7
8
9
10
MSH|^~\&|...
ORC|RE|***^LAB|***^LAB||||||201405190000|||8^RMA^C^^^^^N 
OBR|1|085555535^LAB|1409955555^LAB|001032^Glucose, Serum^labcorp.com|||201405190414||||||CC:74320000 CLINICAL INFORMATION 2 CLIN INFO3|201405191702||||082273035||082273035||201405191735||01|F 
OBX|1|CE|001032^Glucose, Serum^labcorp.com^2345-7^Glucose^LOINC||QNS|mg/dL||||N|X|20055555||201405195558|01 
NTE|1|L|Quantity was not sufficient for analysis. 
ORC|RE|082273035^LAB|14099463030^LAB||||||201405190000|||8^DERMA^C^^^^^N 
OBR|2|082273035^LAB|14099463030^LAB|100875^Request Problem^labcorp.com|||201405190414|||||||201405191702||||082273035||082273035||201405191735||02|F 
OBX|1|CE|100875^Request Problem^labcorp.com||QNS|||||N|F|||201405191708|02 
NTE|1|L|Quantity was not sufficient for analysis. 
NTE|2|L| TEST: 001032 Glucose, Serum 

something like this:

1
2
3
4
5
MSH|^~\&|...
ORC|RE|***^LAB|***^LAB||||||201405190000|||8^RMA^C^^^^^N 
OBR|1|085555535^LAB|1409955555^LAB|001032^Glucose, Serum^labcorp.com|||201405190414||||||CC:74320000 CLINICAL INFORMATION 2 CLIN INFO3|201405191702||||082273035||082273035||201405191735||01|F 
OBX|1|NM|001032^Glucose, Serum^labcorp.com^2345-7^Glucose^LOINC|||mg/dL||||N|X|20055555||201405195558|01
XCL|1|QNS^Quantity Not Sufficient^LabCorp|

XCL here is meant to be exclusion. EXC could be ok. Also ERR. Basically, this is a reason for TNP… so actually…

1
2
OBX|1|NM|001032^Glucose, Serum^labcorp.com^2345-7^Glucose^LOINC|||mg/dL||||N|X|20055555||201405195558|01
TNP|1|QNS^Quantity Not Sufficient^LabCorp|

Some people might say “Oh well, there are some times when we do perform a test but still don’t report a value.”

ok then

TNR for Test Not Reported?
VNR for Value Not Reported???

broader: focus on industry best practice schemas, not format.

Version 3 seems to shift away from pipe-separated-values to XML. It shouldn’t matter. There should be a “this is how it can be stored intelligently” schema. Whether it is communicated in psv, xml, json, etc., should not be relevant.

  • Created March 11, 2014
  • Updated September 04, 2014

AngularJS bits-and-pieces

show a partial in the current context:

  • Created June 18, 2013
  • Updated June 18, 2013

HL7 ORM

1
MSH|^~\&|PhysioAgeSystems|PhysioAgeReporting|HUBWS|THO|#{Time.new(2009, 11, 14, 10,32).strftime("%Y%m%d%H%M")}||ORM^O01|#{my_order_number}|P|2.3

References

http://www.interfaceware.com/hl7-standard/hl7-segment-MSH.html

  • Created November 27, 2012
  • Updated November 27, 2012

C# class: Week

wherein I create a model for a Week and then don’t use it on my project, leaving it here to rot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
using System;
using System.Globalization;

namespace ProBill
{
        public class Week
        {
                public int Year;
        public int WeekOfYear;
        
        public Week()
        {
                
        }
        
                public Week(short year, short week)
                {
                        this.Year = year;
                        this.WeekOfYear = week;
                }
                
                public Week Including( DateTime Date )
                {
                        Week output = new Week();
                        GregorianCalendar cal = new GregorianCalendar(GregorianCalendarTypes.Localized);

                        output.Year = Date.Year;
                        output.WeekOfYear = cal.GetWeekOfYear(Date, CalendarWeekRule.FirstFourDayWeek, DayOfWeek.Monday);

                        return( output );
                }
        }
}
  • Created November 15, 2012
  • Updated November 15, 2012

Handling "Login Incorrect" for Misys Tiger via ODBC

Recently, when connecting to Misys Tiger via ODBC ( see Previous Post ) I found that my user account could not access one company. The company name was “Old Smith”. Smith, in this case, was a physician whose books had been transitioned to another company for unknown reasons. Everyone was apparently locked out of that company to prevent chaos. Still, it came up in the list of companies.

Possibel ways I could handle this, in order of most to least preferable:

  • Find a way to remove it from the master list of companies I retrieved.
  • Filter that company from the list of companies with a blacklist (unpleasant)
  • Handle the exception and allow the app to just move on when it can’t access a company.

The master list of companies.

I generate List by retrieving client information from the Company_Shared.COMPANY_APC table. Looking through that table, I wasn’t able to find any columns that differentiated that company from the ones that should not be skipped.

However, I did notice two columns that might be willing to contain custom flags:
- apc_selected_flag
- apc_play

I will have to get with my Misys administrator to see whether there is a way to add custom data to those fields from the admin side.

Blacklisting

The thing about blacklisting companies is that it doesn’t scale well. I can find a list of companies that the test user can’t access and list those. It might even be a comprehensive list. But at some point, another company is going to be deprecated, and then the software will fail.

They could maintain their own blacklist, but we aren’t persisting any information in the system … so I would have to set up a database or something just for the purpose of storing a blacklist.

Hopefully I’ll find some other way.

Handling the error.

I’m in the early stages of this app. I’m going to have to handle all the ODBC errors at some point in the future.

  • Created November 15, 2012
  • Updated November 15, 2012

research on .NET interface to Creative Solutions

Looking for a way to push payments and invoices to Creative Solutions Accounting version 2012.0.9, Client Bookkeeping Services.

In researching this topic, I found that it’s basically impossible to research. “net” is already sort of ambiguous. Add “creative solutions” and Google has no idea what I want.

With that said, I’m not sure whether there is much information out there. I couldn’t even find a home page for this software. Apparently it has been deprecated in favor of “Accounting CS”, another winner-of-a-name.

I hope to get more information by visiting the clients’ office later this week.

  • Created October 28, 2012
  • Updated October 28, 2012

Interfacing with Misys Tiger

A client has asked me to create an interface between Misys Tiger and their accounting package.

Research Phase

Decompiling

Looking around their server, I found a set of DLL, EXE and GNT files. My initial research shows that GNT files likely some kind of cobol-to-unmanaged-dll files. I don’t plan to go down the GNT rabbit-hole in the near future. GNT files have something to do with MicroFocus.

I have had some success with decompiling .NET DLLs in the past, so I reached in that direction again.

DLLs can contain Native Assemblies or Managed Assemblies. Using ILSpy, a .NET compiler, I was able to decompiled the following managed assemblies… ie .NET:

1
2
3
4
5
6
7
8
9
10
HIDLibrary.dll # I assume HID means Human Interface Device, ie mouse, webcam, keyboard, etc.
IntuitQBMSGateway # apparently this allows you to charge credit cards
MSRClient.dll
Microfocus.COBOL.Messages.dll
MicroFocus.COBOL.Sql.Wrapper
MicroFocus.COBOL.Sql.RunTime
MicroFocus.COBOL.Runtime
MicroFocus.COBOL.VisualStudio
MSRClient
PrintXFDF

Unfortunately, some of the most interestingly-named files, however, could not be decompiled into .NET.

I also tried another compiler… 9net’s “Spices”… with no more success.

Looking around the few classes I was able to decompile, everything I see looks like hooks to the outside world. None of what I see has to do with the medical domain. I do see something about XML, and I would love to find that there’s some web-like API to Misys Tiger, but I just don’t think it’s likely. Seems like this software is written in cobol, and uses some sort of .NET shell for a few limited functions. I also saw something about JAVA. So it’s sort of a hodge-podge. :)

Subsequently, I think I’ll have to go in a different direction to access the data I need.

ODBC

Google brought me to a page with no useful or correct information that seems to suggest that Tiger provides an ODBC interface.

I found a blog post about connecting to AllScripts Tiger’s ODBC connection … it was written two years ago, but the information seems to suggest that an ODBC connection might be available.

I touched base with the author of that post. He said:

1
2
3
You’ll find that the ODBC drivers included with Misys Query only work on workstations, and attempting to install on a Server platform will only result in an error.

Transoft will want to sell you server drivers for this… ;)

At the client’s office, I got access to a workstation.

I navigated to “Control Panel > ODBC Data”. Under “User DSN” I found “Transoft ODBC Driver”. Hurah! boks32.udd. However, when I clicked “configure”, I got an error. “The setup routines for the Transoft ODBC Driver ODBC driver (sic) cound not be found. Please reinstall the driver.” After acknowledging that error, a second message popped up. “The specified DSN contains an architecture mismatch between the Driver and Application.” Peachy. Nothing else interesting in the ODBC config tool other than a SQL Server driver.

I still think ODBC is going to be part of the solution, but I decided to check out this Query tool. Because that’s totally what you should call the tool that issues queries. Because overloaded words are always a good idea.*

Misys Query

Too restrictive for a programmer, but I can appreciate the power this tool brings to a power-user. I was able to find a list of columns that had domain-specific terms like patient, insurance provider, etc. This seems like the Tiger data.

Playing around with Misys Query, I found that the software will translate these constructed queries into SQL, which will be very convenient moving forward.

Buried in the menus for Misys Query, I found an “ODBC Connection” button. Clicking it opened the ODBC window that was previously of no help… except now it had a bunch of connections available.

Apparently, you have to authenticate, or at least have Misys Query running, for these to be visible.

Queries into SQL

To turn Misys Query reports into SQL, load a saved report then go to Report > Query. Under the “Profile” tab, select the “SQL” radio button.

uSQL Error Messages

unknown statement group — does USQL not support the clearly-superfluous “group” statement?

column not grouped — so many it does support groups just … occasionally?

Writing a Spike

Without further ado, here’s the C sharp class I use to connect.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
using System;
using System.Data.Odbc;

namespace Wisteria
{
        /// <summary>
        /// Description of Class1.
        /// </summary>
        public class Tiger
        {
                public class Connection
                {
                        private OdbcConnection _connection = new OdbcConnection("DSN=Company_Shared");
                        
                        public Boolean isConnected
                        {
                                get
                                {
                                        return( _connection.State == System.Data.ConnectionState.Open);
                                }
                        }
                        public Connection()
                        {
:                                _connection.Open(); /* this will trigger a login dialog */
                        }
                }
                public Tiger()
                {
                }
        }
}

When you exclude the username and password from the connection string, a dialog is presented to the user. I prefer this option because it allows them to log in with their own credentials. If you wanted to include a username and password, I think this would be the way. I have not tested it.

1
DSN=Company_Shared;uid=MyUid;pwd=MyPass

References

TranSoft Developer’s Blog

StackOverflow: ODBC to Transoft

  • Created October 12, 2012
  • Updated November 06, 2012

MIME Types for Implementing HL7 via SMTP

After looking around, I found this mime type for hl7: application/edi-hl7

There are several ways to use the mime-type application/edi-hl7 — my thinking is that we should support all of them. Most of the work will be done by the mail handler gems anyway.

Entire email uses that mime-type

Very simple; does not include human-readable format.

1
2
3
4
5
6
7
8
9
10
11
12
13
Date: Tue, 02 Oct 2012 17:12:36 +0000
From: jw@mustmodify.com
To: jw@mustmodify.com
Subject: hl7 test
Mime-Version: 1.0
Content-Type: application/edi-hl7;
 charset=UTF-8
Content-Transfer-Encoding: 7bit

MSH|^~\&|DDTEK LAB|ELAB-1|DDTEK OE|BLDG14|200502150930||ORU^R01^ORU_R01|CTRL-9876|P|2.4
PID|||010-11-1111||Estherhaus^Eva^E^^^^L|Smith|19720520|F|||256 Sherwood Forest Dr.^^Baton Rouge^LA^70809||(225)334-5232|(225)752-1213||||AC010111111||76-B4335^LA^20070520
OBR|1|948642^DDTEK OE|917363^DDTEK LAB|1554-5^GLUCOSE|||200502150730|||||||||020-22-2222^Levin-Epstein^Anna^^^^MD^^Micro-Managed Health Associates|||||||||F|||||||030-33-3333&Honeywell&Carson&&&&MD
OBX|1|SN|1554-5^GLUCOSE^^^POST 12H CFST:MCNC:PT:SER/PLAS:QN||^175|mg/dl|70_105|H|||F

Email is multipart/alternative

This content type tells the email renderer that there are several version of the same content. We include one part that has a disclaimer, and one that has the HL7. “Here, I’m giving you text and HL7. Use whichever one you prefer… they have the same content.” In fact, the spec says that the parts should be included in order of preference The biggest problem with this is that obviously they don’t have the same content. Unless the lab included an HTML part with tables and charts, and also an HL7 part … that would be spectacular.

This is great in that the user will not see the HL7 content, which isn’t really human-readable anyway.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Date: Tue, 02 Oct 2012 14:22:38 +0000
From: me
To: me
Subject: hl7 test
Mime-Version: 1.0
Content-Type: multipart/alternative;
 boundary="--==_mimepart_506af8adf1691_6d3b44505f2825a6";
 charset=UTF-8
Content-Transfer-Encoding: 7bit



----==_mimepart_506af8adf1691_6d3b44505f2825a6
Date: Tue, 02 Oct 2012 14:22:38 +0000
Mime-Version: 1.0
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: <506af8ae11856_6d3b44505f28263@veronica.physioage.com.mail>

This document contains privileged medical information. If you are not the intended recipient, please discard it. Consuming this information could result in immediate death.

----==_mimepart_506af8adf1691_6d3b44505f2825a6
Date: Tue, 02 Oct 2012 14:22:38 +0000
Mime-Version: 1.0
Content-Type: application/edi-hl7;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: <506af8ae140cc_6d3b44505f2827fa@veronica.physioage.com.mail>

MSH|^~\&|DDTEK LAB|ELAB-1|DDTEK OE|BLDG14|200502150930||ORU^R01^ORU_R01|CTRL-9876|P|2.4
PID|||010-11-1111||Estherhaus^Eva^E^^^^L|Smith|19720520|F|||256 Sherwood Forest Dr.^^Baton Rouge^LA^70809||(225)334-5232|(225)752-1213||||AC010111111||76-B4335^LA^20070520
OBR|1|948642^DDTEK OE|917363^DDTEK LAB|1554-5^GLUCOSE|||200502150730|||||||||020-22-2222^Levin-Epstein^Anna^^^^MD^^Micro-Managed Health Associates|||||||||F|||||||030-33-3333&Honeywell&Carson&&&&MD
OBX|1|SN|1554-5^GLUCOSE^^^POST 12H CFST:MCNC:PT:SER/PLAS:QN||^175|mg/dl|70_105|H|||F


----==_mimepart_506af8adf1691_6d3b44505f2825a6--

Email is multipart/mixed

This is, apparently, a more correct type than multipart/alternative, in that all the parts should be displayed. Apparently the renderer can choose whether to display them inline or as attachments.

By default, Rails encoded my HL7 file with Base64. Since the API specifically allows you to change that, I’m going to assume the standard supports it.. The fact that the encoding is specified in the email suggests that it should be ok. Until I run into problems with a vendor, I’m going to run with it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Date: Tue, 02 Oct 2012 17:30:13 +0000
From: jw@mustmodify.com
To: jw@mustmodify.com
Message-ID: <506b24a5bd3e7_472e4dc5e0229589@veronica.physioage.com.mail>
Subject: hl7 test
Mime-Version: 1.0
Content-Type: multipart/mixed;
 boundary="--==_mimepart_506b24a5a926f_472e4dc5e0229384";
 charset=UTF-8
Content-Transfer-Encoding: 7bit



----==_mimepart_506b24a5a926f_472e4dc5e0229384
Date: Tue, 02 Oct 2012 17:30:13 +0000
Mime-Version: 1.0
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-ID: <506b24a5ba1e3_472e4dc5e02294b1@veronica.physioage.com.mail>

This document contains privileged medical information. If you are not the intended recipient, please discard it. Consuming this information could result in immediate death.

----==_mimepart_506b24a5a926f_472e4dc5e0229384
Date: Tue, 02 Oct 2012 17:30:13 +0000
Mime-Version: 1.0
Content-Type: application/edi-hl7;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename=file.hl7
Content-ID: <506b24a5a75cb_472e4dc5e0229218@veronica.physioage.com.mail>

MSH|^~\&|DDTEK LAB|ELAB-1|DDTEK OE|BLDG14|200502150930||ORU^R01^ORU_R01|CTRL-9876|P|2.4
PID|||010-11-1111||Estherhaus^Eva^E^^^^L|Smith|19720520|F|||256 Sherwood Forest Dr.^^Baton Rouge^LA^70809||(225)334-5232|(225)752-1213||||AC010111111||76-B4335^LA^20070520
OBR|1|948642^DDTEK OE|917363^DDTEK LAB|1554-5^GLUCOSE|||200502150730|||||||||020-22-2222^Levin-Epstein^Anna^^^^MD^^Micro-Managed Health Associates|||||||||F|||||||030-33-3333&Honeywell&Carson&&&&MD
OBX|1|SN|1554-5^GLUCOSE^^^POST 12H CFST:MCNC:PT:SER/PLAS:QN||^175|mg/dl|70_105|H|||F


----==_mimepart_506b24a5a926f_472e4dc5e0229384--

Doing all that in Rails

These are lab notes after all. Here’s the Rails implementation.

entire email is application/edi-hl7

1
2
3
4
5
6
7
8
9
10
11
class PostOffice < ActionMailer::Base
  def hl7_sample(recipient='thing@mustmodify.test')
    mail(
      :to      => recipient,
      :from    => 'jw@mustmodify.com',
      :subject => 'hl7 test',
      :content_transfer_encoding => 'application/edi-hl7') do |format|
        format.hl7  { render :text => File.read(Rails.root + 'test/files/tiny.hl7') }
    end
  end
end

multipart/alternative

1
2
3
4
5
6
7
8
9
10
11
12
class PostOffice < ActionMailer::Base
  def hl7_sample(recipient='default@some.test')
    mail(
      :to      => recipient,
      :from    => 'jw@mustmodify.com',
      :subject => 'hl7 test') do |format|

        format.text { render :text => 'This document contains privileged medical information. If you are not the intended recipient, please discard it. Consuming this information could result in immediate death.' }
        format.hl7  { render :text => File.read(Rails.root + 'test/files/tiny.hl7') }
    end
  end
end

multipart/mixed with 7bit attachment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class PostOffice < ActionMailer::Base
  def hl7_sample(recipient='default@some.test')
    attachments['file.hl7'] =
    {
      :data => File.read(Rails.root + 'test/files/tiny.hl7'),
      :mime_type => 'application/edi-hl7',
      :content_transfer_encoding => '7bit'
    }

    mail(
      :to      => recipient,
      :from    => 'jw@mustmodify.com',
      :subject => 'hl7 test') do |format|

        format.text { render :text => 'This document contains privileged medical information. If you are not the intended recipient, please discard it. Consuming this information could result in immediate death.' }
    end
  end
end
  • Created October 02, 2012
  • Updated October 02, 2012

HL7 via SMTP

One of my projects involves transferring medical data from a laboratory to a physician. There’s a fantastic open(ish) standard for medical data called HL7 which describes a sort of flat-file schema for data. Another standard, LOINC, provides a standardized set of identifiers for medical data.

My technical counterpart at the lab suggested we use one of the ways they normally transmit these files.

  • They could SCP / SFTP the files to us.
  • We could SCP / SFTP the files from them.
  • For windows clients, they could provide an application that would copy the files directly to a folder of your choosing. Since we’re on Ubuntu, this was not an option for us.

During our discussion, I mentioned some other options:

My understanding of HIPAA (<insert long disclaimer of responsibility />) apparently requires that I keep pristine logs of what data is transmitted and received, by whom, when, etc… from that perspective, SCP/SFTP makes me nervous. It isn’t really “discreet messaging.” I told my client this would be like us setting up a shared folder and opening up documents in notepad instead of using email.

I later found out that they recently developed a SOAP interface… surprisingly, they were hesitant to set that up with other clients.

A few days later, the lab wrote me to say they had heard my hesitation and thought they could easily accommodate an SMTP+TLS interface. ( TLS is the new SSL )

The SMTP Interface Overview

  • Step 1: vendor transmits the HL7 files via SMTP+TLS to our email provider.
  • Step 2: our app uses IMAP+TLS (or whatever) to retrieve messages from server.
  • Step 3: our app replies with an acknowledgement.
  • Step 4: Journalize, parse, and process the message

Alternatively, if we couldn’t get a HIPAA compliant business partner agreement with RackSpace, we would have set up an IMAP or SMTP endpoint on our server. This would be more overhead, and is outside my area of expertise, so I’m thrilled to say that RackSpace did provide us with the necessary documents.

HIPAA requirements.

Looking through a document provided by my client, I see that I am required to

  • log PHI received from, created by, or received on behalf of my client
  • comply with “HIPAA Standards for Security”:
    • maintain reasonable and appropriate administrative, technical, and physical safeguards for protecting e-PHI
    • ensure the confidentiality, integrity and availability of e-PHI I create, receive, maintain or transmit
    • Identify and protect against reasonable threats

Reasonable Security Measures

After reading Merging secure data elements to EDI messages I have some concerns. I will need to check with my client to see whether they have a Business Associate Agreement with the email vendor. If so, then the chain-of-HL7-hush-hushness is preserved.

If not, then I must look for a different way to meet the criteria above. Some options:

  • Set up a TLS/SMTP endpoint on the server just for receiving those messages: I bet there’s a rubygem for that. Still, it would be way easier, infrastructure-wise, to have our existing email vendor handle it, so that’s the preferred option.
  • Get that agreement: If they don’t already have one, it may be that the vendor already decided it was too odious.
  • Use a ‘Security Envelope’: this is actually a great solution whether or not we have an agreement.
    • verifiable data
    • would work across platforms and messaging protocols ( LLP, SMTP, XMPP, etc. )
    • might not be necessary if we have the signed agreement
    • have to investigate how much work would be involved
    • See “Security Envelope Notes” below.

Action Items

Translating all that into relevant actionable items:

  • Find out whether the email vendor is a trusted HIPAA partner … DONE, they are.
  • Ensure that emails are retrieved using encryption … DONE, they are
  • Log the receipt and transmission of HL7 documents … DONE
  • Look into using checsums or some other signature (although I think what they mean by integrity is actually… “make sure people can’t go to your interface and destroy / irrevocably mess up all your data”)

Security Envelope notes

This seems to mean “Take a message, sign it with your private key, encrypt it with my public key, and send it to me.” Intrinsic are security, authenticity and integrity.

From Merging secure data elements to EDI messages

the security services providing HL7 communication security MUST be placed on the transport layer or application layer of each principal. Additional protection MAY be applied using security services provided by protocols located at the lower layers (placed on the network layer or data link layer).

For interoperability reasons, only standard documents available as ISO Standards, IETF/IESG Internet Standards (RFCs), IETF Internet Drafts (IDs), NIST publications (NIST FIPS PUB) or similar MUST be used for the security enhancement of existing protocols (standards conformance).

Communication Protocol Security Requirements
* principal authentication (applications and systems)
* data origin authentication
* confidentiality
* integrity
* non repudiation of origin
* non-repudiation of receipt

I recommend this document. Their use of all-caps to emphasize MUST and MAY makes me think they’re a bit full of themselves. You MAY want to think of what they are saying as what you MUST do, or you MAY think of it as OPTIONAL. I don’t doubt their dedication, just their standards for determining what MUST be done. :)

References

  • Created September 24, 2012
  • Updated October 04, 2012

Encoding YAML with Ruby 1.9 ( Psych gem )

Psych is apparently new for 1.9… I can see that there have been some improvements over the previous to_yaml situation, but there are still alot of unanswered questions. I’ve been digging through the code, and it has given me a greater appreciation of what it is like for other people to dig through my meta-code. I vow, for the next 10 minutes, not to write any meta-code without absurd amounts of documentation.

Using Psych

here are the things I’ve been able to figure out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
    require 'valuable'

    class Person < Valuable
      has_value :name

      def encode_with coder
        coder['name'] = self.name
      end
    end

    >> Person.new(:name => 'Johnathon').to_yaml

    --- !ruby/object:Person
    name: Johnathon

    # Note: I'm totally cheating. I actually did puts Person.new(...)

Getting rid of the object declaration

for my app, humans may actually use the output, so we want it to be as techno-babble free as possible. I want to remove the !ruby/object:Person bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
    class Person < Valuable
      has_value :name

      def encode_with coder
        coder.tag = nil
        coder['name'] = self.name
      end
    end

    >> Person.new(:name => 'Johnathon').to_yaml

    --
    name: Johnathon

much cleaner. Let’s see what it looks like in an array:

1
2
3
4
5
6
7
    >> [Person.new(:name => 'Bill'), Person.new(:name => 'Ted')].to_yaml

    --
    - name: Bill
    - name: Ted

abstracting to the generic case:

1
2
3
4
5
6
7
8
9
10
11
12
13
    class Person < Valuable
      has_value :first
      has_value :middle
      has_value :last

      def encode_with coder
        coder.tag = nil

        self.attributes.each do |name, value|
          coder[name.to_s] = value
        end
      end
    end

note that, unlike Rails, Valuable (generally) won’t return attributes with nil values, so it’ll still just show first names. If you want all attributes, whether or not they are blank, you can use Person.attributes instead.

WIth a Rails model, this will encode just your attributes, not associations:

1
2
3
4
5
6
7
8
9
10
11
12
13
    module GenericActiveRecordEncoder
      def encode_with coder
        coder.tag = nil

        my_atts = self.attributes.keys - COMMON_ATTS # to exclude things like created_at
        my_methods = [:imperial_display_units, :metric_display_units]

        my_atts.each do |att|
          value = read_attribute(att)
          coder[att.to_s] = value unless value.blank?
        end
      end
    end

If you to include the associations as they would ordinarily be encoded, append something like this:

1
2
3
4
5
    associations = [:memberships, :mom]

    associations.each do |assoc|
      coder[:assoc] = self.send(assoc)
    end

I find that I typically want an association to appear differently than the model would typically be encoded. I really loved the “named encodings” feature from the long-neglected serialize_with_options gem… but anyway:

1
2
3
    coder[:father] = father.name
    coder[:occupation] = job.occupation.to_s
    coder[:can_vote] = (age >= voting_age) ? 'yes' : 'no'

On further use, it seems like the abstraction doesn’t work well for models of any complexity. There are just too many exceptions, caviats, etc. My new pattern is more verbose, out of necessity. The last block feels like something Psych should be handling as an option.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
      def encode_with coder
        coder.tag = nil

        data =
        {
          'gender' => self.gender,
          'starting_age' => self.starting_age,
          'ending_age' => self.ending_age,
          'low' => self.low,
          'high' => self.high,
          'complex' => self.complex ? 'yes' : 'no',
          'phase' => self.phase,
          'start_date' => self.start_date.to_s,
          'end_date' => self.end_date.to_s
        }

        data.each do |n, v|
          case v
          when Array
            coder[n.to_s] = v unless v.empty?
          else
            coder[n.to_s] = v unless v.blank?
          end
        end
      end

Questions

How do I specify whether I want lists in the Flow format or the default format?

1
2
3
4
5
6
7
8
9
10
11
--
letters: ['a', 'b', 'c']

vs 

--
letters:
- a
- b
- c

Can I get Psych to automatically encode true as yes, false as no ?

1
2
3
4
5
Person.new(:last => 'Bieber', :can_vote => false).to_yaml

--
name: Bieber
can_vote: no

Can I get Psych to automatically ignore blank/empty attributes?

1
2
3
4
5
6
7
8
9
10
11
Person.new(:first => 'Johnathon', :languages => ['Ruby', 'C#', 'Perl', 'PHP'], :empty_collection => [] ).to_yaml

--
first: Johnathon
languages:
- Ruby
- C#
- Perl
- PHP

# Note absence of collection 'empty collection'.
  • Created September 14, 2012
  • Updated September 14, 2012