Skip to content

OPERATING.md

Latest commit

 

History

History
314 lines (228 loc) · 15 KB

OPERATING.md

File metadata and controls

314 lines (228 loc) · 15 KB
Raw

Operating

This document contains requirements and tips for operating the adaptor in a production environment.

AMQP Message Broker Requirements

  • The broker must be configured with a limited number of retries and deadletter queues
  • It is the responsibility of the GP supplier to configure adequate monitoring against the deadletter queues that allows ALL undeliverable messages to be investigated fully
  • The broker must use persistent queues to avoid loss of data
  • The GP System must persist the relevant transaction data before acknowledging the message from the queue to avoid loss of data

Using AmazonMQ

  • A persistent broker (not in-memory) must be used to avoid data loss.
  • A configuration profile that includes settings for retry and deadlettering must be applied.
  • AmazonMQ uses the scheme amqp+ssl:// but this MUST be changed to amqps:// when configuring this adaptor.

Using Azure Service Bus

  • The ASB must use MaxDeliveryCount and dead-lettering
  • Azure Service Bus may require some parameters as part of the URL configuration. For example: NHAIS_AMQP_BROKERS=amqps://<NAME>.servicebus.windows.net/;SharedAccessKeyName=<KEY NAME>;SharedAccessKey=<KEY VALUE>

MongoDB Database Requirements

  • This GP Links Adaptor and the PCRM/NHAIS system communications are synchronised through a sequence number mechanism
  • The MongoDB database preserves this synchronisation
  • Deleting the MongoDB database and/or its collections will break the link with the PCRM/NHAIS system
  • The database should be used to monitor for any missing transactions. See REPORTING.md for details

Amazon DocumentDB Tips

In the "Connectivity & Security" tab of the cluster a URI is provided to "Connect to this cluster with an application". Replace : with the actual mongo username and password to be used by the application. The value of NHAIS_MONGO_URI should be set to this value. Since the URI string contains credentials we recommend managing the entire value as a secured secret.

The user must have the readWrite role or a custom role with specific privileges.

Azure Cosmos DB Tips

Follow Azure documentation on Cosmos DB's API for MongoDB.

Database Collections

The default database name is nhais but this can be changed through an environment variable. Each deployment of the adaptor MUST have its own database, but multiple databases could be hosted on a single cluster. The collection names used by the adaptor cannot be changed.

Outbound Sequence IDs

Tracks the sequence numbers used to "link" a GP and to HA using EDIFACT messaging. See Linking a GP Practice to an NHAIS system section for more information.

Collection Name: outboundSequenceId

Properties:

  • _id the key for the sequence in the format -- where:
    • <type> is one of TN (transaction number), SIS (send interchange sequence), SMS (send message sequence)
    • <sender> is the GP Trading Partner Code
    • <recipient> is the HA Trading Partner Code
  • sequenceNumber is the most recently generated number for the sequence

Example:

{
    "_id" : "SIS-TES5-XX11",
    "sequenceNumber" : 2
}

Outbound State

Records the metadata of outbound messages sent through the adaptor. When an inbound RECEP is received for a previously sent outbound message then the adaptor updates the corresponding document with an embedded/nested document named recep.

Collection Name: outboundState

Properties:

  • _id: ObjectId generated by the database system. Not used by the adaptor.
  • workflowId: Refers to the MESH workflow id used to send this message. One of: REGISTRATION, RECEP
  • operationId: The generated operationId returned in the response headers for this transaction
  • tn: The EDIFACT transaction number used for this transaction
  • translationTimestamp: The timestamp when the adaptor translated the transaction into EDIFACT
  • transactionType: Three-letter abbreviation identifying the type of transaction
  • intSeq: The EDIFACT send interchange sequence used for this transaction
  • msgSeq: The EDIFACT send message sequence used for this transaction
  • sndr: The sender's GP Trading Partner code
  • recip: The recipient's HA Trading Partner code
  • _class: Automatically generated by the Spring framework
  • conversationId: The conversation id of the request that initiated this transaction
  • recep: If present, an embedded/nested document with information about the received RECEP
    • code: Should always be SUCCESS as NHAIS systems no longer send negative RECEPs
    • translationTimestamp: The timestamp, as reported in the inbound EDIFACT interchange, when the NHAIS system produced the RECEP
    • processedTimestamp: The timestamp when the adaptor processed the RECEP message
    • interchangeSequence: The Receive Interchange Sequence (RIS) of the interchange in which the RECEP was received

Example:

{
    "_id" : ObjectId("5f71e6480a998d3f32224979"),
    "workflowId" : "REGISTRATION",
    "operationId" : "c35b1432682a04ca77b287b11f240fbd8c46f22fe589b513a84e307b57dcb820",
    "tn" : NumberLong(1),
    "translationTimestamp" : ISODate("2020-09-28T13:33:59.000Z"),
    "transactionType" : "ACG",
    "intSeq" : NumberLong(1),
    "msgSeq" : NumberLong(1),
    "sndr" : "TES5",
    "recip" : "XX11",
    "_class" : "uk.nhs.digital.nhsconnect.nhais.outbound.state.OutboundState",
    "conversationId" : "8DF7420F4CB0416DB894C2170ED408B2",
    "recep" : {
        "code" : "SUCCESS",
        "translationTimestamp" : ISODate("2020-06-20T13:00:00.000Z"),
        "processedTimestamp" : ISODate("2020-09-28T13:48:40.000Z"),
        "interchangeSequence" : NumberLong(164)
    }
}

Inbound State

Records the metadata of inbound transactions received by this adaptor.

Collection Name: inboundState

Properties:

  • _id: ObjectId generated by the database system. Not used by the adaptor.
  • workflowId: Refers to the MESH workflow id used to send this message. One of: REGISTRATION, RECEP
  • operationId: The generated operationId returned in the response headers for this transaction
  • intSeq: The EDIFACT receive interchange sequence used for this transaction
  • msgSeq: The EDIFACT receive message sequence used for this transaction
  • sndr: The sender's HA Trading Partner code
  • recip: The recipient's GP Trading Partner code
  • tn: The EDIFACT transaction number used for this transaction
  • translationTimestamp: The timestamp, as reported in the inbound EDIFACT interchange, when the PCRM/NHAIS system produced the interchange
  • transactionType: Three-letter abbreviation identifying the type of transaction
  • processedTimestamp: The timestamp when the adaptor processed the transaction
  • conversationId: The conversation id of the request that initiated this transaction
  • _class: Automatically generated by the Spring framework

Example:

{
    "_id" : ObjectId("5f7efdc78fd23123462794fa"),
    "workflowId" : "REGISTRATION",
    "operationId" : "93ed8f29dff97c422a228970afb7f7bca6031b4d8a9f2e64415ec2afd32fae39",
    "intSeq" : NumberLong(10000003),
    "msgSeq" : NumberLong(10000009),
    "sndr" : "XX11",
    "recip" : "TES5",
    "tn" : NumberLong(113),
    "translationTimestamp" : ISODate("2092-01-25T12:35:00.000Z"),
    "transactionType" : "APF",
    "processedTimestamp" : ISODate("2020-10-08T11:53:42.944Z"),
    "conversationId" : "4311128167C84E3795BD49A7F7E51E94",
    "_class" : "uk.nhs.digital.nhsconnect.nhais.inbound.state.InboundState"
}

Scheduler Timestamp

A collection used to coordinate running the MESH polling cycle across multiple instances of the adaptor. README describes this mechanism in more detail.

Collection Name: schedulerTimestamp

Properties:

  • _id: ObjectId generated by the database system. Not used by the adaptor.
  • schedulerType: Always "meshTimestamp"
  • updateTimestamp: The time the MESH polling cycle last ran
  • _class: Automatically generated by the Spring framework

Example:

{
    "_id" : ObjectId("5f638befb3bd281ba51000a7"),
    "schedulerType" : "meshTimestamp",
    "updateTimestamp" : ISODate("2020-10-08T11:55:43.520Z"),
    "_class" : "uk.nhs.digital.nhsconnect.nhais.mesh.scheduler.SchedulerTimestamp"
}

Time-to-live Indexes

The adaptor creates TTL (time to live) indexes on the outboundState and inboundState collections to automatically remove old documents. The variable NHAIS_MONGO_TTL described above controls the duration. There are differences between how TTL indexes work between MongoDb and Azure Cosmos DB. When using Cosmos the NHAIS_COSMOS_DB_ENABLED flag must be true.

TTL Indexes in MongoDB / AWS DocumentDB

The property translationTimestamp is indexed. For outbound, this is the timestamp when the adaptor translates FHIR into EDIFACT and is the timestamp enclosed in the EDIFACT interchange sent to PCRM/NHAIS. For inbound, this is the timestamp enclosed within the EDIFACT interchange received from PCRM/NHAIS.

TTL Indexes in Azure Cosmos DB

TTL Index in Cosmos are limited in that only a specific _ts property may have this index. The _ts property "is a system generated property (that) specifies the last updated timestamp of the resource". (Reference)

For outbound, each document is "last updated" when the adaptor processes the inbound RECEP for that transaction. For inbound each document is "last updated" after publishing the FHIR message to the inbound GP System message queue.

The impact is that documents in Cosmos may live slightly longer than those stored in MongoDB.

AWS DocumentDB TLS configuration

AWS DocumentDB uses a private CA certificate and therefore requires a custom keystore to manage the CA certificates effectively.

To use TLS, it has to be enabled in the DocumentDB instance with the Mongo connection string containing the tls=true. For more information on TLS configuration in MongoDB see TLS options for Mongo connection string.

SSH tunneling might require adding the connection string option: tlsAllowInvalidHostnames=true (as CA would try to resolve localhost as hostname). This option should only be used for local tests as this might create a vulnerability.

Use of DocumentDB TLS requires java trust store to be provided manually. Instructions on how to create trust store can be found here: Connect to Document DB programmatically

To configure custom trust store with AWS CA certificates set the NHAIS_SSL_TRUST_STORE_URL with a runtime-accessible S3 URL with the JKS file (Example: s3://my-bucket/my-trust-store.jks). The AWS resource running the adaptor must have sufficient permission to read this s3 object. Additionally, set the NHAIS_SSL_TRUST_STORE_PASSWORD to trust store password

MESH Requirements

Note: The "Development" section of the README refers to a fake-mesh component. fake-mesh is not part of this adaptor API and should only be used to assist local development.

NHS England manage access to MESH, allocate mailboxes, and provide connection details / credentials.

Management Endpoints

Spring Boot Actuator provides three management endpoints:

  • /healthcheck
  • /metrics
  • /info

Monitoring GP Links messaging state

There are two MongoDB collections recording the state of GP Links transactions:

  • outboundState records every transaction sent by the adaptor (GP->HA)
  • inboundState records every transaction received by the adaptor (HA->GP)

These collections should be monitored to identify any transactions that may be missing.

See REPORTING.md for details about how to run these reports.

Logging and Tracing

The adaptor follows usual conventions for containerised solutions and logs to the console. The chosen container orchestration solution must be configured to store and aggregate the adaptor's logs.

Conversation Id

The adaptor uses a conversation id to make tracing individual requests easier. All log lines will contain the text ConversationId=<Conversation Id>. For outbound requests the value of <Conversation Id> is either declared in the request headers or randomly generated. The OpenAPI Specification describes the ConversationId header.

For inbound transactions the adaptor generates a conversation id for every message (interchange) downloaded from MESH. An interchange may contain multiple transactions, and the adaptor uses same conversation id for every transaction within an interchange.

Some log lines are uncorrelated to a specific transaction. These logs have an empty <Conversation Id> value.

Documents in the outboundState and inboundState collections also contain the conversation id.

Messages published to the adaptor's three AMQP message queues include a ConversationId header.

Linking a GP Practice to an NHAIS system

This GP Links Adaptor and the PCRM/NHAIS system communications are synchronised through a sequence number mechanism. Linking a GP Practice to an PCRM system which have never previously exchanged messages requires no additional setup for synchronisation. All the sequences begin at 1, and the adaptor will start them automatically.

In the case that a new market entrant GP System takes over from an incumbent system the new system must pick up the sequences where the incumbent left off. For every GP/PCRM link established, the incumbent supplier or PCRM operator must advise the following:

  • Most recently used Send Interchange Sequence (SIS) number, GP -> HA
  • Most recently used Send Message Sequence (SMS) number, GP -> HA
  • Most recently used Transaction Number (TN), GP -> HA

For each GP/PCRM pair the following documents must be inserted into the outboundSequenceId collection of the adaptor's database. The angle-bracketed values must be replaced (including the brackets) with the relevant data items. The _id property should have the type String, and the sequenceNumber property should have the type int32. Any existing documents with the same _id must be replaced.

{
    _id: 'SIS-<GP Link (Trading Partner) Code>-<HA Link (Trading Partner) Code>',
    sequenceNumber: <Send Interchange Sequence (SIS) number>
}

{
    _id: 'SMS-<GP Link (Trading Partner) Code>-<HA Link (Trading Partner) Code>',
    sequenceNumber: <Send Message Sequence (SMS) number>
}

{
    _id: 'TN-<GP Link (Trading Partner) Code>',
    sequenceNumber: <Transaction Number (TN)>
}