Here at GoMOOS we've stood up a java version of the OpenNode2? on Linux using Apache Tomcat and PostgreSQL.
The node itself was straight forward to get running, however, testing it has been a challenge.
We are developing a new exchange and probably will not even be exchanging data with the CDX at the beginning so our experience is somewhat non-standard.
Plugins were not quite what we expected, with each one implementing a different subset of the WSDL services. A fast way to navigate these (what plugin has what functionality) is the documentation ( I know who would have thought ).
Our major challenge right now is accepting an XML file, parsing it and inserting it into our own database. We are currently experimenting with STAX and it's event driven file parsing to create something which will handle arbitrarily large files. We investigate JAXB but have had trouble finding a path through it's many possible use cases which would give us the ability to "chunk" up the incoming file.
This is a pretty short and perhaps not to the point description of where we are. Feel free to drop me a line.
Re: cfli1688: Implementing Flows. Yes, there is a document for each of the data exchange plugins that have been released. This document describes the steps and configuration options to implement each of these exchanges.
Re: asairam: Staging table population. Typically we have a parallel process that pushes data from production tables to the staging tables on a regularly scheduled basis, for example nightly or weekly, depending on flow frequency. We tend to use the same scheduling process that the agency prefers for other such database tasks, for example using the RDBMS scheduler, an ETL processor, or a web publishing/DMZ migration process. For security reasons, it is always better to push data into these staging tables, then to pull it from the 'web zone' that the Node lives within.
Sorry this isn't really feed back on implementation.
Never the less....
We're working with OpenNode2? primarily as a Server. Our partners will be submitting data to our node using stand alone clients and programatically with Perl and perhaps Python scripts.
It would be great to have a web based Node Client available for Node 2.
A web based client would remove any software installation road blocks, updates and in general any issues related to the local machine. Also it would be great for non-Windows users.
OpenNode2? has all the client software building blocks built it. It just needs an interface that I'll bet a better Java programmer than myself could churn out without too much effort.
I am trying to implement a plug-in for the .NET OpenNode2? using ISubmitProcessor and following the source code of the UIC plug-in as a model. I wonder which tool has been used to generate the ORM annotations, or has this been done manually? I generated the classes using the .NET xsd tool from my XSD, but wonder if there is an easier way to generate the annotations as well, opposed to adding them manually in the generated class. Thank you!
We typically add the mapping attributes by hand using the Data Exchange Templates (found on the Exchange Network site) as a guide for each exchange. By default, the xsdorm library will generate database table and columns automatically for you from the xml classes. Could you tell me more about the exchange/plugin you are implementing?
Thanks Ted for answering my question. Yes, we are trying to implement an OpenNode2? plug-in to accept, store in a database and translate data from the NPSBMP schema into a simplified format usable to another tool.
Ted, I have another question. I need to send an email from my plug-in and I wonder if there is any API in the node I can use to send it. Something like SendNotificationsPriv? from NotificationManager?.cs would be useful, but this is private in that class. I need a custom email message with a custom attachment. Are there any building blocks in the node I could use for email sending? Thanks!
I don't know if you saw my previous message, I'm trying to elaborate. The requirement is to send a custom email from the submission processor to an email address that comes from the submitted xml. I want to do this the right way, if possible within the node API, like not having to define again the SMTP server and all the other configuration parameters that are already in the node's configuration file. I couldn't find anything in the node able to send an email message from the plugin using the node's configuration, other than the notification mechanism which won't help in this case. If there is no other way, I would need to create configuration parameters for the plug-in in order to configure again the SMTP server. Any idea would be higly appreciated.
I have a question on schedules, and using certain options to create a schedule. For the OWIR dataflow we have created the xml based on the published standard and would like to use a schedule to send the data to the EPA. We configured the Data Source to be a file system and pointed it to the XML. The result process is to submit result to an exchange network partner and have it going to the EPA. From the documentation it seems this would be appropriate, but we get the response "Exception: SoapException?, Message: NodeFaultMessage?" Should this configuration work or is there a better way to do that type of submittal utilizing the OpenNode??
It sounds like you are doing everything right. I would need a little more information to help diagnose the cause of the error. I don't have any experience implementing the OWIR flow and I do not know if any other ON2 users have attempted this flow, so you are likely in new territory. One thing you can do is have the schedule submit the file to your own node's endpoint as a test. If the schedule completes successfully, you may be dealing with an interoperability issue with the target endpoint. What is the target node URL? This may help in troubleshooting. Please reply to bill@windsorsolutions.com to continue this discussion. Thank you.
We have changed the Deployment.config file to test as a Production environment: changed the flags to true on "node.is.production", assured the username and email are those of the NAAS Production account.
Upon logging into the admin screen, an error shows: unable to authenticate user; though, we could still log in successfully using the development user/pass combo.
I noticed that an SSL certificate needs to be included to be able to use the server as a production environment. We attempted to use a self-signed certificate to accomplish this, but the error remained the same. Maybe self-signed are not allowable?
Does anyone have any other thoughts on what the issue might be or what we may have missed? I would appreciate any information.
Hi Erin, Please note that any changes to the Deployment.config file require that you stop and restart the Node Windows service for the changes to take effect. Please check that you have done that, and let us know if you have further issues. Thanks, Ted
Is there a simple way to configure a schedule event to write the TRI files to a directory location? Or would you need to create a new submissionprocessor to write the files?
I'm assuming you want to write files that are submitted to your node to a folder of your choice.
The Windsor plugin contains an implementor called "CopySubmitFilesToFolder?." You could use this implementor to perform this task:
1) Create a TRI Exchange on the node 2) Upload the Windsor plugin to the TRI exchange using the plugin upload page 3) Create a service under the TRI exchange and call it ''. An service with a name will be used for all submits that do not specify an operation (which are all version 1.1 submits). Select the CopySubmitFilesToFolder? as the implementor for this service and specify the folder where submitted documents should be placed. The folder must be physically accessible from the location where the node Windows service is running. 4) When a Submit comes in for the TRI exchange, the submitted document will be copied to the folder you specified in step 3 above.
Thanks Ted. I found in the documentation after I posted that you needed a submission processor which makes perfect sense. I implemented one as you described, I created one using the base of the TRIsubmissionprocessor. Changed for a destinationpath as a parameter and removed the write to a database. Thanks for the help.
The Security tab in the Admin screen mentions: "A user account can be setup to be a Reader (only receive data) or an Author (receive data and create schedules)". Where is this setting located?
Erin, the text on the Security tab is out of date. The available roles are "Endpoint User" (formerly 'Authed' in prior node versions) meaning they can access web services via the node endpoints, "Program User" can log into Node Admin and administer schedules, and "Admin" that offers access to all Admin and Endpoint functions.
We will be updating the language throughout the node in the next release to avoid this type of confusion.
I have a question in regards to the HERE - TIERII plugin. The TIER II SQL tables appear to cover the top level TIERIISubmission node, but there does not appear to be any capture for either the AlertNotification? or EmergencyAdvisory? nodes. Is this correct that these elements aren't part of the plugin?
Yes Drew, that is correct. The HERE project reused the existing TIER II XML schema but had no use for those somewhat extraneous data structures, so the plugin only support the data the HERE Client consumes.
We're experiencing issues with the WQX Submission.
As soon as we started processing submissions with extraneous attachments, we started receiving the error: "There is an error in XML document (1, 1)."
However, if we set to submit with no extra attachments, it passes through or node successfully but the report from EPA says that attachments are missing.
Has anyone else had this issue? Or anyone have suggestions to what we can do to repair this?
Again, still having issues when attempting to submit WQX data along with attachments. I am not absolutely sure on how this is all processed, but it appears to me that the code may not be recognizing there are 3 files and the last one (alphabetically?) is the XML file it should be processing. The error runs through as shown below. If anyone has suggestions, I would love to hear them!
- Deserializing document data to WQX data - Failed to deserialize WQX v2, trying WQX v1 ... - Failed to process document with id "2cebb7d7-716f-48c8-b600-f146a8c779be." EXCEPTION: Message: There is an error in XML document (1, 1). INNER EXCEPTION: Message: Data at the root level is invalid. Line 1, position 1.
WQX Delete Submission is giving me an error: "There is an error in XML document (1, 2)." The file submitted is a ZIP with XML containing the following. Does anyone have suggestions or any idea why this error is being thrown in the ON2?
Erin, I looked into this issue briefly. Using the latest version of the OpenNode2? binaries and WQX plugin, I am able to generate a valid Delete XML file. The file validated successfully using the built in Schedule Parameter "ValidateXml?" = true as reported in the activity log for execution of the GetDeleteSubmission? schedule I created.
If you would like us to look into this further, please submit a support request to Greg McNelly? at ECOS. He can authorize Windsor to expend ECOS support budget to support you in investigating this issue further. Thank you.
Thanks for the attention. This error is happening in the WQXSubmissionProcessor code when a delete request is being sent to our node to be stored in the Staging database; the process hasn't made it to the Delete schedule as of yet.
If you have any other notes, it would be much appreciated. Erin
Here at GoMOOS we've stood up a java version of the OpenNode2? on Linux using Apache Tomcat and PostgreSQL.
The node itself was straight forward to get running, however, testing it has been a challenge.
We are developing a new exchange and probably will not even be exchanging data with the CDX at the beginning so our experience is somewhat non-standard.
Plugins were not quite what we expected, with each one implementing a different subset of the WSDL services. A fast way to navigate these (what plugin has what functionality) is the documentation ( I know who would have thought ).
Our major challenge right now is accepting an XML file, parsing it and inserting it into our own database. We are currently experimenting with STAX and it's event driven file parsing to create something which will handle arbitrarily large files. We investigate JAXB but have had trouble finding a path through it's many possible use cases which would give us the ability to "chunk" up the incoming file.
This is a pretty short and perhaps not to the point description of where we are. Feel free to drop me a line.
Ian ian@gomoos.org
I am trying to understand the solution. Where, at what stage and how are you pushing the data into the staging tables of a plugin?
In the introduction page, it said the node "... includes a number of common data exchanges used by EPA ..."
Is there a document that we can follow to get these up and running?
Re: cfli1688: Implementing Flows. Yes, there is a document for each of the data exchange plugins that have been released. This document describes the steps and configuration options to implement each of these exchanges.
Re: asairam: Staging table population. Typically we have a parallel process that pushes data from production tables to the staging tables on a regularly scheduled basis, for example nightly or weekly, depending on flow frequency. We tend to use the same scheduling process that the agency prefers for other such database tasks, for example using the RDBMS scheduler, an ETL processor, or a web publishing/DMZ migration process. For security reasons, it is always better to push data into these staging tables, then to pull it from the 'web zone' that the Node lives within.
Sorry this isn't really feed back on implementation.
Never the less....
We're working with OpenNode2? primarily as a Server. Our partners will be submitting data to our node using stand alone clients and programatically with Perl and perhaps Python scripts.
It would be great to have a web based Node Client available for Node 2.
A web based client would remove any software installation road blocks, updates and in general any issues related to the local machine. Also it would be great for non-Windows users.
OpenNode2? has all the client software building blocks built it. It just needs an interface that I'll bet a better Java programmer than myself could churn out without too much effort.
Anyone else out there with a similar need?
Thanks!
Ian ian@gomoos.org
I am trying to implement a plug-in for the .NET OpenNode2? using ISubmitProcessor and following the source code of the UIC plug-in as a model. I wonder which tool has been used to generate the ORM annotations, or has this been done manually? I generated the classes using the .NET xsd tool from my XSD, but wonder if there is an easier way to generate the annotations as well, opposed to adding them manually in the generated class. Thank you!
Hi Danny,
We typically add the mapping attributes by hand using the Data Exchange Templates (found on the Exchange Network site) as a guide for each exchange. By default, the xsdorm library will generate database table and columns automatically for you from the xml classes. Could you tell me more about the exchange/plugin you are implementing?
Thanks, Ted
Thanks Ted for answering my question. Yes, we are trying to implement an OpenNode2? plug-in to accept, store in a database and translate data from the NPSBMP schema into a simplified format usable to another tool.
Ted, I have another question. I need to send an email from my plug-in and I wonder if there is any API in the node I can use to send it. Something like SendNotificationsPriv? from NotificationManager?.cs would be useful, but this is private in that class. I need a custom email message with a custom attachment. Are there any building blocks in the node I could use for email sending? Thanks!
Ted,
I don't know if you saw my previous message, I'm trying to elaborate. The requirement is to send a custom email from the submission processor to an email address that comes from the submitted xml. I want to do this the right way, if possible within the node API, like not having to define again the SMTP server and all the other configuration parameters that are already in the node's configuration file. I couldn't find anything in the node able to send an email message from the plugin using the node's configuration, other than the notification mechanism which won't help in this case. If there is no other way, I would need to create configuration parameters for the plug-in in order to configure again the SMTP server. Any idea would be higly appreciated.
Thank you, Danny
I have a question on schedules, and using certain options to create a schedule. For the OWIR dataflow we have created the xml based on the published standard and would like to use a schedule to send the data to the EPA. We configured the Data Source to be a file system and pointed it to the XML. The result process is to submit result to an exchange network partner and have it going to the EPA. From the documentation it seems this would be appropriate, but we get the response "Exception: SoapException?, Message: NodeFaultMessage?" Should this configuration work or is there a better way to do that type of submittal utilizing the OpenNode??
Thanks in advance, Drew
Drew,
It sounds like you are doing everything right. I would need a little more information to help diagnose the cause of the error. I don't have any experience implementing the OWIR flow and I do not know if any other ON2 users have attempted this flow, so you are likely in new territory. One thing you can do is have the schedule submit the file to your own node's endpoint as a test. If the schedule completes successfully, you may be dealing with an interoperability issue with the target endpoint. What is the target node URL? This may help in troubleshooting. Please reply to bill@windsorsolutions.com to continue this discussion. Thank you.
-Bill
We have changed the Deployment.config file to test as a Production environment: changed the flags to true on "node.is.production", assured the username and email are those of the NAAS Production account.
Upon logging into the admin screen, an error shows: unable to authenticate user; though, we could still log in successfully using the development user/pass combo.
I noticed that an SSL certificate needs to be included to be able to use the server as a production environment. We attempted to use a self-signed certificate to accomplish this, but the error remained the same. Maybe self-signed are not allowable?
Does anyone have any other thoughts on what the issue might be or what we may have missed? I would appreciate any information.
Thanks, Erin
Hi Erin, Please note that any changes to the Deployment.config file require that you stop and restart the Node Windows service for the changes to take effect. Please check that you have done that, and let us know if you have further issues. Thanks, Ted
Excellent, it worked! I appreciate the help.
Is there a simple way to configure a schedule event to write the TRI files to a directory location? Or would you need to create a new submissionprocessor to write the files?
Thanks! Drew Block
Hi Drew,
I'm assuming you want to write files that are submitted to your node to a folder of your choice.
The Windsor plugin contains an implementor called "CopySubmitFilesToFolder?." You could use this implementor to perform this task:
1) Create a TRI Exchange on the node 2) Upload the Windsor plugin to the TRI exchange using the plugin upload page 3) Create a service under the TRI exchange and call it ''. An service with a name will be used for all submits that do not specify an operation (which are all version 1.1 submits). Select the CopySubmitFilesToFolder? as the implementor for this service and specify the folder where submitted documents should be placed. The folder must be physically accessible from the location where the node Windows service is running. 4) When a Submit comes in for the TRI exchange, the submitted document will be copied to the folder you specified in step 3 above.
Ted.
Drew, Sorry, the wiki sytanx slightly messed up my comment above! For the service name, you should use a single asterisk character.
Ted.
Thanks Ted. I found in the documentation after I posted that you needed a submission processor which makes perfect sense. I implemented one as you described, I created one using the base of the TRIsubmissionprocessor. Changed for a destinationpath as a parameter and removed the write to a database. Thanks for the help.
The Security tab in the Admin screen mentions: "A user account can be setup to be a Reader (only receive data) or an Author (receive data and create schedules)". Where is this setting located?
Erin, the text on the Security tab is out of date. The available roles are "Endpoint User" (formerly 'Authed' in prior node versions) meaning they can access web services via the node endpoints, "Program User" can log into Node Admin and administer schedules, and "Admin" that offers access to all Admin and Endpoint functions.
We will be updating the language throughout the node in the next release to avoid this type of confusion.
-Bill Rensmith
I have a question in regards to the HERE - TIERII plugin. The TIER II SQL tables appear to cover the top level TIERIISubmission node, but there does not appear to be any capture for either the AlertNotification? or EmergencyAdvisory? nodes. Is this correct that these elements aren't part of the plugin?
Thanks, Drew Block
Yes Drew, that is correct. The HERE project reused the existing TIER II XML schema but had no use for those somewhat extraneous data structures, so the plugin only support the data the HERE Client consumes.
We're experiencing issues with the WQX Submission.
As soon as we started processing submissions with extraneous attachments, we started receiving the error: "There is an error in XML document (1, 1)."
However, if we set to submit with no extra attachments, it passes through or node successfully but the report from EPA says that attachments are missing.
Has anyone else had this issue? Or anyone have suggestions to what we can do to repair this?
Again, still having issues when attempting to submit WQX data along with attachments. I am not absolutely sure on how this is all processed, but it appears to me that the code may not be recognizing there are 3 files and the last one (alphabetically?) is the XML file it should be processing. The error runs through as shown below. If anyone has suggestions, I would love to hear them!
- Deserializing document data to WQX data - Failed to deserialize WQX v2, trying WQX v1 ... - Failed to process document with id "2cebb7d7-716f-48c8-b600-f146a8c779be." EXCEPTION: Message: There is an error in XML document (1, 1). INNER EXCEPTION: Message: Data at the root level is invalid. Line 1, position 1.
Update to my Dec 29, 2011 post: WQX Plugin doesn't support attachments. The attachment is what was causing the issue.
Does anyone know where can I get TRI5.0 plugin ? Thanks, ZEB
WQX Delete Submission is giving me an error: "There is an error in XML document (1, 2)." The file submitted is a ZIP with XML containing the following. Does anyone have suggestions or any idea why this error is being thrown in the ON2?
<Document xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.exchangenetwork.net/schema/v1.0/ExchangeNetworkDocument.xsd" Id="IDEM_WQX-Delete_20120118_290">
Erin, I looked into this issue briefly. Using the latest version of the OpenNode2? binaries and WQX plugin, I am able to generate a valid Delete XML file. The file validated successfully using the built in Schedule Parameter "ValidateXml?" = true as reported in the activity log for execution of the GetDeleteSubmission? schedule I created.
If you would like us to look into this further, please submit a support request to Greg McNelly? at ECOS. He can authorize Windsor to expend ECOS support budget to support you in investigating this issue further. Thank you.
Thanks for the attention. This error is happening in the WQXSubmissionProcessor code when a delete request is being sent to our node to be stored in the Staging database; the process hasn't made it to the Delete schedule as of yet.
If you have any other notes, it would be much appreciated. Erin