E2EDM User’s Manual

E2EDM User’s Manual

Introduction
The Architecture of the E2EDM Technology prototype
Installing, configuring and running E2EDM Data Provider
Installing, configuring and running E2EDM Integration Server
E2EDM user interface guide

Introduction

This document is considered as manual for the E2EDM technology users. It includes sections about E2EDM architecture, E2EDM software installation and running and user guide for using this software.

The Architecture of the E2EDM Technology prototype

The E2EDM technology architecture is represented by three levels of components:

Data source level containing local data systems wrapped by Data Providers. It provides request/response processing services, links with a local data system and resource production Data communication and integration level - E2EDM Integration Server (mediator). It provides management, request navigation, aggregation of information resources from data sources, and makes aggregated data available to client client level containing a Web-browser to use the end-user service of the E2EDM Integration Server or external applications that interact with E2EDM Integration Server to obtain data and products from the distributed data source

E2EDM Integration Server provides the following services (table 1):

Table1. Integration Server structure and functionality

Component	Component functions
Integration Server Management Service	Basic service responsible for the server management, external calls handling.
Metadata service	Management of service metadata, system dictionaries and tables of codes.
Data Access Service	Management of provider’s registry, provider’s availability check, request building according to transport protocol.
Transport Service	Transmission of responses over HTTP to specific Data Provider or in a broadcast mode, reception of responses and transport files.
Navigation Service	Users interface construction

E2EDM Data Provider provides the following services:

Table2. Data Provider structure and functionality

Component	Component functions
Data Provider Management Service	Basic service. Reception and decoding of requests from Integration Server, sending of responses, errors and exceptions handling.
Database Access Service	DiGIR Provider. Provides access to the local databases.
File Access Service	Access to the file data sources.
Data format Service	Building transport data files in NetCDF format.

Installing, configuring and running E2EDM Data Provider

2.1. Installation and configuration.

Data Provider installation consists of two parts: installation of Data Provider Management System (DPMS) and installation if Database Access Service (DiGIR-system).

2.1.1. Data Provider Management System.

2.1.1.1. Setup.

Download and install the latest version of Java 2. You must have at least version 1.4 of the JDK. At the time of writing, the version used in development and testing was Java 2 SDK 1.4.2.

(Visit http://www. java. /downloads).

Follow the instructions available for installing the downloaded package. Be sure to do the following:

Set the JAVA_HOME environment variable to the location of your Java 2 install (e. g. C:\j2sdk1.4.x). This should be done in a permanent fashion.

Modify your PATH environment variable to include the Java 2 bin directory (e. g. C:\j2sdk1.4.x\bin). Again, this should be done in a permanent fashion.

Download and install at least version 4.1.3 of Jakarta Tomcat. Do not install a version earlier than 4.x or you will definitely have compatibility issues. At the time of writing, the versions used in development and testing include 4.1.3 and 4.1.4. The DPMS has undergone additional development using and is deemed to be supported by Tomcat 4.0.x, 4.1.x, Tomcat 5.0.x. Tomcat 5.5 is Apache's current production quality release

Follow the instructions available for installing the downloaded package. Be sure to do the following:

Set the CATALINA_HOME environment variable to the location of the Tomcat install (e. g. C:\Tomcat_4.1). This should be done in a permanent fashion. Note: If there are any space symbols in the Tomcat installation path, replace space by something like "_", because of the path problems in some cases.

The Tomcat 4.1 Servlet/JSP Container

====================================

This subproject contains a server that conforms to the Servlet 2.3 and

JSP 1.2 specifications from Java Software. It includes the following contents:

LICENSE Apache Software License for this release

README. txt This document

RELEASE-NOTES-*.txt Release Notes for this (and previous) releases

of Tomcat 4.1

RUNNING. txt Instructions for installing Tomcat, as well as

starting and stopping the server

bin/ Binary executables and scripts

common/ Classes available to both Catalina internal

classes and web applications:

classes/ Unpacked common classes

lib/ Common classes in JAR files

conf/ Configuration files

logs/ Destination directory for log files

server/ Internal Catalina classes and their dependencies

classes/ Unpacked classes (internal only)

lib/ Classes packed in JAR files (internal only)

shared/ Classes shared by all web applications

classes/ Unpacked shared classes

lib/ Shared classes in JAR files

webapps/ Base directory containing web applications

included with Tomcat 4.1

work/ Scratch directory used by Tomcat for holding

temporary files and directories

temp/ Directory used by JVM for temporary files

(java. io. tmpdir)

You can find related documents at the Jakarta Tomcat web site http://jakarta. apache. org/tomcat/tomcat-4.1-doc/index. html.

Copy dpms folder from the INSTALLATION_PACK/DATA_PROVIDER directory to %CATALINA_HOME%\webapps. This is the location of all web-applications in the Tomcat.

The directory structure for DPMS is:

logs Destination directory for DPMS log files

ncResults Directory used by DPMS for transport files in NetCDF-structure

WEB-INF Base directory, containing DMPMS core modules and configuration files

classes/ This directory contains DPMS core Java class files (and associated resources) required for application

lib/ External libraries, used in DPMS.

2.1.1.2.Configuration.

Configuration of DPMS involves editing of the Tomcat settings, editing DPMS configuration files (code-to-code mapping, codes for countries, organizations, geography, platforms, units), depending from the code standards used in the local data source, and adding the resources by creating an md. xml configuration file.

Tomcat settings:

The full document for server configuration you can find at http://jakarta. apache. org/tomcat/tomcat-4.1-doc/config/index. html.

If you had not configured the running port for the Tomcat during the installation, the default port will be 8080.

IMPORTANT NOTE: You need to check, that 8080 port is available from the external network! This is important for the connection to the Integration Server. If this port is blocked by firewall you can set port 10080 as alternative! We strongly recommend using port 10080 for DPMS. The port address can be changed in %CATALINA_HOME%\conf\server. xml file. Find the following structure in server. xml file:

<Connector className="org. apache. coyote. tomcat4.CoyoteConnector"

port="8080"

minProcessors="5" maxProcessors="75"

enableLookups="true" redirectPort="8443"

acceptCount="100" debug="0" connectionTimeout="20000"

useURIValidationHack="false" disableUploadTimeout="true" />

Change the port="8080" to port="10080". Now the DPMS will run at port 10080.

For security issues make sure that you have configured the admin password for the Tomcat server.

If not, see the full documentation for the Tomcat server configuration to set the password.

DPMS settings:

a. Core configuration.

Open %CATALINA_HOME%\webapps\dpms\WEB-INF\web. xml configuration file.

Find the records like

<init-param>

<param-name>E2ECodesMD_path</param-name>

<param-value>c:/Tomcat_4.1/webapps/dpms/</param-value>

</init-param>

Replace the path in <param-value>c:/Tomcat_4.1/webapps/dpms/</param-value> tag by actual installation path for Tomcat, e. g. c:/Tomcat_4.1/webapps/dpms/.

IMPORTANT: After step 2. Data Access Service installation and configuration please find in this file <init-param>

<param-name>digirConfigPath</param-name>

<param-value>c:/www/digir/config/</param-value>

</init-param>

and change the path to DiGIR configuration directory in case if it is different!!!

IMPORTANT NOTE: DO NOT DELETE THE FILES CONFIGURED IN THIS DOCUMENT TO ANY OTHER LOCATION!

IF YOU ARE MOVING THE CONFIG FILES TO ANOTHER LOCATION MAKE SURE TO CHANGE THE ACTUAL LOCATION ATFER THIS ACTION.

DO NOT MOVE CONFIGURATION FILES IN THE ROOT OF DPMS WEB-APPLICATION TO ANY OTHER LOCATION! THESE FILES ARE USER BY INTEGRATION SERVER FOR ACTUALIZING THE CONTENT OF YOUR DATA PROVIDER SYSTEM.

Open %CATALINA_HOME%\webapps\dpms\WEB-INF\config. xml

Find the record <dispatcher>http://localhost:10080/dpms/PortalServlet</dispatcher> in the file. Set the domain name and port for the DPMS core.

IMPORTANT NOTE: DO NOT LEAVE localhost AS THE DOMAIN NAME.

You can configure such parameters as:

-maxConnections : maximum number of the connections to the DPMS. Depends of the processing power and performance of the computer where DPMS is located.

-connectionWaitTime: connection waiting time in seconds

-connectionEstablishmentTimeout: timeout for connection establishment in seconds

-connectionResponseTimeout: timeout for server response

-connectionRetries : number of connection attempts

But we strongly recommend you to leave these values as they are.

b. Codes configuration.

This step is not mandatory during installation!

All the codes in E2EDM project is stored in XML files with the fixed structure! The schema for all the structures is available at http://data. *****/digir/schema/PP3XMLSchema. xsd.

Codes configuration files are located at %CATALINA_HOME%\webapps\dpms directory.

For more details see E2EDM technical specification document at the project web site (http://data. *****/e2edm).

List of code files:

country_codesMD. xml - country codes (IOC standard)

geography_codesMD. xml - geography area codes (IHB standard)

unit_codesMD. xml - unit codes (CDI standard)

organization_codesMD. xml - organization codes (CDI standard)

platform_codesMD. xml - platform type codes (CDI standard)

resources_codes. xml - configuration file, containing relations between resources, configured in the DPMS on the local point and the codes used in these resources

ElementsMD. xml - list of concept elements from E2EDM concept XML Schema

Detailed documentation for code standards and support will be provided in the project technical documentation.

c. Resources description.

All resource descriptions provided by the DPMS are stored in the %CATALINA_HOME%\webapps\dpms\WEB-INF\md. xml file. This file contains full description of each resource in XML structure. For more details about the complete structure see technical documentation.

Distributable version of the DPMS includes example of the resource description. Each description must be adequate to one of the Access Services: Database Access Service (actually, DIGIR) and File Access Service (unavailable for this distribution).

2.1.2. Database Access Service

Data Access Service (DAS) is responsible for the data extraction from the various databases, related to the resources. Data Access Service is based on the DiGIR. We are not recommend you to set up the DiGIR that was downloaded from the DiGIR project site, we have fixed a lot of bugs in the DiGIR native scripts.

2.1.2.1. Setup.

Download and install Apache HTTP Server 1.3.x. Later versions not been tested yet, so to avoid problems during configuration and running use 1.3.28 version. Detailed instructions how to install Apache are available at http://httpd. apache. org.

Download and install PHP interpreter for your system. Detailed instructions how to install PHP are available at http://www. /manual/en/install. php. At the time of writing, the versions used in development and testing include 4.3.x. The DAS has undergone additional development using and it is deemed to be supported by PHP 5.

IMPORTANT NOTE: If you using the same machine for DPMS core and Database Access Service make sure

to be running Apache on the port different from the web-server, used by DPMS. We recommend you use port 8080. The reason is that DAS doesn't require external access from the Internet, and port 8080 is blocked by most of the providers. This is good security issue for using this port for DAS.

Once you have installed and basically configured Apache and PHP, please, follow these instructions:

Copy the digir folder from the INSTALATION_PACK/DATA_PROVIDER/

DatabaseAccessService directory to the document root for the Apache.

Create directory named “tmp” in the root of the C drive! This folder is used for the user sessions in DiGIR.

IMPORTANT: We strongly recommend you set document root in Apache on external folder (e. g. c:\www). After setting this option please go to step 1.1.2 and make sure that document root for the DiGIR is right or change the path to actual.

DiGIR manual is available at http:///prov/prov_manual. html.

2.1.2.2. Configuration.

To complete the DiGIR configuration you need to:

a.  Edit the Apache HTTP Server http. conf file.

The primary configuration file for Apache is httpd. conf located in APACHE_INSTALL_DIR\conf which on windows might be c:\program files\Apache Group\Apache2\conf or /etc/httpd/conf on Linux.

The file is plain text with pseudo xml sections for configuring properties and functionality of the web server. You must read the manual to be certain of any changes you make to this file, otherwise you may inadvertently create a security hole that will be quickly breached.

An example configuration section for Apache 2 running mod_php with the DiGIR Provider service unpacked in the folder C:\var\ is shown below. Please check that the configuration settings are appropriate for your system if you are just copying the configuration information.

#PHP Configuration section - this portion enables the PHP interpreter

#The DiGIR administrative interface. The settings allow access

#only from the localhost, meaning that you can only access the admin

# folder from the machine that is running the web server. The admin

# folder is physically located at c:/var/DiGIRprov/admin, and is

# accessible through the web browser at the address: http://localhost/digir/admin

Alias /digir/admin C:/var/DiGIRprov/admin

<Directory C:/var/DiGIRprov/admin>

AllowOverride None

Order deny, allow

Deny from all

Allow from 127.0.0.1

DirectoryIndex index. html, index. php

</Directory>

#Alias for the DiGIR Provider service. The settings are for a

#provider that was installed to c:/var/DiGIRprov, with the default

#folder name (www) for the location of the DiGIR php scripts.

Alias /digir C:/var/DiGIRprov/www

<Directory C:/var/DiGIRprov/www>

AllowOverride None

Order allow, deny

Allow from all

DirectoryIndex index. html, index. php

</Directory>

b.  PHP configuration:

PHP Configuration Directives.

The PHP interpreter engine uses a configuration file to set default operating parameters. This file, php. ini is a plain text file that is formatted like a windows "ini" file. There are a couple of adjustments, or rather settings to check in the php. ini file. On windows, php. ini is located in the $SystemRoot folder (c:\windows). On Linux, it is located in /etc.

PHP File Paths

PHP understands forward (/) and backward (\) slashes to be file path separators. The only difference is that when the backslash is used in PHP code, it must be escaped by typing a double-backslash (e. g. c:\\digir\\phpprovider). To avoid confusion, it is best to get into the habit of just using a forward slash for path separators (e. g. c:/digir/phpprovider).

register_globals

This setting should be off. This setting does not alter the functionality of the DiGIR provider, but it does have a big impact on the security of your web server. See the discussion on .

extension_dir

This entry indicates where php can find it's extensions. These are compiled libraries, which are dynamically loaded by the PHP interpreter. For the purposes of DiGIR, it should be a full path to the folder that contains the extension php_mbstring.*. If you installed PHP to c:\php then the extension folder setting is most likely:

extension_dir = c:/php/extensions

cgi. force_redirect

If you are running the IIS web server, then this must be 1 or On.

extension=php_mbstring

The default configuration setting has this library commented out. If you are running PHP as a CGI application, then it is safe to leave it this way. If you are running PHP as a module loaded by the web server, then you must uncomment this entry so that the PHP interpreter loads it, otherwise it will not be available to the DiGIR provider (and will cause error messages in the responses).

extension=php_sockets

This library is required by the DiGIRm service. If DiGIRm is disabled then there is no need to enable this library which implements low level Internet communication functions. The default configuration setting has this library commented out. If you are running PHP as a CGI application, then it is safe to leave it this way. If you are running PHP as a module loaded by the web server, then you must uncomment this entry so that the PHP interpretor loads it, otherwise it will not be available to the DiGIR provider (and will cause error messages in the responses).

There are numerous other settings in php. ini. Please review the php manual if you are curious about how these affect the operation of your PHP installation.

Uncomment the modules in php. ini used in DIGIR as the DAS main core:

extension=php_iconv. dll

extension=php_mbstring. dll

extension=php_ming. dll

extension=php_sockets. dll

You need to uncomment one of the database dll extension. It depends from the database(s) used in you local data system (e. g. ;extension=php_oracle. dll for Oracle, extension=php_mssql. dll for MSSQL Server and etc.)

*****nning Data Provider.

To run Data Provider follow this steps:

*****nning Apache HTTP Server to start up DiGIR.

If you have installed Apache as the service go to the Control Panel->Administrative Tools->Services and run the service "Apache”. It’s recommended to edit the service properties and set option "Startup type" to "manual".

If you installed Apache not as service go the Apache installation directory and run the server.

Related information how to start up Apache is available at http://httpd. apache. org/docs/invoking. html

Make sure that http://yourdomain:[DIGIR_PORT]/digir is available.

If you have followed the instructions properly you can see the DiGIR Provider Service page with the status information and the endpoint link.

*****nning Tomcat to start up DPMS.

If you have installed Tomcat as the service go to the Control Panel->Administrative Tools->Services and run the service (e. g. "Apache Tomcat 4.1").It's recommended to edit the service properties and set option "Startup type" to "manual".

If you installed Tomcat not as service go the Tomcat installation directory and run the server.

Related information how to start up Tomcat is available at http://jakarta. apache. org/tomcat/tomcat-4.1-doc/RUNNING. txt.

Make sure that http://youdomain:[DPMS_PORT]/dpms/PortalServlet? action=getProviders is available.

If you see the XML structure on the web page, describing your local data provider information, then DPMS is working.

If there are any errors during launch, please, check the installation and configuration options. For any problems with the web-servers see the related documentation.

Installing, configuring and running E2EDM Integration Server

1.  Installation and configuration.

1.1  Setup.

Download and install the latest version of Java 2. You must have at least version 1.4 of the JDK. At the time of writing, the version used in development and testing was Java 2 SDK 1.4.2.

(Visit http://www. java. /downloads).

Follow the instructions available for installing the downloaded package. Be sure to do the following:

Set the JAVA_HOME environment variable to the location of your Java 2 install (e. g. C:\j2sdk1.4.x). This should be done in a permanent fashion.

Modify your PATH environment variable to include the Java 2 bin directory (e. g. C:\j2sdk1.4.x\bin). Again, this should be done in a permanent fashion.

Download and install JBoss Application Server 4.0.0. Do not install a version earlier than 4.x or you will definitely have compatibility issues. (Visit http:///project/showfiles. php? group_id=22866&package_id=16942 or http://www. jboss. org/products/jbossas)

For more details about JBoss installation visit http://www. jboss. org/docs/index#as.

Copy iserv. war folder from the INSTALLATION_PACK/INTEGRATION_SERVER directory to %JBOSS_HOME%\server\default\deploy. This is the location of all web-applications in JBoss.

The directory structure for Integration Server is:

codes	Codes used in the E2EDM project (countries, organizations, etc.)
data	MapServer data files (shp-, shx-, dbf-, sbn-, sbx - files)
jBox	MapServer Java-applet
map	MapServer system directory
symbols	Fonts and symbols for MapServer
tmp	Directory used by Integration Server for temporary files
WEB-INF	Base directory, containing Integration Server modules and configuration files

Copy the libraries form iserv. war\WEB-INF\cgi\libwww_dll to the %WINDOWS%\system32 or %WINDOWS%\system, or you can add the path to these libraries in the environment variable PATH.

Copy the *.jar files from INSTALLATION_PACK/INTEGRATION_SERVER\lib directory to the %JBOSS_HOME%\server\default\lib.

1.2  Configuration.

Configuration of Integration Server involves editing of JBoss settings and editing Integration Server configuration files.

JBoss settings:

The full document for server configuration you can find at http://www. jboss. org/docs/index#as.

The default port for JBoss AS is set to 8080.

IMPORTANT NOTE: You need to check, that 8080 port is available from the external network! This is important for the connection to the Integration Server. If this port is blocked by firewall you can set port 80 as alternative! We strongly recommend using port 80 for the Integration Server. The port address can be changed in %JBOSS_HOME%\server\default\deploy\jbossweb-tomcat50.sar\server. xml.

Find the following structure in server. xml file:

<!-- A HTTP/1.1 Connector on port 8080 -->

<Connector port="8080" address="${jboss. bind. address}"

maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" redirectPort="8443" acceptCount="100"

connectionTimeout="20000" disableUploadTimeout="true"/>

Change the port="8080" to port="80". Now JBoss will run at port 80.

For security issues:

Open the %JBOSS_HOME%\server\default\deploy\jmx-console. war\WEB-INF\web. xml file. Find the following structure:

<security-constraint>

<web-resource-collection>

<web-resource-name>HtmlAdaptor</web-resource-name>

<description>An example security config that only allows users with the

role JBossAdmin to access the HTML JMX console web application

</description>

<url-pattern>/*</url-pattern>

<http-method>GET</http-method>

<http-method>POST</http-method>

</web-resource-collection>

<auth-constraint>

<role-name>JBossAdmin</role-name>

</auth-constraint>

</security-constraint>

Uncomment this structure. Now the jmx-console (control module) is secured. See the full documentation for JBoss AS server configuration.

Integration Server settings:

Open %JBOSS_HOME%\server\default\deploy\iserv. war\WEB-INF\web. xml configuration file.

Find the records like

<init-param>

<param-name>E2ELinkMD_path</param-name>

<param-value>c:\jboss-4.0.0\server\default\deploy\iserv. war\WEB-INF\LinkMDList. xml</param-value>

<description>Path to the E2ELinkMD record structure, describing local data sources</description>

</init-param>

Replace the path in <param-value> tag by actual installation path for JBoss.

IMPORTANT NOTE: DO NOT DELETE THE FILES CONFIGURED IN THIS DOCUMENT TO ANY OTHER LOCATION!

Open %JBOSS_HOME%\server\default\deploy\iserv. war\WEB-INF\LinkMDList. xml file and configure the providers that Integration Server will be working with. Each <E2ELinkMD> structure is one Data Provider system. Set the access point to the Data Provider in <objectOnlineId>http://providerdomain:10080/dpms/controller</objectOnlineId>. You need to set Provider domain name and port. The Provider ID (objectSystemId) is unique in the E2EDM integration technology, so the value must be the same, as the Provider has.

Set the domain names in the following structures:

<E2EObjectReference>

<!-- path to the local datasystem resource description storage-->

<objectTitle>PROVIDER_RESOURCE_LIST</objectTitle>

<objectInfoURL>http://providerdomain:10080/dpms/md. xml</objectInfoURL>

</E2EObjectReference>

<E2EObjectReference>

<!-- path to the local datasystem codes storage-->

<objectTitle>PROVIDER_CODE_FILES</objectTitle>

<objectInfoURL>http:// providerdomain:10080/dpms/</objectInfoURL>

</E2EObjectReference>

Resources description.

All resource descriptions provided by the Integration Server are stored in the %JBOSS_HOME%\server\default\deploy\iserv. war\WEB-INF\md. xml file. This file contains full description of each resource in XML structure. For more details about the complete structure see technical documentation. Integration Server collects resource descriptions from all the Data Providers, configured in LinkMDList. xml every night and on server start-up! DO NOT CHANGE THIS FILE ONLY IF YOU ARE NOT SURED THAT IT WILL NOT CAUSE ANY PROBLEMS DURING INTEGRATION SERVER WORK.

Distributable version of the Integration Server includes example of the resource description.

2.  Running Integration Server.

To run Integration Server follow this steps:

1)  Run JBoss AS by executing run. bat in %JBOSS_HOME%\bin directory. You will see the server console and status of the JBoss start up.

2)  Make sure that http://yourdomain:[JBOSS_PORT]/iserv/ is available.

If you are experiencing some problems during start up go to the JBoss AS documentation for more information.

E2EDM user interface guide

E2EDM user interface is available through the web-browser. It was tested on such browsers as Internet Explorer 6.0, Opera 7.0 and higher, Netscape 7.0 and higher and Mozilla. Users can access to the E2EDM user interface through the E2EDM project site (http://data. *****/e2edm/).

Fig.1. Main page of the E2EDM user interface.

Page is divided into 3 parts – left, right side and bottom. Left side includes map navigation component with “Map tools” (zooming, refreshing and flushing the current map), “Layer controls” and “Label controls”. Navigation component is based on the MapServer which is open source map server. Bottom side contains status information during the search process (search progress and search time) and status information about results. Right side includes search form. It consists of 5 sections: resources list, parameters, date and time period, vertical extent and region:

“Resources”

Resources list is dynamical; it represents how many resources are available on the distributed Data Provider systems at moment. User can select single or multiple resources. If there is no resources selected default value - first resource in the list.

Below there are two buttons “Show resources on map” and “Remove resources form map” that allow to the user see on the map what region each resource covers (fig.2).

Fig.2. Geographic coverage of selected resources.

“Parameters”

Contains list of marine and meteorological parameters.

“Date & time”

Date period for the data (e. g. observation begin/end date and time, etc.)

“Depth”

List of vertical extent intervals.

“Bounding box”

Upper-left and lower-right corner. If there is selected area no the left side (map interface) coordinates are inserted automatically.

To begin search user must submit form by clicking “Submit button”.

Fig.3. Search in progress.

Fig.4. Result status.

After receiving message that all request are handled and all data is receiver user can see result page clicking button “View DataPoint” or “View DataProfile” or “View DataGrid”, depending on the resource data type.

Result page consists of 2 sections: data table (left) and map view (right). All instances, series and grids are represented on the map.

E2EDM user interface is demo at current moment. More functionality as user projects, result saving and export will be provided.