• SQL Express 2014 or later
• Visual Studio 2017 or later
• Docker (Optional) - Used to host ElasticSearch and RabbitMQ, which enable full-text search.

• The folder holding the bhl-us source code is referred to throughout this document as <BHLRoot>.
• These instructions assume that the databases will be named "BHL", "BHLImport", "BHLAuditArchive", and "IAAnalysis".

After downloading the bhl-us source code, do the following to get the web sites and utility applications running.

1. Open a Windows command prompt.

2. Make sure that the sqlcmd utility, which is part of the SQL Server client tools, is included in your path. More information can be found at http://technet.microsoft.com/en-us/library/ms162773.aspx.

3. Navigate to the <BHLRoot>\Database-BHL folder.

4. Run the BHLDBBuildScript.bat batch file. This will build the primary database.

   Usage:

   BHLDBBuildScript SERVERNAME DATABASENAME FULLTEXTCATALOGFILEPATH ISPRODUCTION DATAORSTRUCTURE

   where

   SERVERNAME is the name of the database server DATABASENAME is the name of the database. It is recommended that "BHL" be used as the database name. FULLTEXTCATALOGFILEPATH is the path in which to place the full-text catalog file. Use quotes around this value if the path contains spaces. ISPRODUCTION is true for a production database, and false for a development database. Auditing triggers are removed from development databases. DATAORSTRUCTURE is "structure" to build the empty database (no data), "data" to add data to an existing database, or "all" to build the structure and add the data.

   Example:

   BHLDBBuildScript localhost BHL "C:\Program Files\Microsoft SQL Server\MSSQL11.SQLEXPRESS\MSSQL\DATA" false all

5. In the new BHL database, create roles named db_executor and db_webuser.

   USE [BHL]; CREATE ROLE db_executor; GRANT EXECUTE TO db_executor;

   CREATE ROLE db_webuser; GRANT INSERT ON dbo.PDF TO db_webuser; GRANT INSERT ON dbo.PDFPage TO db_webuser; GRANT UPDATE ON dbo.Page TO db_webuser;

6. Navigate to the <BHLRoot>\Database-BHLImport folder.

7. Run the BHLImportDBBuildScript.bat batch file. This will build the database used as a staging area for new material.

   Usage:

   BHLImportDBBuildScript SERVERNAME DATABASENAME DATAORSTRUCTURE

   SERVERNAME is the name of the database server DATABASENAME is the name of the database. It is recommended that "BHLImport" be used as the database name. DATAORSTRUCTURE is "structure" to build the empty database (no data), "data" to add data to an existing database, or "all" to build the structure and add the data.

   Example:

   BHLImportDBBuildScript localhost BHLImport all

8. In the new BHLImport database, create a role named db_webuser.

   USE [BHLImport]; CREATE ROLE db_webuser; GRANT SELECT ON dbo.IAFile TO db_webuser; GRANT SELECT ON dbo.IAItem TO db_webuser;

9. Navigate to the <BHLRoot>\Database-BHLAuditArchive folder.

10. Run the BHLAuditArchiveDBBuildScript.bat batch file. This will build the auditing database.

    Usage:

    BHLAuditArchiveDBBuildScript SERVERNAME DATABASENAME

    where

    SERVERNAME is the name of the database server DATABASENAME is the name of the database. It is recommended that "BHLAuditArchive" be used as the database name.

    Example:

    BHLAuditArchiveDBBuildScript localhost BHLAuditArchive

11. Navigate to the <BHLRoot>\Database-IAAnalysis folder.

12. Run the IAAnalysisDBBuildScript.bat batch file. This will build the database used to ingest non-biodiversity-collection items from Internet Archive.

    Usage:

    IAAnalysisDBBuildScript SERVERNAME DATABASENAME DATAORSTRUCTURE

    where

    SERVERNAME is the name of the database server DATABASENAME is the name of the database. It is recommended that "IAAnalysis" be used as the database name. DATAORSTRUCTURE is "structure" to build the empty database (no data), "data" to add data to an existing database, or "all" to build the structure and add the data.

    Example:

    IAAnalysisDBBuildScript localhost IAAnalysis all

13. Create a new SQL Server login named BHLWebUser. Map it to a user named BHLWebUser in the new BHL database, and assign it to the "db_executor" and "db_webuser" database roles.

    USE [master]; CREATE LOGIN [BHLWebUser] WITH PASSWORD=N'BHLWebUser';

    USE [BHL]; CREATE USER [BHLWebUser] FOR LOGIN [BHLWebUser] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_executor] ADD MEMBER [BHLWebUser]; ALTER ROLE [db_webuser] ADD MEMBER [BHLWebUser];

14. Map the BHLWebUser login to a user named BHLWebUser in the new BHLImport database, and assign it to the "db_webuser" database role.

    USE [BHLImport]; CREATE USER [BHLWebUser] FOR LOGIN [BHLWebUser] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_webuser] ADD MEMBER [BHLWebUser];

15. Create a new SQL Server login named BHLService. Map it to a user named BHLService in the BHL, BHLAuditArchive, BHLImport, and IAAnalysis databases. In each database, assign the new user to the "db_datareader", "db_datawriter", and "db_owner" database roles.

    USE [master]; CREATE LOGIN [BHLService] WITH PASSWORD=N'BHLService';

    USE [BHL]; CREATE USER [BHLService] FOR LOGIN [BHLService] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_datareader] ADD MEMBER [BHLService]; ALTER ROLE [db_datawriter] ADD MEMBER [BHLService]; ALTER ROLE [db_owner] ADD MEMBER [BHLService];

    USE [BHLAuditArchive]; CREATE USER [BHLService] FOR LOGIN [BHLService] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_datareader] ADD MEMBER [BHLService]; ALTER ROLE [db_datawriter] ADD MEMBER [BHLService]; ALTER ROLE [db_owner] ADD MEMBER [BHLService];

    USE [BHLImport]; CREATE USER [BHLService] FOR LOGIN [BHLService] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_datareader] ADD MEMBER [BHLService]; ALTER ROLE [db_datawriter] ADD MEMBER [BHLService]; ALTER ROLE [db_owner] ADD MEMBER [BHLService];

    USE [IAAnalysis]; CREATE USER [BHLService] FOR LOGIN [BHLService] WITH DEFAULT_SCHEMA=[dbo]; ALTER ROLE [db_datareader] ADD MEMBER [BHLService]; ALTER ROLE [db_datawriter] ADD MEMBER [BHLService]; ALTER ROLE [db_owner] ADD MEMBER [BHLService];

The BHL web site uses ElasticSearch to implement many of its search capabilities. However, if ElasticSearch is not present, the web site will fall back to a SQL Server search implementation. The SQL Server search performs basic searches of catalog metadata. Full-text searches are disabled, faceting is not available, and the “search within a book” feature will not work. Therefore, for a full-featured BHL implementation, the instructions in this section should be followed. If a limited search is satisfactory, then this section, the “RabbitMQ” section, and the “Index Data in ElasticSearch” section can be skipped.

1. (If necessary) Download and install Docker from docker.com. As of January 2019, the product to install is called “Docker Desktop”.

2. Open a Windows command prompt.

3. Get the official ElasticSearch image.

   docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.1

4. Start a new ElasticSearch docker container that will be accessible at http://localhost:9200.

   docker run -d --name es791 -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:7.9.1

5. Locate the "elasticsearch.yml" file within the running Docker container. The following command should return something like "/usr/share/elasticsearch/config/elasticsearch.yml"

   docker exec -it es791 find / -name "elasticsearch.yml"

6. Copy the elasticsearch.yml file from the container to the host.

   docker cp es791:/usr/share/elasticsearch/config/elasticsearch.yml c:\elasticsearch.yml

7. On the host, use a text editor to disable Xpack security by adding the following line to the elasticsearch.yml file:

   xpack.security.enabled: false

   The file contents should now look something like this:

   cluster.name: "docker-cluster" network.host: 0.0.0.0

   # minimum_master_nodes need to be explicitly set when bound on a public IP # set to 1 to allow single node clusters # Details: elastic/elasticsearch#17288 discovery.zen.minimum_master_nodes: 1

   xpack.security.enabled: false

8. Copy the edited elasticsearch.yml file from the host to the running Docker container.

   docker cp c:\elasticsearch.yml es791:/usr/share/elasticsearch/config/elasticsearch.yml

9. Add the ICU analysis plug-in to ElasticSearch to add better support for Unicode characters, including Asian characters.

   docker exec -it es791 /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu

10. Stop the running container.

    docker stop es791

11. Restart the ElasticSearch container. Note that by using the name (es791) assigned to the container is Step 4, all of the other arguments we specified in Step 4 (-d -e -p ) are used by default.

docker start es791

12. Verify the operation of ElasticSearch by opening a browser and navigating to http://localhost:9200. You should get a response that looks something like this:

    { "name" : "90WsOOT", "cluster_name" : "docker-cluster", "cluster_uuid" : "Ok4_vavaTuSxn9qrsAGZwA", "version" : { "number" : "5.4.2", "build_hash" : "f9d9b74", "build_date" : "2017-02-24T17:26:45.835Z", "build_snapshot" : false, "lucene_version" : "6.4.1" }, "tagline" : "You Know, for Search" }

13. Create new indexes using curl or a comparable tool.

    curl –X PUT http://localhost:9200/items -d @\ElasticSearch\items.json –H “Content-Type:application/json” curl –X PUT http://localhost:9200/catalog -d @\ElasticSearch\catalog.json –H “Content-Type:application/json” curl –X PUT http://localhost:9200/authors -d @\ElasticSearch\authors.json –H “Content-Type:application/json” curl –X PUT http://localhost:9200/keywords -d @\ElasticSearch\keywords.json –H “Content-Type:application/json” curl –X PUT http://localhost:9200/names -d @\ElasticSearch\names.json –H “Content-Type:application/json” curl –X PUT http://localhost:9200/pages -d @\ElasticSearch\pages.json –H “Content-Type:application/json”

1. Open a Windows command prompt.

2. Get the official RabbitMQ image

   docker pull rabbitmq

3. Start a new ElasticSearch docker container that will be accessible at http://localhost:5672.

   docker run -d --name rabbit373 --hostname my-rabbit -p 5672:5672 rabbitmq

   (RECOMMENDED) To alternately include the RabbitMQ management console, which will be accessible at http://localhost:15672, use this instead:

   docker run -d --name rabbit373 --hostname my-rabbit -p 5672:5672 -p 15672:15672 rabbitmq:management

4. the operation of RabbitMQ by opening a browser and navigating to http://localhost:5672. You should get a response that looks something like this:

   AMQP

5. Verify the operation of the RabbitMQ management console by opening a browser and navigating to http://localhost:15672. Use guest/guest as the username/password.

   NOTE: To supply a different username and password, change the command that creates a RabbitMQ container with the management console to the following:

   docker run -d --name rabbit373mgmt --hostname my-rabbit -p 5672:5672 -p 15672:15672 -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:management

1. Make copies of the config files as indicated in the following list:

 
2) Make the specified modifications to each of the config files in the following list:

# = denotes optional modifications that are not required for development installations

 
WWW.BIODIVERSITYLIBRARY.ORG

<BHLRoot>\BHLService\Web.config

<BHLRoot>\BHLUSWeb2\Web.config

 
ADMIN.BIODIVERSITYLIBRARY.ORG

<BHLRoot>\BHLAdminWeb\Web.config

 
<BHLRoot>\SiteServices\Web.config

 
DATA IMPORT APPS

<BHLRoot>\BHLBioStorHarvest\app.config

<BHLRoot>\BHLImportEFDataModel\App.Config

<BHLRoot>\BHLImportService\Web.config

<BHLRoot>\IAAnalysisHarvest\App.config

<BHLRoot>\IAHarvest\App.config

 
UTILITY APPS

<BHLRoot>\BHLDOIService\app.config

<BHLRoot>\BHLExportProcessor\App.config

<BHLRoot>\BHLFlickrTagHarvest\app.config

<BHLRoot>\BHLFlickrThumbGrab\app.config

<BHLRoot>\BHLMETSUpload\app.config

<BHLRoot>\BHLNameFileGenerator\app.config

<BHLRoot>\BHLOAIHarvester\app.config

<BHLRoot>\BHLOCRRefresh\app.config

<BHLRoot>\BHLPageNameRefresh\app.config

<BHLRoot>\BHLPDFGenerator\app.config

<BHLRoot>\BHLSearchIndexer\AppConfig.xml

<BHLRoot>\BHLSearchIndexer\AppConfig.Names.xml

<BHLRoot>\BHLSearchIndexer\AppConfig.Full.xml

<BHLRoot>\BHLSearchIndexQueueLoad\AppConfig.xml

<BHLRoot>\BHLTextImportProcessor\app.config

TEST PROJECTS

<BHLRoot>\BHLApiDALTest\App.config

<BHLRoot>\BHLCoreDALTest\App.config

<BHLRoot>\BHLServerTest\app.config

1. Open the BHLUtility solution in Visual Studio.
2. Build the BHLSearchIndexer project.
3. Make sure you have completed the configuration within the AppConfig.Full.xml file.
4. Run the BHLSearchIndex project, specifying “AppConfig.Full.xml” as the application argument.