Configure and deploy raster analytics

Raster analytics using ArcGIS Enterprise is a flexible raster processing, storage, and sharing system that employs distributed computing and storage technology based on ArcGIS Image Server. Use raster analytics to apply raster processing tools and functions offered in ArcGIS, build your own custom functions and tools, or combine multiple tools and functions into raster processing chains to execute your custom algorithms on large collections of raster data. Source data and processed results are stored, published, and shared across your enterprise according to your needs and priorities.

This extensive capability can be further expanded by leveraging cloud computing capabilities and resources. The net result is that image processing and analysis jobs that used to take days or weeks can now be done in minutes or hours, and jobs that were too large or daunting can now be handled.

Introduction

Raster analytics is ArcGIS Image Server configured for raster analysis in a processing and storage environment that maximizes processing speed and efficiency. Built-in tools and functions cover preprocessing, orthorectification and mosaicking, remote sensing analysis, and an extensive range of math and trigonometry operators. Your custom functions can extend the platform's analytical capabilities even further.

Fully utilize your existing ArcGIS Image Server on-site, or exploit the elastic processing and storage capacity of cloud computing and storage platforms such as ArcGIS Enterprise on Amazon Web Services and ArcGIS Enterprise on Microsoft Azure to dynamically increase or reduce your capacity depending on the size and urgency of your projects. The scalable environment of raster analytics empowers you to implement computationally intensive image processing that used to be out of reach or cost-prohibitive. This implementation saves you time, money, and resources.

Raster analytics is also designed to streamline and simplify collaboration and sharing. Users across your enterprise can contribute data, processing models, and expertise to your imagery project, and share results with individuals, departments, and organizations in your enterprise.

Configure a raster analytics deployment

The foundation of raster analytics is composed of 3 main components: ArcGIS Enterprise; raster analysis, based on ArcGIS Image Server; and an image hosting site, including raster data store, based on ArcGIS Image Server.

Raster analysis requires an ArcGIS Enterprise base deployment, which provides software and user interfaces to organize and manage your processing, storage, and sharing of raster and feature data, maps, and other geographic information on a variety of devices. This integrated system manages the dissemination of processing and storage of results on-premises and behind the firewall, in cloud processing and storage environments, or a combination of both environments.

The ideal deployment of raster analytics is composed of three server sites to perform the primary roles of the image hosting site, raster analysis site, and dynamic image site. Two licenses are required for raster analytics, ArcGIS Enterprise and ArcGIS Image Server.

The advantage of configuring raster analysis with three servers having distinct roles is to maximize efficiency and productivity. Separating distributed raster analysis processing and image service hosting roles enables resources to be dedicated to specific tasks without interruption. In this way, multiple users requesting access to various raster products and services do not compete for and impact raster analysis and image processing resources.

Set up ArcGIS Enterprise

ArcGIS Enterprise integrates the components of the raster analytics system to support scalable raster processing and storage workflows.

You can install an ArcGIS Enterprise base deployment across one or more machines, any of which can be physical, virtual, or cloud machines. For details about the required machine specifications for the ArcGIS Enterprise components, see the system requirements.

The process of installing and configuring an ArcGIS Enterprise base deployment can be done manually or automatically with the use of the ArcGIS Enterprise Builder.

Esri also provides templates to enable you to deploy ArcGIS Enterprise in the cloud. These templates set up ArcGIS Enterprise in a configuration of your choice. For a single-machine deployment, use the following template:

• Single-machine deployment using CloudFormation templates on Amazon Web Services.

For a multiple-machine deployment, use the following templates:

• Multiple-machine deployment on-premises—See the highly available system configuration template.
• Provision a highly available ArcGIS Enterprise deployment on Amazon Web Services.

Raster analysis

Distributed raster processing is performed by the ArcGIS Image Server assigned to the role of raster analysis. It executes the raster function processing chains on any supported raster data types. Source imagery and raster data can come from the distributed raster data store, a local client such as ArcGIS Pro, or other servers not configured in your raster analysis deployment. Ideally, your source data is staged on the distributed raster data store for optimum performance and efficiency.

Members of your portal will not interact directly with ArcGIS Image Server; they will use ArcGIS Pro, Map Viewer, or clients built from APIs such as ArcGIS REST API, ArcGIS API for Python, or ArcGIS API for JavaScript to perform raster analysis. ArcGIS Image Server will perform the actual processing, storage, and publishing roles.

You need to set up and federate an ArcGIS Image Server in the role of a raster analysis server to perform distributed raster analysis and processing workflows. The raster analysis server automatically manages and distributes image processing and raster analysis tasks at the block level or the scene level depending on the tools and functions used. The raster analysis server manages the distribution of processing results to either ArcGIS Data Store on the hosting server for feature data products, or to the raster data store for imagery and raster data products.

The steps to set up and deploy an on-premises ArcGIS Image Server for raster analysis are outlined below.

Set up an ArcGIS Image Server raster analysis site

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding.

1. Make sure that you have configured a base ArcGIS Enterprise deployment.
2. Install an additional ArcGIS Server site to function as the ArcGIS Image Server.
3. Authorize ArcGIS Server with an ArcGIS Image Server license.
4. Register the data store with ArcGIS Image Server. Raster analysis services store processed rasters in a data store. This can be either a file share or cloud store type of data store.

   Note:

   If an image hosting server is specified in your raster analytics deployment, and you want to publish an image service, the same raster stores need to be registered in both your image hosting server and raster analysis server.

   If the input data is accessed from your data stores, those data stores should be registered in both the raster analysis and image hosting servers.

   1. To use a folder, use ArcGIS Server Manager, register a UNC share in the data store, and give it a name.

      Note:

      If there are multiple instances of _raster_store defined, for example, raster_store1, raster_store2, and so on, raster analysis will randomly pick one for a task request.

   2. To use a cloud store, you need to provide the access key. The cloud store settings include subfolder support.

Federate and configure the server for raster analysis

Next, you will designate the ArcGIS Image Server site to perform raster analysis.

1. Sign in to the portal as a member with administrative privileges. The URL is in the format https://webadaptorhost.domain.com/<webadaptorname>/home.
2. Browse to Organization > Settings > Servers.
3. Federate your ArcGIS Image Server site with your portal.
4. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
5. Click the ellipsis next to the configured server and select Configure server role.
6. On the Configure server role dialog, enable Raster Analysis.
7. Click Save.

Assign permissions

Next, you will assign the minimum permissions needed to perform raster analysis.

1. Navigate to the Organization > Member roles page.
   1. Beside the role that will be performing raster analysis, click the ellipsis and select Edit.
   2. On the Edit role window, expand Content.
   3. Enable the options for Create, update, and delete and Publish hosted feature layers.
   4. Expand Content and Analysis.
   5. Enable Imagery Analysis.

Configure raster analysis services

It is recommended that you configure raster analysis services and increase the SOC maximum heap size.

ArcGIS Image Server raster analysis services have a default processing capacity configuration. You may need to customize this configuration based on your hardware configuration. Log in to ArcGIS Server Manager and browse to Services > System folder, where you can configure the following GIS services:

• To increase the processing speed and capacity, increase the maximum instance of the RasterProcessing service.
• To increase availability of the analytical results, increase the maximum instance of the RasterRendering service.
• To increase the number of processing tasks that can be executed concurrently, increase the maximum instance of the RasterAnalysisTools service.

1. Log in to the ArcGIS Server Administrator Directory using the URL format http://machine.domain.com:6443/arcgis/admin.
2. From the directory home page, click machines.
3. From the list of machines, select the first machine licensed with ArcGIS Image Server, whether it is used as an image hosting server or as a raster analytics server.
4. In the Server Machine Properties menu, check the value for SOC maximum heap size. If it is set to 64MB, click Edit in the Supported Operations list.
5. On the Edit Machine page that opens, change the value for SOC maximum heap size from 64MB to 128MB.
6. Click Save Edits. The server restarts.
7. Repeat these steps for each machine in your deployment that is licensed with ArcGIS Image Server.

Note:

If you want to deploy your image hosting server in the cloud with ArcGIS Enterprise on Amazon Web Services, see ArcGIS Enterprise Cloud Builder CLI for AWSand AWS CloudFormation and ArcGIS. If you want to deploy your raster analysis server in the cloud with Microsoft Azure, see Deploy ArcGIS Enterprise on Microsoft Azure.

Configure a temporary directory for cloud raster format

Raster analysis tools can utilize a temporary directory to speed up reading and writing. The directory has two primary purposes:

• Caching cloud stored .crf datasets
  • Accommodates all raster analysis tools when reading and writing cloud stored .crf files
  • The disk space usage is controlled by a maximum of 30 bundles cached per opened .crf dataset. Using this number, the average size of a bundle (greater than 50MB), and estimated number of opened .crf datasets, the temporary space required for each SOC machine can be determined.
• Storing temporary processing block data for the distributed processing global tools, including the Hydrology, Distance, and Generalization tools.
  • The disk space usage is proportional to the size of the primary input. Depending on the tool, it can be 3 to 5 times the size of the input, distributed across multiple SOC machines.

A temporary folder can be a shared folder or a local folder on each SOC machine. If not specifically configured, the default temporary folder is located under your user’s profile, for example, C:\Users\username\AppData\Local\ESRI\LocalCaches\MapCacheV1.

There are 2 options to configure temp directory for Raster Analytic image server.

1. A single temporary folder path, "localTempFolder" in the server administration system properties, for example, https://servername:6443/arcgis/admin/system/properties.

   It is also used by the server caching tools, for example, {"localTempFolder":"E:/Temp/RA"}.

2. A property defining a list of temporary folder paths (semi-colon separated) “localTempFolders” in the server administration system properties, for example, https://servername:6443/arcgis/admin/system/properties. Raster analysis tools randomly pick a temporary folder from the list. This allows the use of multiple local disks on SOC machines, for example, {"localTempFolders":"E:/Temp/RA;F:/Temp/RA"}, assuming all SOC machines have E: and F: drives.

The list of temporary folder paths “localTempFolders” takes higher priority over the single temporary folder path, "localTempFolder", if both are configured.

Configure a temporary location for raster analysis tools

The raster analysis geoprocessing tools, Summarize Raster Within, Convert Raster To Feature, can execute in parallel on multiple machines. For these tools to share partial results between those machines, specially named entries must be added to the raster data store for the raster analytics server cluster. A fast, local temporary directory on each machine must be identified, and a UNC share containing the string machinename_tmp must be defined for that directory. That share must be added as a file share to the raster data store.

For example, assume the raster analytics cluster has two machines named workerA and workerB. Create a temporary directory on each machine, on a fast local disk with lots of space, and define these UNC shares: \workerA_raster_store_workerA_tmp and \workerB_raster_store_workerB_tmp. The share names can have any name, so long as the text string machinename_tmp is present. When a SummarizeRasterWithin or ConverRasterToFeature job is executing, worker machines will use those locations to write their partial results in parallel. The results will then be read across the local network and collated as needed by the specific operation.

Image hosting site

The image hosting site hosts the raster data store and manages the file share storage as well as the cloud storage, and distributes results according to portal member requests. The raster data store is not part of ArcGIS Data Store but rather a registered file share or cloud share store federated with ArcGIS Enterprise. The raster data store is used by the raster analysis services to store output imagery optimized for reading, writing, and storage in a distributed format on-premises or in the cloud.

The advantage of configuring raster analysis with three servers having distinct roles is to maximize efficiency and productivity. Separating distributed raster analysis processing and image service hosting roles enables resources to be dedicated to specific tasks without interruption. In this way, multiple users requesting access to various raster products and services do not compete for and impact raster analysis and image processing resources.

You need to set up and federate a second ArcGIS Image Server to function as the image hosting server, which hosts all the distributed image services generated by the raster analysis server. It includes the raster data store configured with ArcGIS Image Server Manager, which manages distributed file share storage and cloud storage such as Amazon Simple Storage Service (S3) or Microsoft Azure Blob storage of image services. The image hosting server stores and returns distributed results requested by members in the Enterprise portal. The image hosting server requires an ArcGIS Image Server license.

The instructions to set up and configure the image hosting server are detailed below.

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding. To configure the portal to perform image hosting services, follow the steps below.

1. Install ArcGIS Server.
2. Authorize ArcGIS Server with an ArcGIS Image Server license.
3. Log in to ArcGIS Server Manager and browse to Site > Data Stores. Register a new raster store.

   The raster store is a location for output from raster analysis and should not be used as an input data directory. File shares can be registered directly as a raster data store. For cloud-based stores, first register a cloud store. Then create a raster store to use this cloud store.

   Note:

   If there are multiple raster stores defined, the raster analysis service will randomly pick one as the output location.

4. Make sure you have increased the SOC maximum heap size to 128 MB for ArcGIS Image Server. See steps 9a through 9f in the Set up an ArcGIS raster analysis server section above.

Note:

If you want to deploy your image hosting server in the cloud with Amazon Web Services, see ArcGIS Enterprise Cloud Builder CLI for AWS. If you want to deploy your raster analysis server in the cloud with Microsoft Azure, see Deploy ArcGIS Enterprise on Microsoft Azure.

Make sure you have increased the SOC maximum heap size to 128 MB for ArcGIS Image Server.

Federate and configure the server for image hosting

Next, you will designate the ArcGIS Image Server site to host your image services created from imagery layers.

1. Sign in to the portal as a member with administrative privileges. The URL is in the format https://webadaptorhost.domain.com/<webadaptorname>/home.
2. Browse to Organization > Settings > Servers.
3. Federate your ArcGIS Server site.
4. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
5. Click the ellipsis next to the configured server and select Configure server role.
6. On the Configure server role dialog, enable Image Hosting.
7. Click Save.

Minimum raster analysis system

If an organization has resources and limited conflicts in terms of competing raster analysis processing and image service distribution resources, the raster analysis server and image hosting server functionality can be combined and hosted on one server. In this implementation, the raster analysis and image processing operations are executed on an ArcGIS Image Server site configured as the raster analysis server, while the results are managed and published through the same ArcGIS Image Server site. The ArcGIS Image Server site does the work of processing analytics requests, storing the results, and returning results to members in the Portal for ArcGIS site.

A diagram of the minimum raster analysis system is shown below.

The raster analysis and output image hosting server requires an ArcGIS Image Server license.

The instructions to set up and configure the minimum raster analysis system are detailed below.

Minimum setup for a raster analysis system

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding. To configure the portal to perform image hosting services, follow the steps below.

1. Install ArcGIS Server.
2. Authorize ArcGIS Server with an ArcGIS Image Server license.
3. Log in to ArcGIS Server Manager and browse to Site > Data Stores. Register a new raster store.

   The raster store is a location for output from raster analysis and should not be used as an input data directory.

   Note:

   If there are multiple raster stores defined, the raster analysis service will randomly pick one as the output location.

4. Federate your ArcGIS Server site with your portal.
5. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
6. Click the ellipsis next to the configured server and select Configure server role.
7. On the Configure server role dialog, enable Raster Analysis.
8. Click Save.
9. Assign the minimum permissions needed to perform raster analysis.
10. Make sure you have increased the SOC maximum heap size to 128 MB for ArcGIS Image Server.