diff --git a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md index e843cd2833..52723e53e1 100644 --- a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md +++ b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md @@ -80,7 +80,7 @@ $$$ec-restrictions-apis-kibana$$$ * {{es}} plugins, are not enabled by default for security purposes. Reach out to support if you would like to enable {{es}} plugins support on your account. * Some {{es}} plugins do not apply to {{ecloud}}. For example, you won’t ever need to change discovery, as {{ecloud}} handles how nodes discover one another. % * In {{es}} 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins {{ecloud}} might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not {{es}} itself. -% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/data-store/index-basics.md#manage-indices) if you want more detailed information or perform index management actions. +% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/data-store/index-management.md#manage-indices) if you want more detailed information or perform index management actions. ## Watcher [ec-restrictions-watcher] diff --git a/deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md b/deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md index c9999dedf0..35bb19c313 100644 --- a/deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md +++ b/deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md @@ -45,7 +45,7 @@ To customize the retention period for the different data streams, [create a new 1. In {{kib}}, go to **Index Management → Index Templates** and identify the template that applies to the data stream or indices whose retention you want to change. Refer to [Available index templates](#available-templates) for a list of the most common templates. -2. Open the template’s contextual menu and select **Clone** to [create a new template](/manage-data/data-store/index-basics.md#index-management-manage-index-templates). When cloning the template: +2. Open the template’s contextual menu and select **Clone** to [create a new template](/manage-data/data-store/index-management.md#index-management-manage-index-templates). When cloning the template: 1. Assign a higher `priority` to the new template so it takes precedence over the default template. 2. In the **Index settings** section, set `index.lifecycle.name` to the custom {{ilm-init}} policy that has the required retention settings. diff --git a/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md b/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md index 227ec43578..caaff0c573 100644 --- a/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md +++ b/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md @@ -32,7 +32,7 @@ This guide also provides tips for [restoring to another cluster](#restore-differ - You can only restore a snapshot to a running cluster with an elected [master node](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#master-node-role). The snapshot’s repository must be registered and available to the cluster. - The snapshot and cluster versions must be compatible. See [Snapshot compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility). -- Before you restore a data stream, ensure the cluster contains a [matching index template](/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md#create-ts-index-template) with data stream enabled. To check, use [Kibana’s Index Management](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) feature or the get index template API: +- Before you restore a data stream, ensure the cluster contains a [matching index template](/manage-data/use-case-use-elasticsearch-to-manage-time-series-data.md#create-ts-index-template) with data stream enabled. To check, use [Kibana’s Index Management](/manage-data/data-store/index-management.md#index-management-manage-index-templates) feature or the get index template API: ```console GET _index_template/*?filter_path=index_templates.name,index_templates.index_template.index_patterns,index_templates.index_template.data_stream diff --git a/explore-analyze/alerting/watcher/watcher-ui.md b/explore-analyze/alerting/watcher/watcher-ui.md index 2f01624d59..cf712bc91d 100644 --- a/explore-analyze/alerting/watcher/watcher-ui.md +++ b/explore-analyze/alerting/watcher/watcher-ui.md @@ -37,7 +37,7 @@ To use Watcher in {{kib}}, you must have the built-in `kibana_admin` role and ei To manage roles, go to the **Roles** management page, or use the [role APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-roles). Watches are shared between all users with the same role. ::::{note} -If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](/manage-data/data-store/index-basics.md#index-management) for detailed information. +If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](/manage-data/data-store/index-management.md) for detailed information. :::: ## Create a threshold alert [watcher-create-threshold-alert] diff --git a/manage-data/data-store.md b/manage-data/data-store.md index ccde8ba454..0e6c0f2fa4 100644 --- a/manage-data/data-store.md +++ b/manage-data/data-store.md @@ -12,10 +12,33 @@ products: [{{es}}](https://github.com/elastic/elasticsearch/) is a distributed search and analytics engine, scalable data store, and vector database built on Apache Lucene. -The documentation in this section details how {{es}} works as a _data store_ starting with the fundamental unit of storage in Elasticsearch: the index. An index is a collection of documents uniquely identified by a name or an alias. Read more in [Index basics](/manage-data/data-store/index-basics.md). -Then, learn how these documents and the fields they contain are stored and indexed in [Mapping](/manage-data/data-store/mapping.md), and how unstructured text is converted into a structured format that’s optimized for search in [Text analysis](/manage-data/data-store/text-analysis.md). +This section describes the {{es}} data storage model, from indices as the foundational unit to mappings, templates, and higher-level abstractions that define data organization and access. -You can also read more about working with {{es}} as a data store including how to use [index templates](/manage-data/data-store/templates.md) to tell {{es}} how to configure an index when it is created, how to use [aliases](/manage-data/data-store/aliases.md) to point to multiple indices, and how to use the [command line to manage data](/manage-data/data-store/manage-data-from-the-command-line.md) stored in {{es}}. +## Understand data storage -If your use case involves working with continuous streams of time series data, you can consider using a [data stream](./data-store/data-streams.md). These are optimally suited for storing append-only data. You can access the data through a single, named resource, while {{es}} stores it in a series of hidden, auto-generated backing indices. +Learn about the core storage concepts in {{es}}. + +* [](/manage-data/data-store/index-basics.md): Learn about index fundamentals, including index naming and aliases, document structure, metadata fields, and mappings. +* [](/manage-data/data-store/data-streams.md): Learn when to use data streams for timestamped and append-only time series data, like logs, events, or metrics. You work with one stream name while {{es}} manages multiple backing indices behind the scenes. +* [](/manage-data/data-store/near-real-time-search.md): Understand how {{es}} makes newly indexed data searchable within seconds of indexing. + +## Configure how data is stored + +Control how {{es}} indexes, maps, and analyzes your data. + +* [](/manage-data/data-store/mapping.md): Define how documents and their fields are stored and indexed. Choose between dynamic mapping for automatic field detection and explicit mapping for full control over field types and indexing behavior. +* [](/manage-data/data-store/text-analysis.md): Configure how unstructured text is converted into a structured format optimized for full-text search, including tokenization, normalization, and custom analyzers. +* [](/manage-data/data-store/templates.md): Define reusable index configurations including settings, mappings, and aliases that are automatically applied when new indices or data streams are created. +* [](/manage-data/data-store/aliases.md): Create named references that point to one or more indices or data streams, enabling zero-downtime reindexing and simplified query targeting. + +## Manage data + +Work with your indices and data using the {{kib}} UI or the {{es}} REST API. + +* [](/manage-data/data-store/index-management.md): Use {{kib}}'s **Index Management** page to view and manage your indices, data streams, templates, component templates, and enrich policies. +* [](/manage-data/data-store/manage-data-from-the-command-line.md): Index, update, retrieve, search, and delete documents using curl and the {{es}} REST API. + +::::{tip} +If you manage append-only timestamped data with data streams, use [Data lifecycle](/manage-data/lifecycle.md) to plan retention and performance over time. +:::: \ No newline at end of file diff --git a/manage-data/data-store/data-streams/manage-data-stream.md b/manage-data/data-store/data-streams/manage-data-stream.md index 4985d7cd8f..1db5316985 100644 --- a/manage-data/data-store/data-streams/manage-data-stream.md +++ b/manage-data/data-store/data-streams/manage-data-stream.md @@ -8,7 +8,7 @@ products: - id: cloud-serverless --- -# Manage a data stream [index-management-manage-data-streams] +# Manage a data stream in {{kib}} [index-management-manage-data-streams] Investigate your data streams and address lifecycle management needs in the **Data Streams** view. @@ -25,7 +25,8 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind * To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream’s name. From this view, you can navigate to **Discover** to further explore data within the data stream. * To view information about the stream’s backing indices, click the number in the **Indices** column. -* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. +* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. +* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is managed by a [data stream lifecycle](/manage-data/lifecycle/data-stream.md). ## Manage data streams on the Streams page [manage-data-streams-with-streams] ```{applies_to} diff --git a/manage-data/data-store/index-basics.md b/manage-data/data-store/index-basics.md index 90d4895064..0fc9ebe5a8 100644 --- a/manage-data/data-store/index-basics.md +++ b/manage-data/data-store/index-basics.md @@ -1,8 +1,6 @@ --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html - - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#view-edit-indices - - https://www.elastic.co/guide/en/serverless/current/index-management.html applies_to: stack: ga serverless: ga @@ -13,11 +11,13 @@ products: # Index basics -An index is a fundamental unit of storage in {{es}}. It is a collection of documents uniquely identified by a name or an [alias](/manage-data/data-store/aliases.md). This unique name is important because it’s used to target the index in search queries and other operations. +An _index_ is the fundamental unit of storage in {{es}}: a collection of documents identified by a unique name or an [alias](/manage-data/data-store/aliases.md). This name is used to target the index in search requests and other operations. -::::{tip} -A closely related concept is a [data stream](/manage-data/data-store/data-streams.md). This index abstraction is optimized for append-only timestamped data, and is made up of hidden, auto-generated backing indices. If you’re working with timestamped data, we recommend the [Elastic Observability](/solutions/observability/get-started.md) solution for additional tools and optimized content. -:::: +This page explains the core parts of an index (_documents_, _metadata fields_, and _mappings_) and highlights common design decisions for working with indices. + +:::::{tip} +A closely related concept is a [data stream](/manage-data/data-store/data-streams.md), which is optimized for append-only timestamped data and backed by hidden, auto-generated indices. +::::: :::{note} :applies_to: {"serverless": "ga"} @@ -26,7 +26,7 @@ In {{serverless-full}}, each project supports up to 15,000 indices. This limit h ## Index components -An index is made up of the following components. +Understanding these components helps you design indices that are easier to query, scale, and manage. ### Documents [elasticsearch-intro-documents-fields] @@ -61,279 +61,27 @@ A simple {{es}} document might look like this: ### Metadata fields [elasticsearch-intro-documents-fields-data-metadata] -An indexed document contains data and metadata. [Metadata fields](elasticsearch://reference/elasticsearch/mapping-reference/document-metadata-fields.md) are system fields that store information about the documents. In {{es}}, metadata fields are prefixed with an underscore. For example, the following fields are metadata fields: +An indexed document includes both source data and metadata. [Metadata fields](elasticsearch://reference/elasticsearch/mapping-reference/document-metadata-fields.md) are system-managed fields that describe the document and how {{es}} stores it. In {{es}}, metadata fields are prefixed with an underscore. For example: * `_index`: The name of the index where the document is stored. -* `_id`: The document’s ID. IDs must be unique per index. - +* `_id`: The document's ID. IDs must be unique per index. ### Mappings and data types [elasticsearch-intro-documents-fields-mappings] -Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how the fields in your documents are indexed. A mapping defines the [data type](elasticsearch://reference/elasticsearch/mapping-reference/field-data-types.md) for each field, how the field should be indexed, and how it should be stored. - -## Index management - -Elastic's **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](/manage-data/lifecycle/data-stream.md), [templates](/manage-data/data-store/templates.md), and [enrich policies](/manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. - -To use these features, go to the **Index management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). - -### Required permissions [index-mgm-req-permissions] -```{applies_to} -stack: ga -``` - -If you use {{es}} {{security-features}}, the following [security privileges](elasticsearch://reference/elasticsearch/security-privileges.md) are required: - -* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. -* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data. -* The `manage_index_templates` cluster privilege to manage index templates. - -To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role). - -### Manage indices - -Investigate your indices and perform operations from the **Indices** view. - -:::::{applies-switch} - -::::{applies-item} serverless: - -:::{image} /manage-data/images/serverless-index-management-indices.png -:alt: Index Management indices -:screenshot: -::: - -* To access details and perform operations on indices: - - * For a single index, click the index name to drill down into the index overview, [mappings](/manage-data/data-store/mapping.md), and [settings](elasticsearch://reference/elasticsearch/index-settings/index.md). From this view, you can navigate to **Discover** to further explore the documents in the index. - - * For multiple indices, select their checkboxes and then open the **Manage indices** menu. - - Refer to [Perform operations on indices](/manage-data/data-store/perform-index-operations.md) for details about the actions that you can run. - -* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). - -* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow) or a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps). -:::: - -::::{applies-item} stack: -:sync: stack - -:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png -:alt: Index Management UI -:screenshot: -::: - -* To access details and perform operations on indices: - - * For a single index, click the index name to drill down into the index overview, [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics. From this view, you can navigate to **Discover** to further explore the documents in the index, and you can perform operations using the **Manage index** menu. - - * For multiple indices, select their checkboxes and then open the **Manage indices** menu. - - Refer to [Perform operations on indices](/manage-data/data-store/perform-index-operations.md) for details about the actions that you can run. - -* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). - -* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow) or a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps). -:::: -::::: - -### Manage data streams - -A [data stream](/manage-data/data-store/data-streams.md) lets you store append-only time series data across multiple indices while giving you a single named resource for requests. - -Investigate your data streams and address lifecycle management needs in the **Data Streams** view. - -:::{image} /manage-data/images/serverless-management-data-stream.png -:alt: Data stream details -:screenshot: -::: - -In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a [logs data stream](/manage-data/data-store/data-streams/logs-data-stream.md). - -* To view information about the stream's backing indices, click the number in the **Indices** column. -* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time. -* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy. -* To view more information about a data stream including it's lifecycle settings, click the stream's name. - -:::{admonition} Streams -:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} - -Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for common data management tasks in {{kib}}, including tasks such as [modifying data retention](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams) values. -::: - -### Manage index templates [index-management-manage-index-templates] +Each index has a [mapping](/manage-data/data-store/mapping.md) that defines field types and indexing behavior. Mappings determine how fields are stored, queried, and aggregated. -An index template is a type of [template](/manage-data/data-store/templates.md) that tells {{es}} how to configure an index when it is created. +## Common index design decisions -Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. - -:::{image} /manage-data/images/serverless-index-management-index-templates.png -:alt: Index templates -:screenshot: -::: - -* To show details and perform operations, click the template name. -* To view more information about the component templates within an index template, click the value in the **Component templates** column. -* Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. -* To create new index templates, use the **Create template** wizard. - -#### Try it: Create an index template [_try_it_create_an_index_template] - -In this tutorial, you’ll create an index template and use it to configure two new indices. - -##### Step 1. Add a name and index pattern - -1. In the **Index Templates** view, open the **Create template** wizard. - - :::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png - :alt: Create wizard - :screenshot: - ::: - -2. In the **Name** field, enter `my-index-template`. -3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern. -4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is. - -##### Step 2. Add settings, mappings, and aliases - -When creating an index template, you can define settings, mappings, and aliases directly in the template or include them through one or more component templates. - -A [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) is a type of [template](/manage-data/data-store/templates.md) used as a building block for constructing index templates. {{kib}} displays badges indicating whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three. - -1. Add component templates to your index template. - - Component templates are optional. For this tutorial, do not add any component templates. - - :::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png - :alt: Component templates page - :screenshot: - ::: - -2. Define index settings directly in the index template. When used in conjunction with component templates, settings defined directly in the index template override any conflicting settings from the associated component templates. - - This step is optional. For this tutorial, leave this section blank. -3. Define mappings directly in the index template. When used in conjunction with component templates, these mappings override any conflicting definitions from the associated component templates. - - Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: - - :::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png - :alt: Mapped fields page - :screenshot: - ::: - - Alternatively, you can click the **Load JSON** link and define the mapping as JSON: - - ```js - { - "properties": { - "geo": { - "properties": { - "coordinates": { - "type": "geo_point" - } - } - } - } - } - ``` - - You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings. - -4. Define an alias named `my-index`: - - ```js - { - "my-index": {} - } - ``` - -5. On the review page, check the summary. If everything looks right, click **Create template**. - -##### Step 3. Create new indices - -You’re now ready to create new indices using your index template. - -1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`. - - ```console - POST /my-index-000001/_doc - { - "@timestamp": "2019-05-18T15:57:27.541Z", - "ip": "225.44.217.191", - "extension": "jpg", - "response": "200", - "geo": { - "coordinates": { - "lat": 38.53146222, - "lon": -121.7864906 - } - }, - "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg" - } - - POST /my-index-000002/_doc - { - "@timestamp": "2019-05-20T03:44:20.844Z", - "ip": "198.247.165.49", - "extension": "php", - "response": "200", - "geo": { - "coordinates": { - "lat": 37.13189556, - "lon": -76.4929875 - } - }, - "memory": 241720, - "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile" - } - ``` - -2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier. - - ```console - GET /my-index-000001,my-index-000002 - ``` - -### Manage component templates [index-management-manage-component-templates] - -Component templates are a type of [template](/manage-data/data-store/templates.md) used as reusable building blocks within index templates to configure index settings, mappings, and aliases. - -Create, edit, clone, and delete your component templates in the **Component Templates** view. - -:::{image} /manage-data/images/serverless-management-component-templates.png -:alt: Component templates -:screenshot: -::: - -* To show details and perform operations, click the template name. -* To create new component templates, use the **Create component template** wizard. - -### Manage enrich policies [manage-enrich-policies] - -An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains: - -* The policy type that determines how the policy matches the enrich data to incoming documents -* The source indices that store enrich data as documents -* The fields from the source indices used to match incoming documents -* The enrich fields containing enrich data from the source indices that you want to add to incoming documents -* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md). - -Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. - -:::{image} /manage-data/images/serverless-management-enrich-policies.png -:alt: Enrich policies -:screenshot: -::: +When working with indices, you typically make decisions that focus on: -* To show details click the policy name. -* To perform operations, click the policy name or use the buttons in the **Actions** column. -* To create new policies, use the **Create enrich policy** wizard. +* **Naming and aliases**: Use clear naming patterns for your indices and [aliases](/manage-data/data-store/aliases.md) to simplify query targets and support index changes with minimal disruption. +* **Mapping strategy**: Use dynamic mapping for speed when exploring data, and [explicit mappings](/manage-data/data-store/mapping.md) for production use cases where field control and query behavior matter. +* **Index or data stream**: Use a regular index when you need frequent updates or deletes. Use a [data stream](/manage-data/data-store/data-streams.md) for append-only timestamped data such as logs, events, and metrics. -You must execute a new enrich policy before you can use it with an enrich processor or {{esql}} query. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. +## Learn more -Check out these examples: +After learning index fundamentals, choose the management path that fits your workflow: -* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) -* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) -* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) +* [](/manage-data/data-store/perform-index-operations.md): Run common index operations from the **Manage index** menu. +* [](/manage-data/data-store/index-management.md): Navigate the full **Index Management** experience in {{kib}}. +* [](/manage-data/data-store/manage-data-from-the-command-line.md): Manage indices and documents with the {{es}} REST API. diff --git a/manage-data/data-store/index-management.md b/manage-data/data-store/index-management.md new file mode 100644 index 0000000000..93faad9ce6 --- /dev/null +++ b/manage-data/data-store/index-management.md @@ -0,0 +1,132 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html#view-edit-indices + - https://www.elastic.co/guide/en/serverless/current/index-management.html +applies_to: + stack: ga + serverless: ga +products: + - id: elasticsearch + - id: cloud-serverless +--- + +# Manage indices in {{kib}} [index-management] + +Elastic's **Index Management** features are an easy, convenient way to manage your cluster's indices, [data streams](/manage-data/data-store/data-streams.md), [templates](/manage-data/data-store/templates.md), and [enrich policies](/manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. + +To use these features, go to the **Index management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). + +## Required permissions [index-mgm-req-permissions] +```{applies_to} +stack: ga +``` + +If you use {{es}} {{security-features}}, the following [security privileges](elasticsearch://reference/elasticsearch/security-privileges.md) are required: + +* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. +* The `view_index_metadata` and `manage` index privileges to view a data stream or index's data. +* The `manage_index_templates` cluster privilege to manage index templates. + +To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role). + +## Manage indices [manage-indices] + +Investigate your indices and perform operations from the **Indices** view. + +:::::{applies-switch} + +::::{applies-item} serverless: + +:::{image} /manage-data/images/serverless-index-management-indices.png +:alt: Index Management indices +:screenshot: +::: + +* To access details and perform operations on indices: + + * For a single index, click the index name to drill down into the index overview, [mappings](/manage-data/data-store/mapping.md), and [settings](elasticsearch://reference/elasticsearch/index-settings/index.md). From this view, you can navigate to **Discover** to further explore the documents in the index. + + * For multiple indices, select their checkboxes and then open the **Manage indices** menu. + +* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). + +* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow) or a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps). +:::: + +::::{applies-item} stack: +:sync: stack + +:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png +:alt: Index Management UI +:screenshot: +::: + +* To access details and perform operations on indices: + + * For a single index, click the index name to drill down into the index overview, [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics. From this view, you can navigate to **Discover** to further explore the documents in the index, and you can perform operations using the **Manage index** menu. + + * For multiple indices, select their checkboxes and then open the **Manage indices** menu. + +* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md). + +* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow) or a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps). +:::: +::::: + + +## Manage data streams + +* [](/manage-data/data-store/data-streams/manage-data-stream.md) + +## Manage index templates [index-management-manage-index-templates] + +An index template is a type of [template](/manage-data/data-store/templates.md) that tells {{es}} how to configure an index when it is created. + +Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. + +:::{image} /manage-data/images/serverless-index-management-index-templates.png +:alt: Index templates +:screenshot: +::: + +* To show details and perform operations, click the template name. +* To view more information about the component templates within an index template, click the value in the **Component templates** column. +* Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases. +* To create new index templates, use the **Create template** wizard. + +### Try it: Create an index template [_try_it_create_an_index_template] + +* [Learn more about index templates](/manage-data/data-store/templates.md#index-templates) + +## Manage component templates [index-management-manage-component-templates] + +* [Learn more about component templates](/manage-data/data-store/templates.md#component-templates) + +## Manage enrich policies [manage-enrich-policies] + +An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains: + +* The policy type that determines how the policy matches the enrich data to incoming documents +* The source indices that store enrich data as documents +* The fields from the source indices used to match incoming documents +* The enrich fields containing enrich data from the source indices that you want to add to incoming documents +* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md). + +Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. + +:::{image} /manage-data/images/serverless-management-enrich-policies.png +:alt: Enrich policies +:screenshot: +::: + +* To show details click the policy name. +* To perform operations, click the policy name or use the buttons in the **Actions** column. +* To create new policies, use the **Create enrich policy** wizard. + +You must execute a new enrich policy before you can use it with an enrich processor or {{esql}} query. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. + +Check out these examples: + +* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) +* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) +* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) diff --git a/manage-data/data-store/templates.md b/manage-data/data-store/templates.md index 24d9950f4d..e16a357661 100644 --- a/manage-data/data-store/templates.md +++ b/manage-data/data-store/templates.md @@ -27,6 +27,7 @@ The following conditions apply to using templates: * Composable index templates take precedence over any [legacy templates](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/indices-templates-v1.html), which were deprecated in {{es}} 7.8. If no composable template matches a given index, a legacy template may still match and be applied. * If an index is created with explicit settings and also matches an index template, the settings from the [create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request take precedence over settings specified in the index template and its component templates. * Settings specified in the index template itself take precedence over the settings in its component templates. +* When you specify multiple component templates in the `composed_of` field of an index template, the component templates are merged in the order specified, which means that later component templates override earlier component templates. * If a new data stream or index matches more than one index template, the index template with the highest priority is used. * When you create an index template, be careful to avoid [naming pattern collisions](#avoid-index-pattern-collisions) with built-in {{es}} index templates. @@ -38,11 +39,139 @@ For a detailed exploration and examples of setting up composable templates, refe An **index template** is used to configure an index when it is created. [Mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and [aliases](/manage-data/data-store/aliases.md) specified in the index template are inherited by each created index. These can also be specified in the component templates that the index template is composed of. -You can create and manage index templates on the **Index management** page in {{kib}} or by using the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API. -For the {{kib}} steps and a walk-through example, refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates). +You can create and manage index templates in {{kib}} or using the {{es}} API. -Using the API, the following request creates an index template that is *composed of* the two component templates shown in the [component templates](#component-templates) example. +:::::{tab-set} +:group: template + +::::{tab-item} Kibana +:sync: kibana +To create an index template, complete the following steps: + +1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +1. In the **Index Templates** tab, select **Create template**. + + ![Create template page](/manage-data/images/elasticsearch-reference-create-template-wizard-my_template.png "") + +1. On the **Logistics** page: + 1. Specify a name for the template. + 1. Specify a pattern to match the indices you want to manage with the lifecycle policy. For example, `my-index-*`. + 1. If you're storing continuously generated, append-only data, you can opt to create [data streams](/manage-data/data-store/data-streams.md) instead of indices for more efficient storage. + + :::{note} + When you enable the data stream option, an option to set **Data retention** also becomes available. Data retention is applicable only if you're using a data stream lifecycle, which is an alternative to index lifecycle management. Refer to the [Data stream lifecycle](/manage-data/lifecycle/data-stream.md) to learn more. + ::: + + + 1. Configure any other options you'd like, including: + * The [index mode](elasticsearch://reference/elasticsearch/index-settings/time-series.md) to use for the created indices. + * The template priority, version, and any metadata. + * Whether or not to overwrite the `action.auto_create_index` cluster setting. + + Refer to the [create or update index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API documentation for details about these options. + +1. On the **Component templates** page, you can use the search and filter tools to select any [component templates](/manage-data/data-store/templates.md#component-templates) to include in the index template. The index template will inherit the settings (**S**), mappings (**M**), and aliases (**A**) defined in the component templates and apply them to indices when they're created. + +1. On the **Index settings** page, you can define your index settings directly in the index template. When used together with component templates, the mappings defined on this page will override any conflicting definitions from the associated component templates. + + 1. Optional: Add any additional [index settings](elasticsearch://reference/elasticsearch/index-settings/index.md) that should be applied to the indices as they're created. For example, you can set the number of shards and replicas for each index, as well as configure {{ilm-init}} by specifying [ILM settings](elasticsearch://reference/elasticsearch/configuration-reference/index-lifecycle-management-settings.md) to apply to the indices: + + ```js + { + "index.lifecycle.name": "my_policy", + "index.lifecycle.rollover_alias": "test-alias", + "number_of_shards": 1, + "number_of_replicas": 1 + } + ``` + +1. Optional: On the **Mappings** page, customize the fields and data types used when documents are indexed into {{es}}. When used together with component templates, these mappings override any conflicting definitions from the associated component templates. Refer to [Mapping](/manage-data/data-store/mapping.md) for details. + 1. For example, you can define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`. + + Alternatively, you can click the **Load JSON** link and define the mapping as JSON: + + ```js + { + "properties": { + "geo": { + "properties": { + "coordinates": { + "type": "geo_point" + } + } + } + } + } + ``` + +1. Optional: On the **Aliases** page, specify an [alias](/manage-data/data-store/aliases.md) for each created index. For example, you can define an alias named `my-index`: + + ```js + { + "my-index": {} + } + ``` + + Note that this isn't required when configuring ILM, which instead uses the `index.lifecycle.rollover_alias` setting to access rolling indices. + +1. On the **Review** page, confirm your selections and select **Create template**. You can check your selected options, as well as both the format of the index template that will be created and the associated API request. + +The newly created index template will be used for all new indices with names that match the specified pattern, and for each of these, the specified ILM policy will be applied. + + +:::{dropdown} Optional: Create new indices to test your template +To test your newly created template, you can create two new indices, for example `my-index-000001` and `my-index-000002`. + +1. Index the following documents, to create new indices: + + ```console + POST /my-index-000001/_doc + { + "@timestamp": "2019-05-18T15:57:27.541Z", + "ip": "225.44.217.191", + "extension": "jpg", + "response": "200", + "geo": { + "coordinates": { + "lat": 38.53146222, + "lon": -121.7864906 + } + }, + "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg" + } + + POST /my-index-000002/_doc + { + "@timestamp": "2019-05-20T03:44:20.844Z", + "ip": "198.247.165.49", + "extension": "php", + "response": "200", + "geo": { + "coordinates": { + "lat": 37.13189556, + "lon": -76.4929875 + } + }, + "memory": 241720, + "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile" + } + ``` + +1. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier. + + ```console + GET /my-index-000001,my-index-000002 + ``` +::: + +:::: + +::::{tab-item} API +:sync: api +Use the [create or update index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API to add an index template to a cluster. + +The following request creates an index template that is *composed of* the two component templates shown in the [component templates](#component-templates) example. ```console PUT _index_template/template_1 @@ -78,6 +207,9 @@ PUT _index_template/template_1 } } ``` +:::: + +::::: :::{tip} The following features can be useful when you're setting up index templates: @@ -117,9 +249,28 @@ If you use {{fleet}} or {{agent}}, assign your index templates a priority lower A **component template** is a reusable building block that defines mappings, settings, and aliases. Component templates are not applied directly to indices, but referenced by index templates and used when determining the final configuration of an index. -You can create and manage component templates on the **Index management** page in {{kib}} or by using the [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API. For the {{kib}} steps, refer to [Manage component templates](/manage-data/data-store/index-basics.md#index-management-manage-component-templates). +You can create and manage component templates on the **Index management** page in {{kib}} or by using the [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API. + +:::::{tab-set} +:group: template -Using the API, the following request creates the two component templates used in the previous index template example: +::::{tab-item} Kibana +:sync: kibana +Create, edit, clone, and delete your component templates in the **Component Templates** view. + +:::{image} /manage-data/images/serverless-management-component-templates.png +:alt: Component templates +:screenshot: +::: + +* To show details and perform operations, click the template name. +* To create new component templates, use the **Create component template** wizard. +:::: + +::::{tab-item} API +:sync: api +You can create and manage component templates using the [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API. +The following request creates the two component templates used in the previous index template example: ```console PUT _component_template/component_template1 @@ -153,3 +304,6 @@ PUT _component_template/runtime_component_template ``` 1. This component template adds a [runtime field](mapping/map-runtime-field.md) named `day_of_week` to the mappings when a new index matches the template. +:::: + +::::: diff --git a/manage-data/data-store/templates/index-template-management.md b/manage-data/data-store/templates/index-template-management.md index fb998345b0..13a01c1fcf 100644 --- a/manage-data/data-store/templates/index-template-management.md +++ b/manage-data/data-store/templates/index-template-management.md @@ -12,4 +12,4 @@ products: # Manage index templates [manage-index-templates] -This page has moved. Refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) for all of the steps to create and manage index templates in {{kib}}. +This page has moved. Refer to [Manage index templates](/manage-data/data-store/index-management.md#index-management-manage-index-templates) for all of the steps to create and manage index templates in {{kib}}. diff --git a/manage-data/ingest/transform-enrich/ingest-pipelines.md b/manage-data/ingest/transform-enrich/ingest-pipelines.md index 553c588f06..515efcd555 100644 --- a/manage-data/ingest/transform-enrich/ingest-pipelines.md +++ b/manage-data/ingest/transform-enrich/ingest-pipelines.md @@ -272,7 +272,7 @@ $$$pipeline-custom-logs-index-template$$$ 2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs--*`. - You can create this template using {{kib}}'s [**Index Management**](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). + You can create this template using {{kib}}'s [**Index Management**](/manage-data/data-store/index-management.md#index-management-manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). For example, the following request creates a template matching `logs-my_app-*`. The template uses a component template that contains the `index.default_pipeline` index setting. diff --git a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md index 7704aeba36..c5239a6d09 100644 --- a/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md +++ b/manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md @@ -50,7 +50,7 @@ You also can set up [{{beats}}](beats://reference/index.md), such as a [{{filebe ## Create an enrich policy [create-enrich-policy] -After adding enrich data to your source indices, use the [create enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-put-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to create an enrich policy. +After adding enrich data to your source indices, use the [create enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-put-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-management.md#manage-enrich-policies) to create an enrich policy. ::::{warning} Once created, you can’t update or change an enrich policy. See [Update an enrich policy](#update-enrich-policies). @@ -61,7 +61,7 @@ Once created, you can’t update or change an enrich policy. See [Update an enri ## Execute the enrich policy [execute-enrich-policy] -Once the enrich policy is created, you need to execute it using the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to create an [enrich index](data-enrichment.md#enrich-index). +Once the enrich policy is created, you need to execute it using the [execute enrich policy API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) or [Index Management in {{kib}}](/manage-data/data-store/index-management.md#manage-enrich-policies) to create an [enrich index](data-enrichment.md#enrich-index). :::{image} /manage-data/images/elasticsearch-reference-enrich-policy-index.svg :alt: enrich policy index @@ -123,7 +123,7 @@ Once created, you can’t update or change an enrich policy. Instead, you can: 1. Create and [execute](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy) a new enrich policy. 2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors or {{esql}} queries. -3. Use the [delete enrich policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-delete-policy) API or [Index Management in {{kib}}](/manage-data/data-store/index-basics.md#manage-enrich-policies) to delete the previous enrich policy. +3. Use the [delete enrich policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-delete-policy) API or [Index Management in {{kib}}](/manage-data/data-store/index-management.md#manage-enrich-policies) to delete the previous enrich policy. ## Enrich components [ingest-enrich-components] diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index add40b0cd6..52ab36ab41 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -75,7 +75,7 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics ## Create and manage {{ilm-init}} policies -You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-basics.md#index-management) UI or the [{{ilm-init}} APIs]({{es-apis}}group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: +You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-management.md) UI or the [{{ilm-init}} APIs]({{es-apis}}group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: * [Create an {{ilm}} policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) * [Check the {{ilm-init}} status of {{es}} indices and data streams](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) diff --git a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md index d2198b6a9c..92476d25c1 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md @@ -9,4 +9,4 @@ products: # Index management in Kibana [index-mgmt] -This page has moved. Refer to [Index management](/manage-data/data-store/index-basics.md#index-management) for guidance on managing indices, data streams, index templates, component templates, and enrich policies in {{kib}}. \ No newline at end of file +This page has moved. Refer to [Index management](/manage-data/data-store/index-management.md) for guidance on managing indices, data streams, index templates, component templates, and enrich policies in {{kib}}. \ No newline at end of file diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md index 2df0cd4168..f15d998efe 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md @@ -197,7 +197,7 @@ To create an index template in Kibana, complete these steps: The newly created index template will be used for all new indices with names that match the specified pattern, and for each of these, the specified ILM policy will be applied. -For more information about configuring templates in Kibana, refer to [Manage index templates](/manage-data/data-store/index-basics.md#index-management-manage-index-templates). +For more information about configuring templates in Kibana, refer to [Manage index templates](/manage-data/data-store/index-management.md#index-management-manage-index-templates). ::: :::{tab-item} API diff --git a/manage-data/toc.yml b/manage-data/toc.yml index fe9fda40d2..e32d788e58 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -71,6 +71,7 @@ toc: - file: data-store/templates/ignore-missing-component-templates.md - hidden: data-store/templates/index-template-management.md - file: data-store/aliases.md + - file: data-store/index-management.md - file: data-store/manage-data-from-the-command-line.md - file: ingest.md children: diff --git a/solutions/observability/apm/explore-data-in-elasticsearch.md b/solutions/observability/apm/explore-data-in-elasticsearch.md index dd47e0f456..245bab42ee 100644 --- a/solutions/observability/apm/explore-data-in-elasticsearch.md +++ b/solutions/observability/apm/explore-data-in-elasticsearch.md @@ -18,7 +18,7 @@ Elastic APM data is stored in [Data streams](/solutions/observability/apm/data-s The following examples enable you to interact with {{es}}'s REST API. One possible way to do this is using {{kib}}'s [{{dev-tools-app}} console](/explore-analyze/query-filter/tools/console.md). -Data streams, templates, and index-level operations can also be manged via {{kib}}'s [Index management](/manage-data/data-store/index-basics.md#index-management) UI. +Data streams, templates, and index-level operations can also be manged via {{kib}}'s [Index management](/manage-data/data-store/index-management.md) UI. To see an overview of existing data streams, run: diff --git a/solutions/observability/apm/reduce-storage.md b/solutions/observability/apm/reduce-storage.md index 586eb26e40..9b57f87ba6 100644 --- a/solutions/observability/apm/reduce-storage.md +++ b/solutions/observability/apm/reduce-storage.md @@ -75,7 +75,7 @@ POST /.ds-*-apm*/_delete_by_query ### Delete data with {{kib}} Index Management [apm-delete-data-in-kibana] -{{kib}}'s [Index management](/manage-data/data-store/index-basics.md#index-management) UI allows you to manage your cluster’s indices, data streams, index templates, and much more. +{{kib}}'s [Index management](/manage-data/data-store/index-management.md) UI allows you to manage your cluster’s indices, data streams, index templates, and much more. Open **Index Management** in the navigation menu or using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Data Streams**. Select the data streams you want to delete, and click **Delete data streams**. diff --git a/solutions/observability/synthetics/manage-data-retention.md b/solutions/observability/synthetics/manage-data-retention.md index 062e51dab0..a4c4e924ca 100644 --- a/solutions/observability/synthetics/manage-data-retention.md +++ b/solutions/observability/synthetics/manage-data-retention.md @@ -42,7 +42,7 @@ To find Synthetics data streams: :::{applies-item} stack: -1. In {{kib}}, navigate to [Index Management](/manage-data/data-store/index-basics.md#index-management). +1. In {{kib}}, navigate to [Index Management](/manage-data/data-store/index-management.md). 2. Filter the list of data streams for those containing the term `synthetics`. 1. In the UI there will be three types of browser data streams: `synthetics-browser-*`, `synthetics-browser.network-*`, and `synthetics-browser.screenshot-*`. diff --git a/troubleshoot/observability/troubleshoot-mapping-issues.md b/troubleshoot/observability/troubleshoot-mapping-issues.md index 8533abca46..225c7bc31c 100644 --- a/troubleshoot/observability/troubleshoot-mapping-issues.md +++ b/troubleshoot/observability/troubleshoot-mapping-issues.md @@ -27,7 +27,7 @@ It is necessary to stop all {{heartbeat}}/{{agent}} instances that are targeting To ensure the mapping is applied to all {{heartbeat}} data going forward, delete all the {{heartbeat}} indices that match the pattern the {{uptime-app}} will use. -There are multiple ways to achieve this. You can read about performing this using the [Index Management UI](/manage-data/data-store/index-basics.md#index-management) or with the [Delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete). +There are multiple ways to achieve this. You can read about performing this using the [Index Management UI](/manage-data/data-store/index-management.md) or with the [Delete index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete). If using {{agent}} you will want to fix any issues with custom data stream mappings. We encourage the use of {{fleet}} to eliminate this issue.