Skip to main content
Skip to main content
Edit this page

AzureQueue Table Engine

This engine provides an integration with Azure Blob Storage ecosystem, allowing streaming data import.

Create Table​

Engine parameters

AzureQueue parameters are the same as AzureBlobStorage table engine supports. See parameters section here.

Similar to the AzureBlobStorage table engine, users can use Azurite emulator for local Azure Storage development. Further details here.

Example

Settings​

The set of supported settings is the same as for S3Queue table engine, but without s3queue_ prefix. See full list of settings settings. To get a list of settings, configured for the table, use system.azure_queue_settings table. Available from 24.10.

Description​

SELECT is not particularly useful for streaming import (except for debugging), because each file can be imported only once. It is more practical to create real-time threads using materialized views. To do this:

  1. Use the engine to create a table for consuming from specified path in S3 and consider it a data stream.
  2. Create a table with the desired structure.
  3. Create a materialized view that converts data from the engine and puts it into a previously created table.

When the MATERIALIZED VIEW joins the engine, it starts collecting data in the background.

Example:

Virtual columns​

  • _path — Path to the file.
  • _file — Name of the file.

For more information about virtual columns see here.

Introspection​

Enable logging for the table via the table setting enable_logging_to_s3queue_log=1.

Introspection capabilities are the same as the S3Queue table engine with several distinct differences:

  1. Use the system.azure_queue for the in-memory state of the queue for server versions >= 25.1. For older versions use the system.s3queue (it would contain information for azure tables as well).
  2. Enable the system.azure_queue_log via the main ClickHouse configuration e.g.

This persistent table has the same information as system.s3queue, but for processed and failed files.

The table has the following structure:

Example: