Machbase Database

Read

Database of Things 2 (Machbase Database)

Read

http://machbase.com/download

DBMS For Processing IoT Data

(Image source: https://cdn.pixabay.com/photo/2015/05/25/05/27/network-782707_1280.png)

The Problems With Handling Large Amounts Of Data In Real Time

As previously mentioned, more devices and more sensors mean you’ll start to generate incredible amounts of data. To compound the problem even further, continued storage of data for analysis and historical record means you’ll only ever have more data to deal with.

Typical database management systems (DBMS) are unsuitable and make it difficult to manage petabytes of information within a single system. Also, while conventional Big Data platforms are optimized for batch processing, distributed storage, and massive data retrieval, they do not necessarily make it possible to analyze large amounts of data in real time.

Since the utility of time series data is based on continuing to collect, store, index, and retrieve it without data loss, your DBMS must be scalable, and able to handle large amounts of sensor data in real time.

Specialized IoT DBMS make time series data management much easier by generating search indexes and processing statistical data for visualization very quickly.

Query Language And Interface

IoT sensor data can come as either regular structured data and semi-structured data. While SQL is the most common query language for structure data, there is no query language specifically used for semi-structured data. No-SQL query languages have emerged with the introduction of Big Data systems, but heterogeneous query languages (e.g. MongoDB) are not widely used or supported. As a result, SQL on Hadoop products such as Spark and Impala have become popular and encouraged increased usage of SQL. DBMS supporting the SQL language typically also support traditional interfaces, such as ODBC and JDBC.

Operating historian products use JSON based query interface through HTTP protocol through REST API. Since the REST API is very easy to use, nearly every environment has been adapted and it has become the most popular interface to support.

Transaction Processing

In a distributed data system that needs to be able to process more than 100 billion data per second in real time, it’s difficult to perform transaction processing. Transaction processing can be summarized by ACID or two-phase locking.

Traditional relational DBMS that complete ACID-based transaction processing are difficult to use due to the nature of time series data, since there is typically no data updating operation before data deletion. Typically, this data is being processed with eventual consistency techniques based on CAP theorem in Big Data platform.

In order to process massive amounts of IoT data in real-time, you require a DBMS with more efficient data processing techniques that reflect the characteristics of time series data, rather than than those used in traditional ACID-based transactions.

Statistical Processing For Time Series Data

Time-series data requires time-based statistical processing in order for it to be used for visualization or statistical analysis.

It can be difficult to generate statistical data in an environment where a large amount of data is input in real time. In fact, even general sum, count and average functions in time series statistical processing requires a special sampling function. Since it is difficult to process the time series statistical data with a traditional relational database, stream databases are being studied as a possible alternative.

Machbase DBMS Optimized For IoT Data

The DBMS for IoT data should be able to:

Process large amounts of data in real time,
Support a convenient and efficient query language,
Process effective transactions,
Process time series data statistics.

Machbase is the only available solution that meets all of these requirements.

Real-Time Processing Of Large Data Volumes

With distributed data storage and query structure, Machbase can input and index 200 million data from a single device. With addition of equipment, the performance improves and more than 10 million sensor data can be processed per second.
It has a dedicated API for high-speed data input and an index structure for high-speed index generation.
For increasing time-series data over time, Machbase allows users to add additional equipment to the cluster for better performance and space.

Efficient Query Language and Interface

Machbase supports an optimized SQL language for data processing. No-SQL products are starting to offer SQL language again.
Inverted index and related syntax are provided for efficiently retrieving semi-structured data, making it easy to search and process semi-structured data.
It also provides REST APIs as well as the SQL standard interface, ODBC/JDBC.

Efficient Transaction Processing

Optimal transaction technique for time series data
It does not provide any update but inserts and deletes are possible. Even when restarting by node fails, recovery process is performed and the consistency of data and index is maintained.
Machbase Enterprise Edition solves the data loss caused by node failures with a distributed data storage technique.

Time Series Statistics Processing

Automatic statistics function for time series sensor data: Automatically generate statistics for each sensor per time unit (second, minute, hour) and sensor identifier.
It supports extended query conditional clauses optimized for time series data.

Machbase is the solution that is implemented by considering all the functions and performance requirements for processing time series data and is suitable for processing IoT data.

“Machbase Database”

Do you want to try Machbase?

Contact the Machbase team with your questions!

info@machbase.com

Database of Things 1 (Machbase Database)

Read

http://machbase.com/download

The Internet of things is a network of physical devices--peripherals, appliances, vehicles, wearable, and other items--embedded with electronics, software, and sensors that enable these objects to connect and exchange data over the internet or, sometimes, a private network.

These devices often include sensors, digitally controlled actuators, and communication terminals especially engineered for physical, moving objects. The ability to collect, process and analyze data from these devices and derive valuable insights is the catalyst that began the new trends driving much of the modern tech industry, industry 4.0.

[bctt tweet="IoT and the ability to derive valuable insights from the physical world is driving much of the modern tech industry, industry 4.0." username="machbase@machbase.com"]

Along with these new trends, however, come new technical challenges. The operation of and interactions between thousands of physical objects generates huge volumes of data at a mind-boggling pace. As technology continues to improve and integration becomes increasingly common, more information and data is generated, collected, analyzed and turned into intelligent insights.

This article will describe the types of IoT data generated, unique characteristics of IoT data, and how to overcome the immense challenge of processing it all.

IoT Data Types And Characteristics

In the early days of the Internet of Things, the only data available was via RFID. Nowadays, the improved development of communication and computing technology offers a wider variety of data sources and types--modern sensors and actuators can record almost anything imaginable. Here are the different types of IoT data available, and their characteristics.

RFID (Radio Frequency Identification)

(Image source: https://www.flickr.com/photos/christiaancolen/21845323893)

Radio-frequency identification or RFID technology uses electromagnetic fields to automatically identify and track tagged objects or equipment.

A typical RFID setup consists of an IC chip that stores electronic information and an antenna that transmits and receives the data wirelessly via radio waves. Active RFID often features a battery or other local power source that allow them to interact with an RFID reader hundred of meters away. Unlike with barcodes, RFID chips don’t need to be lined up and scanned directly by their readers, allowing easy interaction and data processing.

Barcodes may still be the method of choice in distribution, but since RFID technology is fairly cheap, it’s beating out barcodes nearly everywhere else. RFID technology has a wide variety of uses including travel, smart devices, factory operations, warehousing, farming, and healthcare.

RFID allows the tracking and recording of positional information through time series data, providing invaluable information for logistical applications.

The problem with RFID technology is security. RFID data can be read from your devices at a distance, often without you knowing. When that data can be personal or confidential in nature, that can be a huge problem.

Log Data

(Image source: https://pixabay.com/en/code-data-programming-code-944504 )

Log data is information automatically generated by software and hardware, and it plays a very important role in managing IoT devices.

At a base level, log data records the time each log is generated. In addition, log data often also includes various environmental information, such as the ID information of IP/MAC address, the system usage/load information, and the temperature/humidity information from the input records. Since log data is unique, a special conversion process--such as parsing of log messages--is required in order for the information to be represented as a schema of relational DBMS.

Typically, log data is generated in a text form and is automatically deleted when a certain capacity is reached. Obviously, that won’t do. Since ongoing data collection is important for IoT, a different approach is required in order to collect and analyze data long-term.

Another challenge that can make log data difficult to process is that it can be recorded in various formats depending on which programs generate it.

Location And Environmental Data

(Image source: http://www.noaanews.noaa.gov/stories2006/images/now-coast-capture-03-14-2006b.jpg )

As we mentioned regarding RFID, location information from moving objects is extremely useful. Another important type of environmental information is weather data. In general, positional information is obtained by using data from global positioning systems (GPS). Since GPS information is obtained through satellites, it is difficult to obtain highly accurate positional data using that technology alone. However, using GPS in conjunction with local positioning technology makes it possible to obtain more accurate and detailed information.

The location data of non-moving equipment can also be treated as very important information. For example, the combination of environmental and location information, such as temperature, humidity, and air pressure from floating sensors in the ocean can be useful for weather forecasts and disaster alarms.

Currently, location and environmental data are being studied in combination with other technologies such as geographic information systems and mobile computing in order to discover additional use cases.

Sensor Data And Time Series Data

Nowadays, every mobile phones is equipped with numerous sensors such as cameras, GPS and acceleration sensors. There are also many sensors in equipment used in factories and public services, such as roads, railways, seaports, and airports. As more and more smart devices are connected, the opportunity to collect and analyze useful IoT data also continues to rise.

Each device and sensor has unique identifiers allowing simultaneous data recording and measurement. This special data, recorded in “Timestamp, Sensor identifier, Sensor value” format, can be sequentially stored based on the input time for data analysis later. This data is called the time series sensor data.

Through analysis, time series sensor data can offer a wealth of real-world data from IoT device sensors that can be used to solve a multitude problems previously impossible to address.

Sensor and Control Data

In time series, sensor data can be collected by actuators in real time while control signal data manages actuator recordings. Since this data keeps changing in real time, a large amount of data is generated and it can be difficult to store and analyze.

This type of data can be used in accident analysis, defective product prediction, quality improvement and production control.

Historical Data

When sensor data includes the time it is collected, it can be categorized as historical data. Since it is time-based, the volume of historical data can increase very quickly depending on the length of the data collection period.

That said, when being collected for detailed analysis, it’s possible to selectively shorten the data collection period in order to avoid larger the data amounts, which can be an issue unless you’re able to resolve within your DBMS.

“Machbase Database”

Do you want to try Machbase?

Contact the Machbase team with your questions!

info@machbase.com

Next Generation Historian 3

Read

What should Historian Products have in the future? The Next Generation Historian.

None of the current solutions provide all the functionalities and performance that are appropriate for the evolving IIoT era. Current Historian solutions lack most of the functionality and performance needed for IIoT: High-speed data entry, Real-time data retrieval & compression, and data standardization, just to name a few.CLICK TO TWEET

The following is a summary of what the next generation Historian should provide and as time goes by, which aspects of functionalities will be required on the market.

High-speed data entry

The fact that the scan rate and the number of sensor data are increased means that the number of sensor data to be input per second is huge. For some customers, more than 10 million data events are generated per second, and a high-speed storage is required to store them.

Real-time data retrieval

In past, to store simple data at high speed, the plain text files would be suitable. However, now users want to retrieve data based on time, sensor name and special conditions.

Flexible scalability

To meet next generation historian performance and storage needs, additional user and storage server or nodes are additionally needed. The need for flexible and simple scale-out solutions are critical for these big data needs.

High Availability

Flexible scalability means that a large number of servers operate physically, which inevitably means server failures will occur. In a critical environment, server failures are not allowed to affect the business operation.

Standardized data extraction and transformation

Many businesses are having to deal with vendor lock-in issues, which limits flexibility for building dynamic solutions as business needs evolve. Ideally standard interfaces and functions should be adapted throughout the industry.

Real-time data compression

Storage has a cost. As data volumes explodes this cost becomes significant. With sensor data reaching millions of time-stamps per second finding vehicles to store the data at the at the lowest possible cost is critical. High speed, real-time “data compression” solves this need.

Interworking with various analysis solutions

There are many analytical solutions in the market. Interworking with these solutions will help businesses recycle their existing assets, and thus lowering their overall costs.

Machbase – A Next Generation Historian

Machbase is an excellent solution for the massive data processing requirements of today’s IIoT industry.

Comparing Machbase to the ideal next generation historian.

High speed data entry

A major Machbase strength.
Based on patented high-speed data transfer protocol, memory and disk-optimized data conversion technology, Machbase shows remarkable input/ingestion performance.
For a typical x86 Linux server, sensor data of 20 bytes or less Machbase achieved more than 2 million record ingestions per second, and with an 8-mode environment, data input capacity of more than 10 million records per second is achieved.

Real-time data retrieval

Machbase uniquely provides a technique for creating user indexes in parallel with real-time data entry/ingestion
Machbase supports real-time and high-speed “search” of input data and returns results with industry leading performance.
Machbase’s log structure merge (LSM) index structure is optimized for big data retrieval, guaranteeing retrieval results in a few seconds even in an environment where more than a billion records are stored in one node.

Flexible Scalability and High Availability

Machbase Enterprise Edition provides a clustered environment and an easy scale-out environment
Machbase scalability enables users to increase the performance and data storage space linearly!
Multiple replication structures can be selected according to the user’s environment and high availability is provided so that even if an arbitrary node is abnormally terminated, the service can be immediately sustained.

Efficiency

Machbase has been developed in ‘C’ for all core engines and ensures ultra-efficient operation to maximize CPU and memory utilization
Compared to other engines developed with Java or similar interpreter languages, users can save up to 70% or greater CPU consumption.
Machbase is also designed to selectively adjust performance and resource utilization to suit user environments by providing various tuning parameters.

Real-time data compression

Machbase compresses all stored data and related indexes in real time.
This saves up to 90% of the data storage space and reduces I/O costs required for data loading, enabling high-speed data retrieval.

Standardized Data Extraction and Transformation.

Machbase directly supports the ANSI SQL language!
No need for users to learn another data access language and reuses all the existing database knowledge.
Machbase provides a standard SDK resulting in maximizing reusability of existing applications.

Interworking with various analysis solutions

In addition to standard development tools, Machbase provides various interfacing environments including Restful API, Python, and R.
Machbase allows users to easily interact with their own tools and maintain an infrastructure that allows for convenient analysis of large amounts of data.

#machbase #machbase database #database # machbase db # machbase dbms #sensor data #time series database #smartX

"Machbase Database"
Do you want to try Machbase?

http://machbase.com/download

Contact the Machbase team with your questions!

info@machbase.com

Next Generation Historian 2 - machbase database

Read

IoT data explosion and changes in
IIoT market

With the explosion of the Internet, the development of sensor technology and the technological advances in big data processing, the concept of Internet of Things (IoT) has started to spread rapidly since the 2000s.

In addition, the same trend has occurred in industry and the term and Industrial Internet of Things (IIoT) is once again changing the world.

This IIoT’s primary agenda is to ensure that all production components in the industrial field generate information and communicate quickly with each other, ultimately aiming at innovative, autonomous production processes. Sometimes this is referred to M2M (machine to machine) capabilities.

The evolutionary change in this industry begins with the simple fact that the performances of the sensor, the data producer and the number of sensors installed are increasing rapidly, as shown in the picture below.

As a result, the amount of sensor data generated from one factory will increase enormously, which is the core of the IIoT change.

Ironically, this explosion of underlying data brought the OT (Operational Technology) market, which refers to the industrial production process where Historian products were extremely active, and convergence of Information Technology (IT), which refers to the traditional information and data management industry. Today’s IIoT needs require better and more real-time Business Intelligence (BI) in analyzing the OT data, and thus providing greater insights in optimizing the operational performance of the industrial operation. This use case falls into the “big data” domain.

The background of this convergence is that Historian products have limited performance and functionality in the IIoT big data market where there are enormous data to handle. Inevitably, customers are looking for a solution to manage this big data from the IT domain.

Until now, it seems that OT and IT are divided into two parts and perform their respective roles. But it is expected that the products which can perform both roles simultaneously will be highly demanded from customers.

#machbase #machbase database #database # machbase db # machbase dbms #sensor data #time series database

Do you want to try Machbase?

http://machbase.com/download

Contact the Machbase team with your questions!

info@machbase.com