Navicat 16 came loaded with numerous improvements and features to address both database developers' and administrators' needs. With over 100 enhancements and a brand new interface, there are more ways than ever to build, manage, and maintain your databases. One of the many improvements aimed at maximizing productivity is the ability to configure multiple connection profiles. This feature is ideal for out-of-office users who may need to switch between settings based on their location or for streamlining database access. Today's blog will outline the process of creating a new connection profile and how to switch between profiles.

Connection Profiles at a Glance

Before demonstrating how to create a connection profile, let's briefly go over what they are. Simply put, Navicat connection profiles serve as predefined configurations that store all the necessary information needed to connect to a specific database. This includes details like the host address, port number, username, password, and the specific database you want to access. By creating and saving these profiles, you can quickly establish connections without manually entering these details every time.

Creating a New Connection Profile

When we create a connection to a database instance, it becomes the Main (active) Profile for the Open Connection command. However, if we open the New Connection or Edit Connection dialog we can see a button to toggle the Connection Profile pane in the lower left corner of the dialog:

There, we will see the Main Profile listed.

We can create a connection profile by clicking the New Connection Profile link at the bottom of the Connection Profile pane:

That will copy over the Main Profile details to the new profile and display a text field where we can assign it a name. We'll call it "Most Used Databases":

On the Databases tab, we'll deselect all but a few core databases:

Upon clicking the OK button, we are prompted by a dialog asking if we want to switch the Active Profile to the new one:

Let's go ahead and click "Switch". That will close the dialog and display our new Connection Profile beside the MySQL connection in the main Connections pane:

Now the Open Connection command will open our new Connection profile. We can easily confirm this because only the databases that we selected are shown in the Connections pane:

Switching Connection Profiles

Once a Connection Profile is open, we can easily switch to another via the Switch Connection Profile command. From there, we can select from any connection profile for that database instance or even create a new one!

A dialog prompt will remind us that the current connection will be closed before switching to the new one:

That will make the selected profile the Main one, so that Open Connection command will open it from now on.

Conclusion

Navicat 16's connection profiles are a powerful tool for simplifying database management tasks. By creating and organizing profiles, you can swiftly connect to databases without the need for manual entry of connection details. As you become more adept at using Navicat, you'll discover additional features and techniques that further enhance your database management experience.

Redis, the blazing-fast, in-memory data structure store, is revered for its prowess in handling key-value pairs. However, its utility extends far beyond basic key operations. In this article, we'll explore some of the most indispensable Redis commands (other than those which involve keys, since we've covered those proviously), unlocking the true potential of this powerful tool. We'll also see how to communicate directly with Redis from Navicat!

1. LPUSH and RPUSH

Redis's versatility shines through its ability to handle complex data structures. Two of the most powerful commands in this regard are LPUSH and RPUSH, which respectively add elements to the left and right ends of a list.

> LPUSH my_list "element1"(integer) 1> RPUSH my_list "element2"(integer) 2> LRANGE my_list 0 -11) "element1"2) "element2"

These commands are instrumental in scenarios where you need to manage ordered data sets.

2. LPOP and RPOP

To complement the list addition commands, Redis provides LPOP and RPOP, which respectively remove and return the first and last elements of a list.

> LPOP my_list"element1"> RPOP my_list"element2"

These commands are particularly useful when implementing queues or stacks.

3. SADD and SMEMBERS

Redis sets are collections of unique elements. SADD adds one or more members to a set, while SMEMBERS retrieves all the members of a set.

> SADD my_set "member1"(integer) 1> SADD my_set "member2"(integer) 1> SMEMBERS my_set1) "member1"2) "member2"

Sets are powerful for scenarios requiring membership testing or storing unique data.

4. ZADD and ZRANGE

Sorted sets in Redis provide an ordered collection of unique elements. ZADD adds elements with a specified score, while ZRANGE retrieves elements within a specified range.

> ZADD my_sorted_set 1 "element1"(integer) 1> ZADD my_sorted_set 2 "element2"(integer) 1> ZRANGE my_sorted_set 0 -1 WITHSCORES1) "element1"2) "1"3) "element2"4) "2"

Sorted sets are excellent for scenarios requiring ordered data retrieval.

5. HSET and HGET

Redis hashes are maps between string field names and string values. HSET sets the value of a field in a hash, while HGET retrieves the value of a field.

> HSET my_hash field1 "value1"(integer) 1> HSET my_hash field2 "value2"(integer) 1> HGET my_hash field1"value1"

Hashes are ideal for scenarios involving structured data.

6. PUBLISH and SUBSCRIBE

Redis excels not only in data storage but also in real-time messaging. The PUBLISH command allows a client to send a message to a channel, while the SUBSCRIBE command enables a client to listen to messages on a channel.

# Terminal 1> SUBSCRIBE my_channelReading messages... (press Ctrl-C to quit)1) "subscribe"2) "my_channel"3) (integer) 1# Terminal 2> PUBLISH my_channel "Hello, Redis!"(integer) 1

This feature is invaluable for building real-time applications and event-driven architectures.

7. SCAN

While not a command for direct data manipulation, the SCAN command is essential for iterating over keys in a Redis database without blocking the server. It provides a cursor-based approach to prevent overloading the system.

> SCAN 01) "0"2) 1) "my_list"   2) "my_set"   3) "my_sorted_set"   4) "my_hash"   5) "my_channel"

This command is crucial for operations involving large datasets.

Executing Commands in Navicat 16 for Redis

While you can accomplish practically everything you need to using Navicat intuitive GUI, you can issue commands directly to Redis via the Console window. It's accessible via the Tools->Console command on the main menu or the Console button on the main toolbar:

Here's some sample output produced by the SCAN command that we learned about above:

Final Thoughts on Redis Commands

Redis commands extend far beyond the key-value operations that we've explored in recent blog entries. By mastering these advanced commands for working with data structures, sets, sorted sets, hashes, and even real-time messaging, you can harness the full potential of Redis for a wide range of applications. Whether you're building a caching layer, implementing queues, or developing real-time applications, Navicat 16 for Redis provides a robust set of tools to meet your needs.

Redis, known for its blazing fast performance, is a versatile NoSQL database that excels in handling key-value pairs. While it's primarily designed for simple data structures, Redis also supports more complex data types like lists, sets, and even JSON documents. In this blog article, we'll delve into the world of JSON documents in Redis, exploring how to work with them both through the command-line interface (CLI) and with the help of Navicat 16 for Redis on macOS.

Understanding JSON in Redis

JSON (JavaScript Object Notation) is a widely used data interchange format that's both human-readable and machine-friendly. Redis introduced native support for JSON documents in version 6.0, allowing users to store, query, and manipulate JSON data efficiently.

JSON documents in Redis are stored as values associated with a specific key. This allows for easy retrieval and manipulation using Redis commands.

CLI: Interacting with JSON Documents

• Storing JSON Documents

  To store a JSON document in Redis, you can use the JSON.SET command:

  JSON.SET mykey . '{"name": "John Doe", "age": 30, "email": "john@example.com"}'

  In this example, we're storing a JSON object with a name, age, and email address under the key mykey.

• Retrieving JSON Documents

  Retrieving a JSON document is straightforward using the JSON.GET command:

  JSON.GET mykey

  This will return the JSON object associated with the key mykey.

• Updating JSON Documents

  Updating a JSON document can be done using the JSON.SET command again:

  JSON.SET mykey . '{"name": "John Doe", "age": 31, "email": "john@example.com"}'

• Querying JSON Documents

  Redis provides the JSON.GET command with a path argument for querying specific elements within a JSON document:

  JSON.GET mykey .name

  This will return the value of the name field.

• Deleting JSON Documents

  Removing a JSON document is as simple as deleting the key associated with it:

  DEL mykey

Using Navicat for Redis

While Redis CLI offers a command-line approach for working with JSON documents, using a graphical tool Navicat can significantly enhance the user experience, especially for those who prefer a more visual approach. Navicat for Redis (macOS) version 16.2.6 supports the JSON key type.

Navicat for Redis (macOS) version 16.2.6 Main Screen

• Connecting to Redis with Navicat

  • Launch Navicat and select Connection -> Redis... from the main Toolbar.
  • Fill in the connection details (Host, Port, Authentication if required).
  • Click "Save" to establish a connection.

• Navigating JSON Documents

  In Navicat, you can view and interact with Redis data in a structured manner. To work with JSON documents:

  • Locate the key containing the JSON document in the main "All Data" table.
  • Select the key and click the Editor button to view the key's value.

• Editing JSON Documents

  Navicat provides a user-friendly JSON editor. You can directly modify the JSON document and save the changes.

Final Thoughts on Working with JSON Documents in Redis

Redis' integration of JSON documents extends its capabilities beyond simple key-value pairs, opening up new possibilities for handling structured data. Whether you're a developer managing complex data structures or a data analyst querying JSON data, Redis provides a robust platform for your needs. Navicat 16 for Redis for macOS's intuitive interface will help you to navigate and manipulate JSON documents with unparalleled ease and efficiency. Its intuitive JSON editor makes Navicat an invaluable tool, particularly for those who prefer a more visual approach to database management.

Redis, short for Remote Dictionary Server, is a versatile and high-performance key-value store that has gained significant popularity in the world of databases. It is known for its exceptional speed and efficiency in handling simple data structures. In this article, we will explore what sets Redis apart from other databases and how Navicat for Redis complements it as a robust management tool.

Speed and Simplicity

Redis distinguishes itself with its remarkable speed, primarily owing to its in-memory nature. Unlike traditional databases that rely on disk storage, Redis stores data in RAM, enabling lightning-fast read and write operations. This makes Redis an ideal choice for applications that require quick data retrieval and low latency.

For example, consider a use case where a social media platform needs to retrieve user profile information. With Redis, this operation is executed almost instantaneously due to the in-memory storage, eliminating the delays associated with disk I/O operations.

Data Structures for Flexibility

One of Redis's strengths lies in its support for a wide range of data structures, each tailored for specific use cases:

• Strings: Basic key-value pairs that can store strings, integers, or floating-point numbers.
  Example:
  SET user:1 "John Doe"
• Lists: Collections of ordered elements allowing push and pop operations from both ends.
  Example:
  LPUSH mylist "item1"
• Sets: Unordered collections of unique elements, useful for tasks like counting unique items or creating tag systems.
  Example:
  SADD tags "Redis" "Database" "NoSQL"
• Hashes: Maps between string field and string values, perfect for representing objects.
  Example:
  HSET user:1 username "johndoe" email "john@example.com"

These data structures empower developers to select the most suitable structure for their specific use case, resulting in optimized performance.

Pub/Sub Messaging

Redis offers robust support for Publish/Subscribe messaging, enabling real-time communication between different parts of an application or even between different applications. This feature is invaluable in scenarios requiring instant updates or notifications.

For example, in a gaming application, Redis Pub/Sub can be employed to notify players about in-game events, such as a new message or a player joining a room.

PUBLISH game:updates "New message: Hello, world!"

Lua Scripting for Complex Operations

Redis provides the ability to execute Lua scripts, allowing developers to perform complex operations in a single command. This is particularly useful for tasks that involve multiple steps or conditional logic.

For example, suppose you need to atomically transfer funds from one account to another while ensuring consistency. This can be accomplished with Lua scripting.

local sender_balance = tonumber(redis.call('GET', KEYS[1]))local receiver_balance = tonumber(redis.call('GET', KEYS[2]))local amount = tonumber(ARGV[1])if sender_balance >= amount then    redis.call('DECRBY', KEYS[1], amount)    redis.call('INCRBY', KEYS[2], amount)    return "SUCCESS"else    return "INSUFFICIENT FUNDS"end

Navicat for Redis: A Comprehensive Management Tool

Navicat for Redis is a powerful GUI tool designed to enhance the management and interaction with Redis databases. It provides an intuitive interface for performing various tasks such as browsing, querying, and modifying data. Here are some key features that set Navicat for Redis apart:

• User-Friendly Interface: Navicat for Redis offers an intuitive and user-friendly interface, making it easy for both novice and experienced developers to navigate and interact with Redis databases.
• Visual Data Manipulation: With Navicat, users can easily view, edit, and manipulate data within Redis databases. This is particularly useful for tasks like updating values or adding new keys.
• Query Building: The tool allows users to construct and execute complex queries using a graphical interface. This can be a significant time-saver for developers who prefer a visual approach to querying.
• Data Import and Export: Navicat supports seamless data import and export operations, facilitating tasks such as migrating data between databases or creating backups.
• Task Automation: Navicat for Redis enables the scheduling and automation of routine tasks, helping to streamline database management processes.

Final Thoughts on What Sets Redis Apart from Other Databases

Redis stands out as a high-performance key-value store, thanks to its in-memory nature and versatile data structures. It excels in scenarios where speed and low latency are paramount. The addition of Navicat for Redis enhances the Redis experience by providing a user-friendly and efficient management tool. Navicat's features like visual data manipulation, query building, and task automation make it a valuable companion for developers working with Redis databases. Together, Redis and Navicat form a powerful combination for building robust and high-performing applications.

Navicat made headlines back in May of 2023 when the company introduced Navicat for Redis. Since then, the development team have added several notable enhancements, the most significant being support for the Redis JSON key type. Version 16.3 marks another milestone in the evolution of both Navicat Premium and Navicat for Redis, which adds support for Redis Cluster. Today's blog will provide a brief overview of Redis Cluster, how to connect to server instances in Navicat, as well as list a few other features that you'll find in Navicat Premium.

Redis Cluster 101

Redis Cluster is a distributed implementation of Redis, the popular in-memory data structure store. It brings high availability and scalability to Redis setups. Introduced in Redis 3.0, it has become a crucial tool for large-scale applications.

One of its key features is automatic data sharding. Redis Cluster partitions the dataset across nodes, allowing for horizontal scaling. Each node holds a specific range of hash slots. This enables the handling of larger datasets compared to a single Redis instance.

Moreover, Redis Cluster ensures high availability through a master-slave replication model. Data is replicated across nodes, providing resilience against node failures. In case of a failure, a failover mechanism promotes a replica to a master, ensuring uninterrupted access to data.

Redis Cluster prioritizes availability and partition tolerance, making it a robust choice for distributed systems. It provides a balance between scalability and fault tolerance, making it a valuable tool for applications with demanding requirements.

Connecting to Redis Cluster

The Connection dialog now contains a Type drop-down where you can choose from a Standalone database instance or one which is part of a Cluster:

Selecting the Cluster item from the drop-down causes the Role drop-down to appear directly beneath it:

It allows you to choose between the Master database or a Replica (i.e. slave).

Other New Features In Navicat Premium 16.3

Navicat Premium 16.3 introduces a few other features, including support for the MongoDB Time-Series Collection as well as support for setting MySQL descending primary key.

New in version 5.0, the MongoDB Time-Series Collection efficiently stores sequences of measurements over a period of time. Time series data is any data that is collected over time and is uniquely identified by one or more unchanging parameters. The unchanging parameters that identify your time series data is generally your data source's metadata. Compared to normal collections, storing time series data in time series collections improves query efficiency and reduces the disk usage for time series data and secondary indexes.

Meanwhile, the MySQL descending primary key utilizes an index that stores rows in a descending order. The query optimizer will choose this type of an index when a descending order is requested by the query. This index type was introduced in MySQL 8.0.

Conclusion

In today's blog, we learned about some of the exciting new features in Navicat 16.3, namely support for Redis Cluster, MongoDB Time-Series Collections and MySQL descending primary keys.

Both Navicat Premium 16.3 and Navicat for Redis 16.3 are available for a free trial of 14 days on Windows, macOS and Linux.

Redis is a powerful open-source, in-memory data structure store that is used for various purposes such as caching, session management, real-time analytics, and more. One of the fundamental data types in Redis is strings, which can hold any kind of text or binary data, up to a maximum limit of 512 megabytes. In today's blog, we'll learn how to work with strings in Redis, both using the CLI and Navicat for Redis.

Using the Command Line Interface (CLI)

Redis provides a command-line interface (CLI) that allows users to interact with the database using a set of commands. Here's how you can work with strings via the Redis CLI:

1. Set a String

To set a string in Redis, you can use the `SET` command. This command assigns a value to a key.

    SET my_key "Hello, Redis!"

In this example, we're setting the value "Hello, Redis!" to the key `my_key`.

2. Get a String

To retrieve the value of a string, you can use the `GET` command.

    GET my_key

This command will return the value associated with the key `my_key`, which is "Hello, Redis!" in this case.

3. Append to a String

The `APPEND` command is used to append a value to an existing string. If the key doesn't exist, a new key is created with the provided value.

    APPEND my_key ", How are you?"

After this operation, the value of `my_key` will be "Hello, Redis!, How are you?".

4. Get a Substring

You can retrieve a substring from a string using the `GETRANGE` command. This command takes two arguments: the key and the range (start and end indices).

    GETRANGE my_key 0 4

Executing this command will return the substring "Hello" from `my_key`.

Using Navicat for Redis

Navicat for Redis is a powerful graphical user interface (GUI) tool that provides a user-friendly environment for working with Redis databases. Here's how you can perform string operations using Navicat:

1. Connecting to Redis

After launching Navicat, start by creating a new connection to your Redis server. Provide the necessary connection details such as host, port, and authentication credentials if required.

2. Navigating to the Keys

Once connected, you'll see the list of databases on the left-hand side. Expand the database containing the key you want to work with and navigate to the "Keys" section.

3. Setting a String

To set a string, right-click on the "Keys" section, select "Add Key", and choose "String" from the dropdown menu.

Enter the desired key name and value, then click "Apply". The new key will appear in the Keys List:

4. Getting a String

To retrieve the value of a string, simply double-click on the key in the "Keys" section. Navicat will display the key details, including its value.

5. Appending to a String

Right-click on the key and select "Edit Key" from the context menu. You can then append the desired text to the existing value.

Conclusion

This blog covered how to work with strings in Redis, both using the CLI and Navicat for Redis. Working with strings in Redis is a fundamental aspect of utilizing the database. Whether you choose to use the command-line interface or a GUI tool like Navicat for Redis, understanding how to set, get, append, and manipulate strings allows you to effectively manage your data.

In the world of relational databases, joining tables on foreign keys is a common and well-understood practice. However, there are situations where you need to join tables based on non-foreign key fields. This might seem unconventional, but it can be a powerful technique when used appropriately. In this article, we will explore the concept of joining database tables on non-foreign key fields, and we'll demonstrate how to do it using Navicat.

Why Join on Non-Foreign Key Fields?

In typical database design, tables are related using foreign keys, which establish clear relationships between data. However, there are scenarios where you might need to join tables based on fields that are not explicitly marked as foreign keys. Here are some reasons why you might consider this approach:

• Data Enrichment: You may want to enrich your data by combining information from different tables based on some shared characteristics.
• Legacy Databases: In legacy databases, foreign keys may not have been established, or the schema might not follow best practices.
• Data Migration: During data migration or integration, you might need to join data from multiple sources.
• Complex Queries: Some complex analytical or reporting queries may require joining tables on non-foreign key fields.

Using Navicat for Non-Foreign Key Joins

Navicat is a powerful database client that supports various database management systems like MySQL, PostgreSQL, SQL Server, and more. It provides a user-friendly interface for designing queries, making it an excellent choice for joining tables on non-foreign key fields.

Example: Combining Customer and Order Data

Let's consider a scenario where you have two tables: Customers and Orders. Normally, these tables would be related through a CustomerID foreign key field in the Orders table. However, in this example, we want to join them based on a shared Email field, which is not a foreign key.

To join the Customers and Orders tables on the Email field, you can use a SQL query like this:

SELECT Customers.*, Orders.*FROM CustomersINNER JOIN Orders ON Customers.Email = Orders.CustomerEmail;

In this query:

• Customers.* and Orders.* select all columns from both tables.
• INNER JOIN combines rows with matching Email and CustomerEmail values.

Tips for Non-Foreign Key Joins

When joining tables on non-foreign key fields, consider the following tips:

• Data Consistency: Ensure that the non-foreign key fields you're joining on have consistent data. In our example, the Email field should be consistently formatted and not contain missing or duplicate values.
• Indexes: Consider creating indexes on the fields you're joining on. Indexes can significantly improve query performance.
• Data Types: Ensure that the data types of the fields being joined match. For example, if you're joining on an email address, both fields should have the same data type, such as VARCHAR.
• Testing: Always thoroughly test your queries to verify that the results are as expected, especially when joining tables on non-foreign key fields.

Conclusion

Joining database tables on non-foreign key fields is a flexible and powerful technique that can help you work with data in unconventional ways. Navicat provides an intuitive interface for crafting SQL queries that perform these joins, making it a valuable tool for database professionals and developers.

Remember that while joining on non-foreign key fields can be useful, it should be done thoughtfully and with attention to data quality and consistency. When used appropriately, this approach can unlock new insights and possibilities in your data analysis and reporting.

Redis is a high-performance, in-memory data store known for its speed and versatility. One of its useful features is the ability to set expiration times for keys. Expiring keys in Redis is crucial for managing data and ensuring that outdated or temporary data is automatically removed from the database. In this article, we'll explore how to expire keys in Redis using the redis-cli and Navicat for Redis as well as how this feature can be applied in various scenarios.

Setting Expiration for a Key

To set an expiration time for a key in Redis, you can use the EXPIRE or SETEX command. The EXPIRE command allows you to set an expiration time in seconds, while SETEX sets both the key's value and its expiration time in one command. Here's how to use both commands:

Using EXPIRE:

127.0.0.1:6379> SET mykey "Hello, Redis"OK127.0.0.1:6379> EXPIRE mykey 60(integer) 1

In this example, we first set the value of mykey to "Hello, Redis" using the SET command. Then, we use EXPIRE to set an expiration time of 60 seconds for mykey. After 60 seconds, the key will automatically be removed from the database.

Using SETEX:

127.0.0.1:6379> SETEX mykey 60 "Hello, Redis"OK

With SETEX, we achieve the same result in a single command by specifying the key, the expiration time (60 seconds in this case), and the value.

Checking the Time-to-Live (TTL)

To check the remaining time until a key expires, you can use the TTL command. This command returns the remaining time in seconds or -2 if the key does not exist or -1 if the key exists but has no associated expiration time (it will never expire). Here's how to use it:

127.0.0.1:6379> TTL mykey(integer) 30

In this example, we check the remaining time for mykey, which was set to expire after 60 seconds. The command returns 30, indicating that there are 30 seconds left until the key expires.

Removing Expired Keys

Redis automatically removes keys when their expiration time is reached. However, you can also manually delete keys using the DEL command. This can be useful if you want to remove a key before it expires. Here's how to use it:

127.0.0.1:6379> DEL mykey(integer) 1

In this example, we use the DEL command to remove mykey manually. After running this command, the key will no longer exist in the database.

Setting Key Expiration in Navicat

In Navicat, the data editor includes a TTL drop-down for setting a key's expiration:

Options include "No TTL", "Expire In (seconds)" and "Expire At (Local Time)". Here's how to expire a key in 60 second:

The key's expiry information will be set when the Apply button is clicked.

Common Use Cases for Expiring Keys

Expire keys in Redis can be used in various scenarios to manage data effectively:

1. Caching

Redis is often used as a caching layer. By setting short expiration times for cache keys, you can ensure that the cache remains fresh and relevant, preventing the storage of stale data.

2. Session Management

Managing user sessions in a web application becomes more manageable with Redis. Setting session data to expire after a certain period of inactivity can help free up resources and enhance security.

3. Rate Limiting

Rate limiting is a common use case for API throttling. Redis can be used to count and limit the number of requests from a client within a specific time frame by expiring rate limit keys after a predefined time.

4. Temporary Data Storage

Redis can serve as a temporary data store for background jobs or temporary data processing. Expiring keys automatically clean up data that is no longer needed, reducing manual intervention.

Conclusion

In this article, we learned how to expire keys in Redis using the redis-cli and Navicat for Redis as well as how this feature can be applied in various scenarios. Key expiration in Redis is a powerful feature that helps manage data efficiently, ensuring that outdated or temporary data is automatically removed from the database. Whether you're using Redis for caching, session management, rate limiting, or temporary data storage, the ability to set expiration times for keys can significantly improve the performance and reliability of your applications.

In the realm of database management and development, the choice of tools can greatly impact efficiency and productivity. Java-based tools have emerged as strong contenders, offering diverse capabilities for working with databases. However, when it comes to native database connectivity, the differences between tools can be quite impactful. Let's explore how Navicat's ability to connect to databases natively sets it apart from other Java-based tools in the market.

Understanding Native Database Connectivity

Native database connectivity refers to a tool's ability to communicate directly with a database using the database system's native protocol. This eliminates intermediaries or translation layers, resulting in optimized and efficient connections. Java-based tools that support native database connectivity can harness the inherent optimizations and features provided by each database system, leading to enhanced performance and smoother workflows.

The Efficiency Factor

Navicat's standout feature is its native database connectivity, which significantly enhances efficiency. When compared to some Java-based tools, the difference becomes evident. Native connectivity eliminates the need for additional translations, resulting in faster data transfer, query execution, and overall performance. This can be crucial for managing large datasets, executing complex queries, and ensuring real-time interactions with the database.

Streamlined Development Workflows

Java-based tools that lack native database connectivity might encounter bottlenecks in development workflows. These tools often require additional steps for data translation and interpretation, leading to delays in coding and testing. Navicat's native connectivity streamlines development by directly communicating with the database system, reducing wait times, and enabling agile iterations. This agility is a boon for developers seeking optimal productivity.

Accuracy in Data Manipulation

Another area where Navicat's native connectivity shines is data manipulation. Java-based tools that rely on intermediate layers might introduce inaccuracies during data transformation and visualization. Navicat's direct interaction with the database's native format ensures accurate data previews, making it an ideal choice for tasks involving data analysis, transformation, and reporting.

Security and Compatibility

Native connectivity not only enhances efficiency but also contributes to security and compatibility. Java-based tools may require additional configurations to match the authentication and authorization mechanisms of different database systems. Navicat's native connectivity adheres to these protocols, providing enhanced security and better compatibility with the latest features and updates of supported databases.

Final Thoughts

When comparing Navicat's ability to connect to databases natively with other Java-based tools, it's clear that the former offers distinct advantages. Native database connectivity propels Navicat's efficiency, development workflows, accuracy in data manipulation, and compatibility. These benefits collectively contribute to a seamless database management and development experience.

As the field of database management continues to evolve, the importance of native connectivity becomes even more pronounced. By choosing a tool like Navicat, which prioritizes native connectivity, users can harness the full potential of their database systems, optimize their workflows, and ensure secure and reliable interactions with their data.

In Redis, a Hash is a data structure that maps a string key with field-value pairs. Thus, Hashes are useful for representing basic objects and for storing groupings of counters, among other things. This article will go over some of the main commands for managing hashes, both via the redis-cli and using Navicat for Redis.

Creating and Updating a Hash

In Redis, the key is the name of the Hash and the value represents a sequence of field-name field-value entries. For instance, we could describe a vehicle object as follows:

vehicle make Toyota model Crown trim Platinum year 2023 color black

To work with Hashes, we use commands that are similar to what we use with strings, since Hash field values are strings. Case in point, the command HSET sets field in the Hash to value. If key does not exist, a new key storing a hash is created. If field already exists in the hash, it is overwritten.

HSET key field value

For each HSET command, Redis replies with an integer as follows:

• 1 if field is a new field in the hash and value was set.
• 0 if field already exists in the hash and the value was updated.

Let's create the vehicle hash described above:

HSET vehicle make "Toyota"   // 1HSET vehicle model "Crown"   // 1HSET vehicle trim "Platinum" // 1HSET vehicle year 2015       // 1HSET vehicle color "black"   // 1

Now, if we update the value of the year field to 2022, HSET returns 0:

HSET vehicle year 2022 // 0

Creating a Hash in Navicat

In the Navicat for Redis, Hash fields may be added using the built-in Editor. Clicking on the ellipsis [...] button on the right of a field opens a special Editor where you can enter individual field values:

Clicking the Apply button adds the new Hash or field.

Fetching a Hash Field's Value

We can fetch the value associated with field in a Hash using the HGET command:

HGET key field

For example, we can use it to verify that we are getting 2022 as the value of year instead of 2015:

HGET vehicle year // 2022

We can also get all hash contents (fields and values) using the HGETALL command:

HGETALL key

Let's try it out:

HGETALL vehicle/* Returns:makeToyotamodelCrowntrimPlatinumyear2022colorblack*/

HGETALL replies with an empty list when the provided key argument doesn't exist.

Deleting a Field

The HDEL command removes the specified fields from the hash stored at key. Specified fields that do not exist within this hash are ignored. HDEL returns the number of fields that were removed from the hash. If a key does not exist, it is treated as an empty hash and HDEL returns 0.

HDEL key field [field ...]

Let's use HDEL to delete the year and color fields:

HDEL vehicle year color // 2

In the Navicat Editor, we can remove a field by selecting it and clicking the Delete [-] button located under the fields list:

Conclusion

This blog article highlighted some of the main commands for managing Hashes in Redis, both via the redis-cli and using Navicat for Redis.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

Redis HashHash redis-cli Navicat for Redis Hash

Hash

Redis key Hash value

vehicle make Toyota model Crown trim Platinum year 2023 color black

Hash string Hash HSET Hash Hash Hash

HSET key field value

HSET Redis

• Hash 1
• Hash 0

vehicleHash

HSET vehicle make "Toyota"   // 1HSET vehicle model "Crown"   // 1HSET vehicle trim "Platinum" // 1HSET vehicle year 2015       // 1HSET vehicle color "black"   // 1

year 2022HSET 0

HSET vehicle year 2022 // 0

Navicat Hash

Navicat for Redis Hash [...]

Hash

Hash

HGET Hash

HGET key field

year 20222015

HGET vehicle year // 2022

HGETALL Hash

HGETALL key

HGETALL vehicle/* Returns:makeToyotamodelCrowntrimPlatinumyear2022colorblack*/

HGETALL

HDEL Hash Hash HDEL Hash HashHDEL 0

HDEL key field [field ...]

HDEL year color

HDEL vehicle year color // 2

Navicat [-]

redis-cli Navicat for Redis Redis Hash

Navicat for Redis 14

Redis supports several data types for storing collections of items. These include lists, sets, and hashes. Last week's blog article covered the List data type and highlighted some of the main commands for managing them. In today's follow-up we'll be turning our attention to the set type. In Redis, a Set is similar to a List except that it doesn't keep any specific order for its elements and each element must be unique. This article will go over some of the main commands for managing sets, both via the redis-cli and using Navicat for Redis.

Creating a Set

In Redis, we can create a Set by using the SADD command that adds the specified members to the key:

SADD key member [member ...]

As mentioned previously, each element must be unique. For that reason, specified members that are already part of the Set are ignored. If the key doesn't exist, a new Set is created and the unique specified members are added. If the key already exists or it is not a Set, an error is returned.

Here's the command to create a "vehicles" set:

SADD vehicles "Infiniti"         // 1SADD vehicles "Mazda"            // 1SADD vehicles "Ford" "Mercedes"    // 2SADD vehicles "Porsche" "Mercedes" // 1

Note that the SADD command returns the number of members that were added in that statement, not the size of the Set. We can see in the last line that only one element was added as there was already a "Mercedes" value.

Creating a Set in Navicat

In the Navicat for Redis Editor, Set values are represented as a Elements. Clicking on the ellipsis [...] button on the right of the Element opens a special Editor where you can enter individual Set elements:

Clicking the Apply button adds the new Set or element. Navicat automatically removes duplicate values.

Removing Members From a Set

We can remove members from a Set by using the SREM command:

SREM key member [member ...]

SREM vehicles "Mazda" "Mercedes" // 2SREM vehicles "Dodge" // 0

Similar to the SADD command, SREM return the number of members that were removed.

In the Navicat Editor, we can remove any Set element by selecting it and clicking the Delete [-] button located under the Element values:

Verifying That a Value Exists

To verify that a member is part of a Set, we can use the SISMEMBER command:

SISMEMBER key member

If the member is part of the Set, this command returns 1; otherwise, it returns 0:

SISMEMBER vehicles "Infiniti" // 1SISMEMBER vehicles "Alfa Romeo" // 0

Viewing a Set

To show all the members that exist in a Set, we can use the SMEMBERS command:

SMEMBERS key

Let's see what is currently contained in the vehicles Set:

SMEMBERS vehicles// returns "Infiniti", "Ford", "Porsche"

Merging Sets

We can combine Sets very easily using the SUNION command:

SUNION key [key ...]

Each argument to SUNION represents a Set that we want to merge into a larger Set. Note that any overlapping members will be removed in order to maintain element uniqueness.

Say that we had another Set named more_vehicles that contained the values "Corvette" and "Alfa Romeo". We could view all the members of both the vehicles and more_vehicles Sets as follows:

SUNION vehicles more_vehicles// "Infiniti", "Ford", "Porsche", "Corvette", "Alfa Romeo"

Conclusion

This blog article highlighted some of the main commands for managing Sets in Redis, both via the redis-cli and using Navicat for Redis.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

In a resounding testament to its commitment to excellence and innovation, Navicat has been announced as the winner of the prestigious Best DBA Solution category in the 2023 DBTA Readers' Choice Awards. (Navicat Data Modeler was also a finalist in the Best Data Modeling Solution category.) The annual awards program, hosted by Database Trends and Applications (DBTA) magazine, celebrates outstanding products and solutions in the dynamic landscape of data management and analytics. The recognition is a result of votes and opinions from DBTA's readershipcomprising data and IT professionals hailing from diverse industries.

Comprising the perfect mix of cutting-edge technology and pragmatic utility, Navicat has solidified its place as a leading provider of natively designed database management and development solutions. The award underscores Navicat's dedication to empowering data professionals with tools that streamline database operations and enhance development endeavors.

The award-winning category, Best DBA Solution, stands as a nod to Navicat's unrivaled expertise in this realm. With Navicat's flagship product, Navicat Premium, data professionals are equipped with an all-in-one database management tool that supports a multitude of database systems, including MySQL, PostgreSQL, MariaDB, MongoDB, SQL Server, Oracle, SQLite, and now Redis. The tool's hallmark user-friendly interface combined with powerful features like data visualization, data modeling, and data synchronization have secured Navicat's spot as a trusted solution for over 50% of Fortune 500 companies, effectively meeting their diverse database management needs.

Ken Lin, the CEO of Navicat, expressed his elation and honor at receiving this prestigious recognition. "We are thrilled and honored to receive this recognition from the DBTA readers," Lin remarked. "At Navicat, we are committed to providing our customers with the best possible tools for managing and developing their databases, and this award is a testament to the hard work and dedication of our team."

One of Navicat's notable features lies in its platform-specific design approach. The tools are natively designed for specific platforms, ensuring a seamless and optimized experience that aligns naturally with the system in use. This approach offers stability, usability, and an intuitive experience for efficient database management.

Navicat's commitment to continuous innovation and improvement is evident through its product range, culminating in the recent release of Navicat 16.2, which introduces Redis compatibility an enhancement poised to further elevate users' capabilities in the database management sphere.

As the data management landscape continues to evolve, Navicat's commitment to innovation and excellence positions it at the forefront of the industry, a trusted partner for professionals navigating the complexities of database management and development.

In this recent blog article, we learned about Redis' six data types. Redis Lists contain a collection of strings, sorted by the same order in which they were added. This article will expand on the List data type and highlight some of the main commands for managing them.

List Performance

In Redis, it's important to note that Lists are implemented as linked lists. A linked list is one whose nodes contain a data field as well as a "next" reference (link) to the next node in the list:

This has some important implications regarding performance. It is fast to add elements to the head and tail of the List but it's slower to search for elements within the List as we do not have indexed access to the elements (like we do in an array).

Creating a List

A List is created by using a Redis command that pushes data followed by a key name. There are two commands that we can use: RPUSH and LPUSH. If the key doesn't exist, these commands will return a new List with the passed arguments as elements. If the key already exists or it is not a List, an error is returned.

RPUSH

RPUSH inserts a new element at the end of the List (at the tail):

RPUSH key value [value ...]

Let's create a "guitars" key that represents a List of guitar brands:

RPUSH guitars "Jackson" // 1RPUSH guitars "Fender"  // 2RPUSH guitars "Gibson"  // 3

Each time we insert an element, Redis replies with the length of the List after that insertion. After the above three statements, the guitars should contain the following three elements:

Jackson Fender Gibson

LPUSH

LPUSH behaves the same as RPUSH except that it inserts the element at the front of the List (at the header):

LPUSH key value [value ...]

We can use LPUSH to insert a new value at the front of the guitars list as follows:

LPUSH guitars "Ibanez" //4

We now have four guitars, starting with "Ibanez":

Ibanez Jackson Fender Gibson

Creating a List in Navicat

In the Navicat for Redis Editor, list values are represented as Elements. Clicking on the ellipsis [...] button on the right of the Element opens a special Editor where you can enter individual list elements:

Clicking the Apply button adds the new list or element.

Once added, an element's position in the list may be changed using the up and down arrow buttons.

Fetching List Items using LRANGE

LRANGE returns a subset of the List based on a specified start and stop index:

LRANGE key start stop

We can see the full List by supplying 0 and -1 for the start and stop indexes:

LRANGE guitars 0 -1 //returns Ibanez Jackson Fender Gibson

Meanwhile, the following command retrieves the first two guitars:

LRANGE guitars 0 1 //returns Ibanez Jackson

Removing Elements from a List

LPOP removes and returns the first element of the List while RPOP removes and returns the last element of the List. Here are some examples:

LPOP guitars //returns Ibanez RPOP guitars //returns Gibson 

In the Navicat Editor, we can remove any List element by selecting it and clicking the Delete [-] button located under the Element values:

Conclusion

This blog article highlighted some of the main commands for managing Lists in Redis, both via the redis-cli and using Navicat for Redis.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

Since Redis is a key-value store that lets us associate values with a key, it does not use the Data Manipulation Language (DML) and query syntax as relational databases. So how to we write, read, update, and delete data in Redis? This tutorial will cover how to write, read, update, and delete keys using the redis-cli as well as Navicat for Redis.

Reading Data

We can use the GET command to ask Redis for the string value of a key:

GET key

Here's an example in Navicat for Redis that fetches the value for the key "auth service" shown below:

As expected, it returns its value of "auth0":

However, if we try to fetch the value for "indiana_jones_episodes", we get an error "WRONGTYPE Operation against a key holding the wrong kind of value". That's because its value is a zset. Since Redis supports 6 data types, you need to know what type of value that a key maps to, as for each data type, the command to retrieve it is different.

Here are the commands to retrieve key value(s):

• if value is of type string -> GET <key>
• if value is of type hash -> HGET or HMGET or HGETALL <key>
• if value is of type lists -> lrange <key> <start> <end>
• if value is of type sets -> smembers <key>
• if value is of type sorted sets -> ZRANGEBYSCORE <key> <min> <max>
• if value is of type stream -> xread count <count> streams <key> <ID>.

So, to retrieve the values for "indiana_jones_episodes", we can use ZRANGEBYSCORE and include the min and max arguments as follows:

That returns the first three values of the sorted set.

Writing and Updating Data

In Redis, the SET key Value command works for both setting the initial value as well as for updates.

Of course, in Navicat, both keys and values can be modified at any time using the Editor:

Deleting Data

In Redis, we can use the DEL command to delete a key, along with all of its associated values. It's syntax is:

DEL key

For example, the following command would delete the "auth service" key:

DEL "auth service"

Be careful; Redis does not ask you to confirm the operation!

In Navicat, we can delete a key by selecting it in the table and then clicking the Delete [-] button. A dialog will ask us to confirm before proceeding with the delete, in case we happened to click it by accident!

Conclusion

In this tutorial, we learned how to write, read, update, and delete keys using the redis-cli as well as Navicat for Redis. Next time, we'll learn some more commands for working with data using redis-cli commands, along with how to accomplish the same thing using Navicat.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

More than a mere database, Redis can also act as a message broker that supports both publishing and subscribing (pub/sub) operations. This blog will provide a brief overview of Redis's Pub/Sub capabilities using Navicat for Redis.

About the Publish/Subscribe Messaging Paradigm

Pub/Sub is a pattern whereby a sender (broadcaster) cannot send messages to specific receivers (subscribers). Instead, published messages are sent over channels, without any knowledge of how many (if any) subscribers are tuning in. Subscribers then sign up for one or more channels so that they only receive messages that are of interest to them. Decoupling publishers and subscribers in this way allows for greater scalability and makes it easier to manage the flow of information in a complex system.

Redis Pub/Sub provides a lightweight, fast, and scalable messaging solution that can be used for various purposes, such as real-time notifications, sending messages between microservices, or communicating between different parts of a single application.

Message Delivery in Redis

Redis employs an at-most-once message delivery system. As the name suggests, it means that a message will be delivered only once, if at all. As such, once the message is sent by the Redis server, it's never sent again. If the subscriber is unable to receive the message (for example, due to an error or a network outage) the message is forever lost. Much like catching your favorite show on the radio, if you happen to miss it, you're out of luck. If your application requires stronger delivery guarantees, you should use Redis Streams instead.

Moreover, Pub/Sub has no relation to the key space. This means that a message published on database 10 will be heard by a subscriber on database 1. If you need scoping, Redis suggests prefixing the channel name (i.e., prod_mychannel, test_mychannel).

Publishing with Navicat for Redis

In Navicat for Redis (or Navicat Premium), we can access the Pub/Sub screen via the Pub/Sub button on the main toolbar. From there, we can publish messages using the Publish Pane:

In Redis, channels are not explicitly created by the user. The channels are created automatically when either the first message is published or a client subscribes to them. To demonstrate, we'll open two connections with the same Redis server. Each connection will act as a different client. The first connection will subscribe to the "test_channel", while the second one will publish a message to the same channel. By doing so, we would expect our message to be delivered to the subscriber as soon as it's published.

To subscribe to a channel in Navicat, we simply need to click the Subscribe button. That will open the Subscribe Dialog:

There, we would enter the channel name - "test_channel" - and then click Subscribe. After the dialog closes, the channel will appear in the Channels list, along with a record of the subscribe action:

To publish a message in Navicat, we would select it in the Channels list (it is the default since we only have one channel at this point), enter out message in the Message text field, and click on Publish. At that point, we should see a notification that the message was received:

Conclusion

In this blog, we explored Redis's Pub/Sub capabilities using Navicat for Redis.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

SQL supports the use of aliases to give a table or a column a temporary name. Not only can they save on typing, but aliases can also make your queries more readable and understandable. In today's blog, we'll learn how to incorporate aliases into our queries using Navicat Premium 16.2.

Overview of SQL Aliases

As mentioned in the introduction, both table and column names may be aliased. Here is the syntax for each:

Alias Column Syntax

SELECT   column_name [AS] alias_name,  column_name AS 'Alias Name' -- for names with spacesFROM table_name;

Alias Table Syntax

SELECT column_name(s)FROM table_name [AS] alias_name;

Two points to consider regarding aliases:

• An alias is usually preceded by the AS keyword, but it is optional.
• An alias only exists for the duration of that query.

Table Aliases in Join Queries

Here's a query against the Sakila Sample Database that fetches information about all copies of a particular film:

SELECT *FROM film f   INNER JOIN inventory i ON i.film_id = f.film_idWHERE i.store_id = 1 AND f.title = "Academy Dinosaur";

In the above query, since both the film and inventory tables contain a film_id column, they must be fully qualified, i.e., prefixed by the table name. In this case, aliases may be employed to shorten the statement.

Here is the query in Navicat along with the results:

Column Aliases

In the case of column names, abbreviations are often utilized to keep column names short when designing database tables. For example:

• "so_no" for "sales order number".
• "qty" stands "quantity"

Here, column aliases may be employed to make the column contents more intuitive. Here's an example:

SELECTinv_no AS invoice_no,amount,due_date AS 'Due date',cust_no 'Customer No'FROMinvoices;

You can also assign column aliases to expressions, as seen below:

The above query selects both the current and future price of products after applying a price increase.

Limitations of Column Aliases

Since column aliases are assigned in the SELECT clause, you can only reference the aliases in the clauses that are evaluated after the SELECT clause. Hence, you cannot include aliases in the WHERE clause; doing so will result in an error:

This happens because the database evaluates the WHERE clause before the SELECT clause. Therefore, at the time it evaluates the WHERE clause, the database doesn't have the information of the NewPrice column alias.

It is however permissible to use column aliases in the ORDER BY clause because it is evaluated after the SELECT clause:

The database evaluates the clauses of the query in the following order:

FROM > SELECT > ORDER BY

Table Aliases and Navicat

In Navicat, once a table alias has been defined, it will come up in the auto-complete list.

That makes using aliases even more time saving!

Final Thoughts on Using Database Aliases

In today's blog, we learned how to incorporate aliases into our queries using Navicat Premium 16.2. Aliases are an easy way to make your queries more readable and understandable, which is important because code isn't just about execution; it's also a communication mechanism.

One of the key features that sets Redis apart from other key-value stores is its support of numerous data types, which include strings, lists, sets, sorted sets, and hashes. This makes it easier for developers to solve problems because they tend to know which data type to use for a given task. This blog will outline the six data types supported by Redis.

Strings

Redis stores strings as a sequence of bytes. Strings in Redis are binary safe, meaning they have a fixed length rather than have it determined by one or more special terminating characters. As such, you can store anything up to 512 megabytes in one string.

In Navicat for Redis, we can create a new key/value pair via Edit -> Add Key from the main menu. That will add a new empty row in the Data View and open the Editor:

We can use the Editor to set the:

• Key Name
• Key Type
• Value
• TTL (Time To Live)

Clicking the Apply button populates the new row with the contents of the Editor form. It also shows the size of the value:

Hashes

In Redis, a hash is a collection of key value pairs. As such, they are a good choice for representing objects and to store groupings of counters, among other things. Every hash can store up to 2^32 - 1 field-value pairs (that's more than 4 billion!).

For hash values, the Navicat Editor employs a table with Field and Value columns:

Lists

Redis Lists are simply lists of strings, sorted by the same order in which they were added. You can add elements to a Redis List on the head or on the tail. The max length of a list is 2^32 - 1, or 4294967295, elements (that's more than 4 billion of elements per list!).

In the Navicat Editor, list values are represented as an Element. Clicking on the ellipsis [...] button on the right of the Element opens a special Editor where you can enter the complete list:

Sets

Redis Sets are an unordered collection of strings. A Set is similar to a list, except that Set doesn't allow duplicates and doesn't preserve insertion order.

Sets can be sorted as well. In a Sorted Set, every member is associated with a score, that is used in order to take the sorted set ordered, from the smallest to the greatest score. While members remain unique, the scores may be repeated.

Navicat handles Sets much in the same way as Lists. Here's an example:

Sorted Sets are listed as "zset" in the Key Type drop-down:

Streams

The Redis stream data type was introduced in Redis 5.0. Streams model a log data structure but also implement several operations to overcome some of the limits of a typical append-only log.

Yes, Navicat for Redis supports the Stream data type!

Final Thoughts on Redis Data Types

This blog outlined the six data types supported by Redis, including the new Stream type.

Interested in giving Navicat for Redis a try. Download it here. The trial version is fully functional for 14 days.

Redis is an open source, BSD licensed, advanced key-value store, written in C. It's also referred to as a data structure server, since the keys can contain strings, hashes, lists, sets and sorted sets. This tutorial will provide the fundamentals of Redis concepts needed to start using it right away.

Why Use Redis?

Redis is certainly not the only key-value store to choose from. However, it does offer some advantages over its competitors. For instance:

• Redis supports more data types than most other key-value data stores. Developers already know most types, including list, set, sorted set, and hashes. This makes it easy to solve problems since developers tend to know which data type to use for a task.
• Redis holds its database entirely in memory, using the disk only for persistence, making it exceptionally fast. In fact, it can perform about 110,000 SETs per second, about 81,000 GETs per second!
• Redis can replicate data to any number of slaves.
• All Redis operations are atomic, which ensures that if two clients concurrently access the same data, Redis server will receive the updated value(s).
• Redis natively supports Publish/Subscribe, making it ideal for messaging-queues.
• Redis is well suited for managing any short-lived data in your application, such as web application sessions, web page hit counts, etc.

When Not to Use Redis

Of course, Redis is not without its flaws. It's not your best choice if you need to minimize the chance of data loss in case of outages, such as a sudden loss of power. You can configure multiple save points, such as every five minutes and/or 100 writes against the data set. However should Redis stop working without a proper shutdown for any reason, you should be prepared to lose the latest several minutes of data.

Another issue is that Redis often needs to fork a child process in order to persist data to disk. This can consume a lot of system resources if the dataset is large, and may result in an interruption of service for clients ranging from a few milliseconds to a full second, depending on dataset size and CPU power.

Installing Redis

How you install Redis depends on your operating system and whether you'd like to install it bundled with Redis Stack and Redis UI. The official Redis site has guides for every O/S:

• Install Redis on Linux
• Install Redis on macOS
• Install Redis on Windows
• Install Redis with Redis Stack and RedisInsight
• Install Redis from Source (requires C compiler and libc)

Once you have Redis up and running, and can connect using Navicat for Redis, and continue with the tutorial below.

Exploring the Redis CLI

Navicat for Redis includes a console, which allows you to communicate directly with a database instance:

One advantage to using the CLI in Navicat is that it provides auto-completion on every aspect of CLI commands, including command names as well as their parameters:

Conclusion

This tutorial provided the fundamentals of Redis concepts needed to start using it right away. There will be plenty more articles on Redis in the coming weeks, so be sure to check back often!

Version 16.2 of Navicat Premium added several exciting new features to an already stellar product, the most noteworthy being Redis support. Now, there is a Navicat administration and development client specifically for Redis. Navicat for Redis offers users an easy-to-access visual interface to visualize and optimize Redis data. It includes a rich set of features for making routine management tasks simpler, easier and more efficient than ever before. It can connect to any local/remote Redis server, and is compatible with cloud databases like Redis Enterprise Cloud, Amazon ElastiCache, Google Memorystore and Microsoft Azure. This blog will outline some of Navicat for Redis's most important features.

Data Viewer

Beyond a simple key/value store, Redis is actually a data structures server with support for many different kinds of values. Viewing such complex data structures can be a challenge, but not with Navicat for Redis's data viewer; you can use it to view, edit, search and sort keys and data via the classic spreadsheet-like Grid View, Tree View, all with a built-in editor. Navicat provides you with the tools you need to manage your data in both a smooth and efficient manner.

Data view in Windows:

Query Editing

As with other Navicat products, Navicat for Redis lets you create, edit and run queries within the Query Editor, all without having to worry about syntax and proper usage of commands. It helps you to code quickly thanks to Code Completion, which gives you suggestions for keywords and reduces the repetition in coding.

Query Editor:

You can also save queries and commands as Snippets that you can reuse over and over again.

Pub/Sub

The Pub/Sub Tool allows you to send messages and subscribe to specific channels using a simple and intuitive UI. You can save the channels as a profile, or assign colors to the channels to easily distinguish the corresponding channels and their messages. You can choose from active channels, custom channels, and custom patterns.

Pub/Sub Tool:

Collaboration

Navicat for Redis includes Navicat Cloud, which allows you to synchronize your connection settings, queries, snippets and virtual group information to the cloud. The Navicat Cloud service provides real-time access to these, and share them with your coworkers anytime and anywhere.

Navicat Cloud:

Dark Mode

The dark theme helps protect your eyes from the brightness of the standard "white" computer themes. Most importantly, dark mode only affects how application screens look, and does not alter their behavior in any way.

Redis data in dark mode

User Management

Users and their associated permissions can be managed using an intuitive interface. There, you can create, edit and delete users in minutes without having to type commands, as well as easily create new privilege groups to apply multiple sets of rules to a user all at once.

User management screen:

Conclusion

This blog outlined just some of Navicat for Redis's most important features. There are plenty of others, including Backup/Restore facilities, Automation, Secure Connections, and more.Cross-platform licensing is now available. Whether you're a Windows, macOS, or Linux user, you can purchase once and select a platform to activate and transfer your license at a later date if needed.

During the process of normalization, groups of fields that represent a distinct entity are removed from a larger and/or more central table to a separate one. Common fields (usually IDs) are then employed to maintain their relationship. We can see an example below:

In relational database, referential integrity between tables is enforced using foreign key constraints.

This blog will cover how foreign keys work as well as how to create a foreign key constraint in MySQL using Navicat 16 for MySQL .

About the Film and Inventory Relation

The model that we saw in the intro depicts a one-to-many relationship between the film and inventory tables whereby a film entity (1 row) may link to zero or more entities (rows) in the inventory table.

The film table is called the parent table or referenced table, and the inventory table is known as the child table or referencing table. As such, the foreign key columns of the child table often refer to the primary key columns of the parent table.

In this example, we are only focusing on one relation. In fact, a table can have more than one foreign key where each foreign key references to a primary key of the different parent tables.

Once a foreign key constraint is in place, the foreign key columns from the child table must have the corresponding row in the parent key columns of the parent table or values in these foreign key columns must be NULL. For example, each row in the inventory table has a film_id that exists in the film_id column of the film table. Multiple rows in the inventory table can have the same film_id.

In the next section we'll create a Foreign Key Constraint for this relationship in Navicat 16 for MySQL.

Creating a Foreign Key Constraint in Navicat

In Navicat, you'll find Foreign Key Constraints on the Foreign Keys tab of the Table Designer. To create a new Foreign Key Constraint, open the parent table - in this case film - in the Table Designer and click the Add Foreign Key button. That will create a new row in the Foreign Keys list:

Next, select the "film" table from the Fields drop-down, the "inventory" table from the Referenced Table drop-down and the "film_id" for the Referenced Fields:

The next step is to choose the On Delete and On Update actions. MySQL supports five different referential options, as follows:

• CASCADE: It is used when we delete or update any row from the parent table, the values of the matching rows in the child table will be deleted or updated automatically.
• SET NULL: It is used when we delete or update any row from the parent table, the values of the foreign key columns in the child table are set to NULL.
• RESTRICT: It is used when we delete or update any row from the parent table that has a matching row in the reference(child) table, MySQL does not allow to delete or update rows in the parent table.
• NO ACTION: It is similar to RESTRICT. But it has one difference that it checks referential integrity after trying to modify the table.
• SET DEFAULT: The MySQL parser recognizes this action. However, the InnoDB and NDB tables both rejected this action.

Let's follow the example of the existing FK and choose an On Delete action of RESTRICT and an On Update action of CASCADE:

Finally, click the Save button to create the new Foreign Key Constraint. Note that Navicat will create the name for you if you do not populate the Name field.

Conclusion

Foreign Keys play an essential role in maintain referential integrity between tables. As such, one should be created for every table relationship. Navicat 16 for MySQL makes it quite easy to manage your Foreign Key Constraints without having to write any SQL commands.

As part of the process of normalizing database tables, redundant columns are extracted from higher level tables into separate subsidiary ones. This often occurs due to some fields having a one to many relationship with the parent entity. For example, take the following model that was generated using Navicat Data Modeler:

Appraisals were initially part of the ups table, but this led to data redundancy because there can be multiple vehicles appraised in one visit. Therefore, it made sense to remove the vehicle fields from the ups table and place them in their own table.

The drawback to normalization to third normal form (3NF) is that you wind up with a lot of ID fields in the main table. As a database practitioner looking at a table, it becomes very challenging to know what entity each ID column points to. As an illustration, take a look at the ups table from the above model diagram, and notice how the CSRs, customers, and vehicles have all been reduced to numeric IDs that don't help identify the underlying entities in any way:

This is partially related to the use of auto-incrementing IDs as well as normalization, but, in any event, we can make the data much easier to read by creating a view. A database view is a subset of a database and is based on a query that runs on one or more database tables. Database views are saved in the database as named queries and can be used to save frequently used, complex queries.

In Navicat 16, we can create a new view by choosing File -> New -> View... from the main menu:

That will add a new view tab.

The next step is to add the SQL statement that will generate the view fields:

If you need any help in writing your statement, there are Preview, Explain, View Builder, and Beautify SQL buttons on the tab toolbar.

Let's say we don't want to wait for the view to be created before viewing the results, we can click the Preview button to see it now:

Now the ID columns contain more descriptive - and meaningful - textual data .

Under the tab buttons, there are three more tabs - Definition, Advanced, and SQL Preview. The Advanced tab contains additional options such as the Algorithm, Definer, Security, and Check option, while the SQL Preview shows the generated CREATE VIEW statement:

The new view is named `Untitled` until we save it. At that point, a dialog appears in which we can specify the View Name:

Upon saving, the new view will be added to the Navigation Pane on the left-hand side and may be summoned at any time:

Final Thoughts on Creating Views in Navicat 16

In today's blog, we learned about database views and went through the process of making one to help identify records in a table that links to a number of dependent tables via ID fields.

Whereas most database systems employ locks for concurrency control, PostgreSQL does things a little differently: it maintains data consistency by using a multi-version model, otherwise known as Multi-Version Concurrency Control, or MVCC for short. As a result, when querying a database, each transaction sees a snapshot of data as it was some time before, regardless of the current state of the underlying data. This prevents the transaction from viewing inconsistent data that could be caused by other concurrent transaction updates on the same data, and provides transaction isolation for each database session. This blog article will provide a brief overview of how the MVCC protocol works as well as cover some of the pros and cons of the MVCC approach.

The MVCC Protocol Explained

The main difference between lock models and MVCC is that the latter ensures that reading never blocks writing and writing never blocks reading.

In MVCC, every transaction has a transaction-timestamp that indicates (a) when it started and (b) when a transaction updates a certain data-item, such as a field, or a record, or a table. A new version of this data-item is created while also retaining the older version. Each version is provided with:

• a write-timestamp to indicate the timestamp of the transaction (i.e. the time the transaction started) that created it and
• a read-timestamp to indicate the latest timestamp of all the transactions that have read it.

The basic idea of the MVCC protocol is that the transaction manager only allows operations if they can be allowed in a manner that is consistent with all transactions executing in their entirety at the moment of their timestamp. This is referred to as the presumed execution order. Database researcher Jan Hidders explains how the transaction manager accomplishes this as follows:

If a transaction wants to read an item, it is given access to the version that it would have read in the presumed execution order. This will be the one with the latest write-timestamp just before its own timestamp. For example, if there are versions with write-timestamps 5, 12 and 20, and the timestamp of the transaction is 14, then the version with write-timestamp 12 is the one read by this transaction in the presumed execution order.

If a transaction wants to write an item, it is checked if there is not a read operation that was allowed earlier and that in the presumed execution order would read the new version caused by the requested write operation, but when it was allowed read another version. For example, assume again we have versions with write-timestamps 5, 10 and 16. Moreover assume the read-timestamps of these versions are 8, 12 and 20, respectively. If a transaction with timestamp 11 wants to update the item, there is a problem, because the version with write-timestamp 10 was read by a transaction with timestamp 12. So, if a version with timestamp 11 is created, the transaction with timestamp 12 would in the presumed execution order not have seen the version created by the transaction with timestamp 10, but the one that is now about to be created with timestamp 11. If, on other hand, a transaction with timestamp 14 wants to write the item, this is fine, since as far as we know after t=12 in the presumed execution order the item was not read by any transaction until the moment it was updated at t=16.

Pros and Cons of MVCC

Pros:

• As you can tell from the description above, all read operations will always be allowed immediately. This is usually not the case in a lock-based approach, where read-locks might be refused because of existing write-locks.
• It tends to allow also more write operations to go through immediately than lock-based approaches usually do.

Cons

• If a write operation is refused, there is no alternative but rolling back or restarting the transaction: once the update is refused it will also be refused if we retry it later. This differs from lock-based approaches, where we usually can wait until the lock becomes available. It is for this reason that MVCC is categorised as an optimistic protocol: it is very efficient if there are no conflicts, but once there is one you may have to undo a lot of work.
• The many versions of an item might require significantly more storage space. In lock-based approaches only the one version needs to be stored.
• The removal of versions that are no longer needed can cause some overhead.

Final Thoughts on Multi-Version Concurrency Control in PostgreSQL

This blog article provided an overview of how the MVCC protocol works and presented a few of its pros and cons.

Interested in working with PostgreSQL? You can try Navicat 16 for PostgreSQL for FREE for 14 days!

At the top of Navicat Monitor 3's Query Analyzer screen, there's a chart that shows queries with the longest wait times:

It's essential to identify laggard queries because they can bring everything crashing to a crawl.

Besides fixing a slow query once it's been identified, another strategy might include limiting query execution times across the board. In professional grade databases such as PostgreSQL, there are settings to cap query execution time for the entire database or even per user, via the statement_timeout variable. In this blog, we'll learn how to work with this important database variable in Navicat 16 For PostgreSQL.

Setting the statement_timeout Variable at the Database Level

Setting a default statement timeout for your database is an excellent starting point. This ensures that any application or person connecting to the database will not have queries running longer than that. A sane default would be either 30 or 60 seconds, but you can go higher if you wish. Here a statement that sets a value of 60 seconds:

ALTER DATABASE mydatabase SET statement_timeout = '60s';

In Navicat 16 For PostgreSQL we can view the statement_timeout via Tools > Server Monitor > PostgreSQL from the main menu. You'll find it on the Variables tab:

In fact, you may want to employ the Find tool in order to pinpoint the statement_timeout variable, as there are many! You can click the Highlight All toggle button to better help identify the variable once matched.

Of course the Show statement works as well:

Setting a Query Timeout for a Specific User

For even more fine grained control, we can set a query timeout value for a specific user (you know, the one who always selects the entire database!). This is achieved using the ALTER ROLE statement, which can set many database variables, including statement_timeout.

To try it out, let's create a new user role called "guest":

Now we can use the ALTER ROLE statement to limit query execution time as follows:

ALTER ROLE guest SET statement_timeout='5min';

We can query the pg_roles table to obtain information about the statement_timeout (including how it was set):

The rolconfig value is an array, so we can unnest it to get one setting per row:

Final Thoughts on Setting Query Timeouts in PostgreSQL

It's crucial to be able to identify laggard queries because they can bring your database performance down to a crawl. For that, there's Navicat Monitor 3's Long Running Queries chart at the top of the Query Analyzer screen.

Another approach is to limit how long a query can execute before it times out. As we saw in today's blog, in PostgreSQL, this can be done at the database, session, and even at the individual role level.

If you haven't already setup your statement_timeout variable(s), I would encourage you to do so ASAP. This is just one component of proper database tuning that will help to ensure your database instance stays healthy and available.

Interested in giving Navicat 16 For PostgreSQL a try? You can download the fully functioning application here to get a free 14 day trial!

The idea behind database auditing is to know who accessed your database tables and when, along with what modifications were made to them. It's not only considered to be the standard minimum requirement for any enterprise level application, but is also a legal requirement for many domains such as banking and cybersecurity. Database Audit Trails are essential in investigating all sorts of application issues such as unauthorized access, problematic configuration changes, and many more.

In today's blog, we're going to add logging to the MySQL Sakila Sample Database to audit the rental table. It's a key table because the database represents the business processes of a DVD rental store.

Creating a Table to Store Audit Trail Data

Ideally it's best to have an audit table for each table being audited. Here's the DDL statement to create the audit trail table for the rental table:

create table rental_audit_log( id int NOT NULL AUTO_INCREMENT, rental_id int NOT NULL, old_values varchar(255), new_values varchar(255), done_by varchar(255) NOT NULL, done_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id));

Alternatively, in Navicat, you can use the Table Designer to specify all of the fields and attributes without having to write a DDL statement:

Creating the Audit Logging Triggers

We'll need to create 3 database triggers to insert records in the rental_audit_log table, one for each type of DML statement performed on the rental table (INSERT, UPDATE, and DELETE).

AFTER INSERT Trigger

INSERT statements on the rental table will be intercepted by the rental_insert_audit_trigger. We'll get it to fire AFTER Insert operations and provide all of the new data as a JSON_OBJECT. In Navicat, all of those details may be supplied on the Triggers tab of the Table Designer:

After adding a new row to the rental table, we can see a new record in the rental_audit_log as well:

AFTER UPDATE Trigger

UPDATE statements on the rental table will be captured by the following rental_update_audit_trigger:

Now, every time a rental record is updated, the rental_update_audit_trigger is executed, and a rental_audit_log row will be created to capture both the old and the new state of the modified record. In this case, we can see that user robg changed the rental_date from "2005-05-25 17:17:04" to "2005-05-31 19:47:04":

AFTER DELETE Trigger

To track DELETE statements on the rental table, we will create the following rental_delete_audit_trigger:

In this case, only the old_values column is set since there is no new record state. Hence the empty new_values column in the generated rental_audit_log row:

Here, we can see that user fsmith deleted record 1114 from the rental table on 2023-03-22 at 08:46:07.

Final Thoughts on Audit Trail Logging Using Triggers

In today's blog, we added logging to the MySQL Sakila Sample Database to audit the rental table. Our logging table included some of the most common audit fields. Some organizations include others, such as the DML operation type, while others only include changed fields. It's really whatever works best for the organization.

Once again, Navicat has been nominated for the prestigious Database Trends and Applications (DBTA) Readers Choice Awards in the following categories:

• Best DBA Solution (Navicat)
• Best Data Modeling Solution (Navicat Data Modeler)
• Best Database Development Solution (Navicat for MySQL)
• Best Database Performance Solution (Navicat Monitor)

Right now is the time to vote for Navicat as the voting period will only be open until Wednesday May 10, 2023.

Winners will be showcased in a special section on the DBTA website and in the August 2023 issue of Database Trends and Applications magazine.

More on Navicat and Nominated Products

Navicat is owned by PremiumSoft. Founded in 1999, PremiumSoft has developed a wide variety of applications for Windows, macOS, Linux and iOS. Navicat is the choice of over 5 million database users all around the world. Over 180,000 registered customers across 7 continents and 138 countries have chosen our products. More than 50% of the Fortune 500 rely on Navicat every day. Some notable customers include Apple Inc., Google Inc., Oracle, Intel, Microsoft, Fujitsu, Accenture, HP, IBM, Ebay, Samsung, Sony, JP Morgan, KPMG, Barclays, DHL, Federal Express, General Electric, and many more.

Navicat Data Modeler 3 is a powerful and cost-effective database design tool which helps you build high-quality conceptual, logical and physical data models. It supports various database systems, including MySQL, MariaDB, Oracle, SQL Server, PostgreSQL, and SQLite.

Navicat 16 for MySQL is the ideal solution for MySQL/MariaDB administration and development.

Navicat Monitor 3 is a safe, simple and agentless remote server monitoring tool that is packed with powerful features to make your monitoring effective as possible. Monitored servers include MySQL, MariaDB, PostgreSQL and SQL Server.

Winner in 2022!

All of the votes from satisfied Navicat customers helped Navicat take the BEST DATABASE DEVELOPMENT SOLUTION category for Navicat for MySQL. In doing so, Navicat beat Quest Toad for Oracle and Devart dbForge Studio.

Navicat Premium also was a finalist for the BEST DBA SOLUTION award, along with Devart dbForge Studio.

About Database Trends and Applications

Database Trends and Applications is a magazine that covers data and information management, big data, and data science. In addition, their website, dbta.com, provides white papers, webinars, and other offerings for learning in the field. DBTA also circulates newsletters that connect subscribers with news and analysis about a diverse range of subjects such as Oracle News, Linux News, MultiValue News, General Information Management News, and more.

Where to Vote

You can cast your vote for your favorite Navicat product directly on the DBTA website. To make your selections:

• Locate the desired category on the page, i.e., Best DBA Solution and select Navicat from the list of candidates:

  

• Once you've made your selections for the categories which you are voting for, enter your name, company, and email at the bottom of the page in the Complete Your Vote section and click the Submit button:

  

You should see the form replaced with a "Thanks for voting!" message confirming a successful form submission.

Now it's Your Turn to Vote!

Don't forget to cast your votes by Wednesday May 10, 2023. Again, the categories and nominated Navicat products are:

• Best DBA Solution (Navicat)
• Best Data Modeling Solution (Navicat Data Modeler)
• Best Database Development Solution (Navicat for MySQL)
• Best Database Performance Solution (Navicat Monitor)

A table column, such as one that stores first names, may contain many duplicate values. If you're interested in listing the different (distinct) values, there needs to be a way to do so without resorting to complex SQL statements. In ANSI SQL compliant databases like PostgreSQL, SQL Server, and MySQL, the way to select only the distinct values from a column is to use the SQL DISTINCT clause. It removes duplicates from the result set of a SELECT statement, leaving only unique values. In this blog article, we'll learn how to use it.

Syntax and Behavior

To use the SQL DISTINCT clause, all you need to do is insert the DISTINCT keyword between in SELECT and columns and/or expressions list like so:

SELECT DISTINCT columns/expressionsFROM tables[WHERE conditions];

You may include one or more columns and/or expressions in your statement, as the query uses the combination of values in all specified columns in the SELECT list to evaluate their uniqueness. Also, if you apply the DISTINCT clause to a column that has NULL values, the DISTINCT clause will keep only one NULL and eliminate the others. In other words, the DISTINCT clause treats all NULL values as the same value.

One Column Example

A common use case for a query is to list all of the cities and/or countries of an organization's customers or users. Here's a query in Navicat Premium 16 against the classicmodels sample database:

As highlighted with the red outline, there are duplicate cities.

To get a list of unique cities, we can add the DISTINCT keyword to the SELECT statement:

We can utilize Navicat's code-completion feature to bring up the DISTINCT keyword. Navicat displays information in drop-down lists as you type your SQL statement in the editor, it assists you with statement completion and the available properties of database objects, for example databases, tables, fields, views etc with their appropriate icons:

Multiple Column Example

The DISTINCT keyword may also be applied to multiple columns. In that context, the query will only return rows where all of the selected columns are unique. First, let's add the country field to our last query:

Once again, we see duplicates, which makes sense because a duplicated city will likely reside in the same country.

Once again, adding the DISTINCT keyword will cause the query engine to look at the combination of values in both city and country columns to evaluate and remove the duplicates:

DISTINCT with Null Values

As mentioned above, the DISTINCT clause treats all NULL values as the same value so that only one instance of NULL is included in the result set. We can test that out for ourselves by querying a column such as this on in the same customers table that we queried previously:

As predicted, adding the DISTINCT keyword removed all but one instance of NULLs :

Final Thoughts on Selecting Distinct Values From a Relational Database

In this blog article, we learned how to use the SQL DISTINCT clause, which removes duplicates from the result set of a SELECT statement, leaving only unique values. As we saw, it can work on one or more columns as well as NULL values. However, should you need apply an aggregate function on one or more columns, you should use the GROUP BY clause instead.

Stored Procedures, Functions, and Views

Welcome to the 3rd and final installment on SQL naming conventions. In Part 1, we covered the rules for naming tables, while Part 2 explored conventions for column names. This installment will offer some guidelines for naming other database objects such as Stored Procedures, Functions, and Views.

Stored procedures

A stored procedure is a set of statement(s) that perform some defined actions. Typically, they contain statements that are used frequently. Stored procedures are similar to functions in programming in that they can accept parameters, and perform operations when we call them.

General Format

Most DBAs like to give their stored procedures a prefix that identifies it as such, followed by the action that the stored procedure takes and then the name representing the object or objects it will affect:

[prefix]_[action]_[object]

Actions that you may take with a stored procedure include:

• Insert
• Delete
• Update
• Select
• Get
• Validate

Choosing a Prefix

The most obvious prefix to use on a stored procedure is "sp_". That being said, there's at least one good reason to avoid it as it's already used by SQL Server as a standard naming convention in the master database. If you do not specify the database where the object is, SQL Server will first search the master database to see if the object exists there and then it will search the user database. Even if you don't host your database(s) on SQL Server, you should probably avoid using this as a naming convention, just in case you ever switch.

Instead, consider a prefix like "usp_" instead.

Putting It All Together

Here are a few examples of well-named stored procedures to formulate your own:

• usp_insert_person
• usp_delete_person
• usp_update_person
• usp_select_person
• usp_get_person
• usp_validate_person

User-defined Functions

Similar to built-in database functions, a user-defined function accepts only input parameters and contains a set of SQL statements that perform actions and return the result, which can be either a single value or a table.

The naming convention a for user-defined function is to have an "fn_" prefix, followed by its action. Hence, the syntax should be very similar to that of stored procedures:

[prefix]_[action]_[object]

Functions that return true or false may follow the rule of using "is" or "are" as the action (verb).

Some examples of function names would include:

• fn_count_string_instances
• fn_get_customer_balance
• fn_is_inventory_in_stock
• fn_get_column_type

Views

A view is a "virtual table" in a database that is defined by a query. A view can combine data from two or more table, using joins, and also just contain a subset of information. This makes them convenient to abstract, or hide, complicated queries.

The naming conventions for a view should have a "v_" or "vw_" prefix, followed by the namespace, results. As such, the syntax should be:

[prefix]_[result]

Here are a few examples:

• vw_actor_full_name
• vw_sales_by_store
• v_staff_list
• v_sales_by_product_category

Final Thoughts on Naming Conventions for Stored Procedures, Functions, and Views

In this three part series, we explored some commonly used naming conventions and considered how best to formulate our own. Part 1 covered Table names, while Part 2 focused on column names. Finally, Part 3 addressed Naming Conventions for other database objects such as Procedures, Functions, and Views.

Remember that you need not apply rules to all database objects. You could choose to apply naming convention rules to tables and column names only. It's really your decision, as using a naming convention is not mandatory, but beneficial nonetheless.

Column Names

Welcome to the 2nd installment on SQL naming conventions. As mentioned in part 1, naming conventions are a set of rules (written or unwritten) that should be utilized in order to increase the readability of the data model. These may be applied to just about anything inside the database, including tables, columns, primary and foreign keys, stored procedures, functions, views, etc. Having covered the rules for naming tables in part 1, we'll be looking at column names in this installment. Other database objects such as Procedures, Functions, and Views will be explored in part 3.

The Primary Key Column

The primary key is a field or a combination of fields in a table that uniquely identify the records in the table. A table can have only one primary key. As such, many DBAs prefer to simply name this column "id". Other's append the "_id" suffix to the table name, as seen here in the Sakila Sample Database:

Likewise, you should also assign your PK constraint a meaningful name. The naming convention for a primary key constraint is that it should have a "pk_" prefix, followed by the table name, i.e. "pk_<table_name>".

Foreign Key Columns

A foreign key is a field in the table that references a primary key in other tables. A good rule to follow is to use the referenced table name and "_id", e.g. customer_id, employee_id. This will help us identify the field as a foreign key column and also point us to the referenced table.

Here's a city table that contains a foreign key to the country table's country_id field:

The naming convention for a foreign key constraint is to have an "fk_" prefix, followed by the target table name, followed by the source table name. Hence, the syntax should be "fk_<target_table>_<source_table>".

Following the foreign key constraint naming convention for the city table would give us the name "fk_city_country":

Data Columns

In the section on Describing Real-World Entities in Part 1, it states:

Any time that you're naming entities that represent real-world things, you should use their proper nouns. These would apply to tables like employee, customer, city, country, etc. Usually, a single word should exactly describes what is in that table.

The same rules can and should be applied to data columns. Again, you should use the least possible words to describe what is stored in that column, e.g., country_name, country_code, customer_name. If two tables will have columns with the same name, you could add something to keep the name unique, although that's not strictly necessary as table prefixing will differentiate the columns in queries. Nonetheless, having unique names for each column is helpful because it reduces the chance to later mix these two columns while writing queries. Names like customer_name city_name are likely to come up in more than one table. If that concerns you, you can always make the names more descriptive, such as order_customer_name or city_of_residence_name.

Dates

For dates, it's good practice to describe what the date represents. Names like start_date and end_date are pretty common and generic. You can describe them more precisely by using names like call_start_date and call_end_date.

Final Thoughts on Naming Conventions for Column Names

You probably noticed from all of the examples presented that both table and column names should be in lowercase with words separated by an underscore ("_"). For example, customer_name and invoice_date as opposed to customerName and invoiceDate. This works well with the SQL style convention of capitalizing statements names, clauses, and other keywords for better code readability, e.g. SELECT customer_name, invoice_date FROM orders;

Navicat Monitor 3 added support for PostgreSQL, one of the most popular modern relational databases in use today. New features include an SQL Profiler for PostgreSQL instances as well as enhanced Query Analyzer and Long Running Queries pages, both of which were touched upon in the Monitoring PostgreSQL with Navicat Monitor 3.0 blog article. Today's topic will be the Instance Details page.

Instance Details Page at a Glance

The Instance Details page is accessible by clicking on an instance card in the Overview page. It shows server parameters and metrics in a visual way, providing you with a quick view of the server load and performance. There, you can view:

• the top 5 databases based on size
• the top 5 tables based on size
• system charts on CPU, Memory, Swap, and DB Disk Usage
• metrics on connections, queries, tables, buffer, cache and sort, as well as locks

Information on the Instance Details Page

The Instance Details page is split up into the following three sections:

• Summary
• Databases & Tables
• Charts

Here's a breakdown of each section:

Summary

The summary displays host information about the server, server properties, alerts and status. There, you can view or edit the instance variables as well as click on raised alerts to open the alert page.

Databases & Tables

This section displays the top five databases and tables by size, as well as a sixth category called "Others" that groups the remaining databases or tables. You can hover over each segment to show the size percentage. To view size information of all databases and tables in the instance, click the View All button.

Charts

Navicat Monitor displays visualizations of server performance metrics as small charts. The charts track and refresh the data at certain intervals, which are configurable by the user. Related metrics are displayed using different predefined colors and symbols. Note that, due to size constraints, the axis scales and labels of the small charts are not printed.

Both the time interval (X-axis) and refresh options are configurable via the AUTO REFRESH drop-down menu, the START FROM datetime picker, the time INTERVAL drop-down menu and the panning arrows:

Hovering the mouse pointer over a chart will display the values at that point:

You can also click on a chart to open the Chart page in order to view the details of an individual chart or to see more charts.

Pausing (and Resuming) Monitoring

You can pause and resume monitoring the instance here by using the Pause Monitoring and Resume Monitoring buttons, located in upper-right quadrant of the Instance Details page:

Navicat Monitor stops collecting information from the server until the monitoring resumes and adds the "[Paused]" indicator beside the names of paused instances in the Instances Pane (on the left):

Final Thoughts on Viewing PostgreSQL Instance Details in Navicat Monitor 3

Today's covered Navicat Monitor 3's Instance Details screen. It shows the server parameters and metrics in a highly visual way, gives you a quick view of the server load and performance, and more!

Navicat Monitor 3 comes packed with a variety of exciting new features. Case in point, you can now create traces that collect query data based on selected filters from the server log. When creating a trace, you can define criteria to filter the data collected by SQL Profiler and set a schedule for executing the trace. In today's blog we'll learn how to create a trace and view its results.

More about Traces

The Tracing feature is part of the SQL Profiler, which is only available for PostgreSQL. The SQL Profiler provides graphical query execution details for locating inefficient and slow queries.

The data collected from traces may be analyzed and used to troubleshoot performance issues. For example, you can see which queries are affecting performances in the production environment.

Creating a Trace

You can create new traces on the SQL Profiler, Query Analyzer, and Long Running Queries pages by clicking the Add Trace icon or + New Trace.

On the SQL Profiler page you'll have to select the instance before clicking the + New Trace button:

Clicking the Add Trace icon or + New Trace button will bring up the New Trace dialog. A prompt may pop up asking you to authorize Navicat Monitor to get relevant data from your instance.

Here are all of the details that you can enter on the New Trace dialog:

• TRACE NAME: the name of the trace.
• USER FILTER: the users/roles whose queries to include in the trace. Empty means including queries from all users/roles.
• DATABASE FILTER: the databases to trace. Empty means including queries against all databases.
• QUERY FILTER: search strings or QueryIDs to filter queries for the trace.
• MAX TRACE ROW COUNT: the maximum number of rows for the trace. SQL Profiler will terminate the trace when it reaches the row count.
• SCHEDULE: scheduling details for executing the trace.
• Share with: who can see the trace.

Here's the New Trace dialog with some of the above fields filled in:

Clicking on Create Trace that starts tracing according to the provided schedule. You should then see results after the first time period has elapsed.

Viewing Trace Results

A trace provides a graphical representation of the execution plan for each query with statistics for its components. Here is the trace for a query against the Sakila Sample Database:

You can see from the above screen capture that the Trace Results are divided into 3 sections:

• Query Table: The query table shows the basic information for the queries. Select a query to show its details and plans.
• Query Details: shows the complete statement of the query.
• Execution Plan: the execution plan that is generated for each query can be viewed in three different formats: Visual, Charts and Text-Based.

Final Thoughts on Tracing Queries on your PostgreSQL Instances with Navicat Monitor 3

In today's blog we saw how easy it is to create a trace and view its results in Navicat Monitor 3. Only available for PostgreSQL, traces collect query data based on selected filters from the server log. I think that you'll find them to be indispensable for locating inefficient and slow queries.

Version 3 of Navicat Monitor has just be released. Unsurprisingly, it packs many outstanding new features, as well as numerous improvements to existing features. One of the most noteworthy changes between version 2 and 3 is added support for PostgreSQL, including an SQL Profiler for PostgreSQL instances.

Today's blog will provide a quick guide on getting setup to monitor your be PostgreSQL instances using Navicat Monitor 3.0.

Adding a PostgreSQL Instance

You can see all of the monitored database instances on the Overview screen. In order to monitor our PostgreSQL instance, we need to add it to this screen. To do that, we simply need to click the "+New Instance" button at the top of the screen. Doing so presents a context list of available database types - both traditional and cloud-based:

Select the PostgreSQL item to open the New PostgreSQL Instance dialog:

Navicat Monitor can connect the database server over a secure SSH tunnel to send and receive monitoring data. It allows you to connect your servers even if remote connections are disabled or are blocked by firewalls.

In the PostgreSQL Server section, enter the following information:

• Host Name: The host name or IP address of the database server.
• Port: The TCP/IP port for connecting to the database server.
• Username: A monitoring user for connecting to the database server.
• Password: The login password of the monitoring user.
• Server Type: The type of the server. Can be Unix-like or Windows.

Navicat Monitor can also collect the DB server's system performance metrics such as CPU and memory resources. If you do not provide this login, you can still monitor your server, but system performance metrics will not be shown.

Once you've entered all of the above information, click the "New" button to create the new instance.

Activating a Token

Now that we've added our PostgreSQL instance, we're ready to activate it. To do that, we'll need to assign a token to it via Configurations > Activate Tolens & License Instances.

To activate the instance, we can locate it in the Unlicensed Instances list, check the box beside it, and click the License button to move it into the Licensed Instances list. Here's our "PostgreSQL Test DB 1" instance in the Licensed list:

We can now receive server statistic details about our instance's performance regarding query execution as well as server load, availability, disk usage, network I/O, table locks and more. By easily tracking the deviations and traffic among servers, we can examine possible solutions and adjust our server settings accordingly.

Monitoring Query Performance

The Query Analyzer tool provides a graphical representation of the query logs that makes interpreting their contents much easier. In addition, the Query Analyzer tool enables us to monitor and optimize query performance, visualize query activity statistics, analyze SQL statements, as well as quickly identify and resolve long running queries. Here's the Query Analyzer for our new instance:

There are no Long Running Queries at this time because the database is new and not currently in use.

Final Thoughts on Monitoring PostgreSQL with Navicat Monitor 3.0

Thanks to Navicat Monitor 3.0, we can now monitor our PostgreSQL instances, via many useful tools, including the Enhanced Query Analyzer and Long Running Queries screens.

Navicat Monitor 3.0 is available fore Windows, macOS (using Homebrew), and Linux. You can try Navicat Monitor 3.0 for 14 days free of charge to sample of all its new features before you buy.

It seems like only yesterday that Navicat Monitor 2.0 was released, adding great new features to an already stellar product. Now, version 3.0 is introducing yet more outstanding features, including:

• Support monitoring PostgreSQL instances.
• Support SQL Profiler for PostgreSQL instances.
• Enhanced Query Analyzer.
• Enhanced Long Running Queries.
• Many other new features and improvements.

Of course, all of Navicat Monitor's existing functionality remains in place, including:

• Agentless Architecture
• Real-time Performance Monitoring
• See how your instances are currently functioning easily
• Advanced root cause analysis
• Set custom alert thresholds
• Get notifications via email, SMS or SNMP
• Replication Monitoring
• Powerful Query Analyzer
• and more...

In today's blog, we'll be taking a look at the brand new Navicat Monitor 3.0 with the emphasis being on the new features listed above.

About Navicat Monitor

Navicat Monitor is a safe, simple and agentless remote server monitoring tool that includes many powerful features to make your monitoring effective as possible. You can access Navicat from anywhere via a web browser to access statistics on server load and performance regarding its availability, disk usage, network I/O, table locks and more. Using this valuable data, you can easily examine possible solutions, tune the databases, and address potential issues before they can become serious problems or costly outages.

Navicat Monitor is a server-based software which can be accessed from anywhere via a web browser. With web access, you can easily and seamlessly keep track of your servers around the world, around the clock.

Support for PostgreSQL Instances

Just as version 2 added support for SQL Server, version 3 can monitor PostgreSQL as well. Here are all supported DB types in the New Instance menu:

Selecting PostgreSQL from the list brings up the New PostgreSQL Instance dialog, which includes all of the configuration options that you might need for a PostgreSQL instance:

You'll be happy to know that the SQL Profiler also works for PostgreSQL instances. It's a tool that provides graphical query execution details for locating inefficient and slow queries. It supports the creation of traces to collect data about the queries executed on an instance. The data can later be analyzed and used to troubleshoot performance issues.

Enhanced Query Analyzer and Long Running Queries

The Query Analyzer tool provides a graphical representation of the query logs that makes interpreting their contents much easier. In addition, the Query Analyzer tool enables you to monitor and optimize query performance, visualize query activity statistics, analyze SQL statements, as well as quickly identify and resolve long running queries. In version 3, the Long Running Queries chart has been moved to the top of the page for easier identification:

Moreover, the entire Long Running Queries section provides the ability to drill down to specific intervals:

Information about Long Running Queries may be exported as a PDF or scheduled report.

Final Thoughts on Navicat Monitor 3.0

In this blog, we explored just a few of the exciting new features of Navicat Monitor 3.0, including Support for PostgreSQL Instances, as well as Enhanced Query Analyzer and Long Running Queries. There are plenty more new features that we'll be looking at over the next several weeks.

Navicat Monitor 3.0 is available for Windows, macOS (using Homebrew), and Linux. You can try Navicat Monitor 3.0 for 14 days free of charge to sample of all its new features before you buy.

Table Names

Naming conventions are a set of rules (written or unwritten) that should be utilized in order to increase the readability of the data model. You may apply these rules when naming anything inside the database, including tables, columns, primary and foreign keys, stored procedures, functions, views, etc. You need not apply rules to all database objects. For instance, it would be perfectly fine to limit naming convention rules to tables and column names. It's really your decision, as using a naming convention is not mandatory, but beneficial nonetheless. This three part series will present some commonly used naming conventions and provide some tips for formulating your own. Part 1 will cover Table names, while Part 2 will focus on column names. Finally, Part 3 will address Naming Conventions for other database objects such as Foreign Keys, Procedures, Functions, and Views.

Why You Should Use a Naming Convention

Databases rarely have a small number of tables. In fact, it's not at all uncommon to have hundreds of tables. By following a naming convention, you'll make your life a lot easier by increasing the overall model readability, and making it easier to locate database (DB) objects.

Another good reason is that the database will slowly evolve over time. Although changes to the schema are usually avoided and done only when necessary, changing the name of a database object could affect your application code in a myriad of ways. Since you can expect the database will remain, more or less, very similar to its initial incarnation, if you apply best practices from the start and continue using them as you add new objects, you'll keep your database structure well organized over time.

Singular vs. Plural Table Names

One of the most commonly asked questions regarding the naming of tables is whether to use the singular or plural form. There are many differing opinions on this matter. In fact, we can see both views expressed in the schemas of the MySQL classicmodels and sakila sample databases, with the former employing plural table names, and the latter utilizing singular naming:

If it helps, most DBAs go with singular names. One reason is that plural names like "users" "roles" could lead to some weird table names down the road such "users_have_roles" rather than "user_has_role".

Describing Real-World Entities

Any time that you're naming entities that represent real-world things, you should use their proper nouns. These would apply to tables like employee, customer, city, country, etc. Usually, a single word should exactly describes what is in that table.

There are times that you'll have to use more than one word to describe what is in a table. One such example can be seen in the classicmodels database. There is one table for "orders" and another for "order_details":

The "orders" table contains fields such as the Customer ID, Employee ID, Order Date, Shipped Date, Freight, and Shipping Address. Meanwhile, the "order_details" contains data about the products ordered, such as the Quantity ordered and Price. The field could have been named "product_details" but that would not convey that the product was associated with an order.

Naming Related Tables

For relations between two tables, it's standard practice to use both tables' names. A verb may also be added between both names to describe what that action is, for example "user_has_role", or simply "user_role". The Sakila Sample Database follows this convention by joining related tables with an intermediary table that combines both names. We can observe two examples in the database model below - "film_actor" and "film_category":

Final Thoughts on Table Naming Conventions in SQL

Don't be afraid to stray from a naming convention if it doesn't make logical sense in a given situation. For example, if we had a product and invoice table, and we wanted to specify which products were on which invoice, the name "invoice_item" might make more sense than either "invoice_product" or "invoice_contains_product".

It's official: the age of Artificial Intelligence (AI) has arrived! Until our new overlords decide to use us to power their machines, let's take the time to fully enjoy all the benefits they provide and the myriad of ways that they make our lives easier. Case in point, the AI-driven chatbot, ChatGPT, by OpenAI, has been lauded for its ability to produce tremendously spot-on answers to questions across a broad range of topics. And, although ChatGPT may not be making our jobs obsolete just yet, it has proven to be amazingly adept at working with data sets, much like a DBMS. In today's blog, we'll explore how ChatGPT could be utilized to supplement a professional database development and administration tool like Navicat.

Creating the Data Set

ChatGPT is able to model a formal dataset from a list of delimited values. All you need to do is tell it what to do using regular, conversational language. ChatGPT is also able to answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. We can see an example on the OUseful.Info blog that created a table named "racerresults". Here are the instructions given to ChatGPT, along with a sampling of the input data:

Treat the following as a tab separated dataset. Using just the first, third and fourth columns, treat the data as if it were a relational SQL database table called "racerresults" with columns "Race", "Driver" and "Team", and the "Race" column as a primary key column. Display a SQL statement that could create the corresponding table and populate it with the data.Bahrain20 Mar 2022Charles LeclercFERRARI571:37:33.584Saudi Arabia27 Mar 2022Max VerstappenRED BULL RACING RBPT501:24:19.293Australia10 Apr 2022Charles LeclercFERRARI581:27:46.548Emilia Romagna24 Apr 2022Max VerstappenRED BULL RACING RBPT631:32:07.986Miami08 May 2022Max VerstappenRED BULL RACING RBPT571:34:24.258Spain22 May 2022Max VerstappenRED BULL RACING RBPT661:37:20.475Monaco29 May 2022Sergio PerezRED BULL RACING RBPT641:56:30.265erc...

From the above instructions and data, ChatGPT generated the following CREATE TABLE and INSERT statements:

With the data in place, we're ready to run queries against it.

Querying a Data Set with ChatGPT

In terms of query formulation, ChatGPT shares some similarities with Navicat, in that both allow you to construct queries with little knowledge of SQL. To do that, Navicat features the Query Builder tool. Here it is in macOS:

As for ChatGPT, it takes a question phrased in regular, conversational language, and produces the required SQL statement(s). For instance, given the following list of historical figures:

We can simply as ChatGPT how it would query for the oldest historical figure. Here is the resulting SQL statement and explanation offered by ChatGPT:

Fun with Data

ChatGPT can do a lot more than generate queries; it can also think creatively to assign emojis to each historical figure:

Final Thoughts on Supercharging Your Queries with Navicat and ChatGPT

While AI bots like ChatGPT are a long way from replacing traditional database tools, they do offer another tool to database practitioners who are looking for new and innovative ways of approaching data-related tasks. At the time of this writing, ChatGPT was at capacity and unable to accept new users, but once things die down a bit, I would urge you to give ChatGPT a try.

Subqueries can be categorized into two types:

• A non-correlated (simple) subquery obtains its results independently of its containing (outer) statement.
• A correlated subquery references values from its outer query in order to execute.

When a non-correlated subquery executes (independently of the outer query), the subquery executes first, and then passes its results to the outer query. Meanwhile, a correlated subquery typically obtains values from its outer query before it executes. When the subquery returns, it passes its results to the outer query.

Now that we know the difference between a correlated subquery and its non-correlated counterpart, this blog will cover how to write a correlated subquery in Navicat Premium 16.

Syntax and Usage

A correlated subquery is evaluated once for each row processed by the parent statement. The parent statement can be a SELECT, UPDATE, or DELETE statement. Here's the syntax for a SELECT query:

SELECT column1, column2, ....FROM table1 outerWHERE column1 operator  (SELECT column1, column2   FROM table2   WHERE expr1 = outer.expr2);

A correlated subquery is one way of reading every row in a table and comparing values in each row against related data. It is used whenever a subquery must return a different result or set of results for each candidate row considered by the main query. In other words, you can use a correlated subquery to answer a multipart question whose answer depends on the value in each row processed by the parent statement.

A Practical Example

Here's a rather ingenious query from stackoverflow against the Sakila sample database that fetches the most viewed film per country.

The first step is to count how many times each film was viewed in each country. Here is the SELECT statement for that:

SELECT   F.title AS title,   CO.country_id AS country_id,  CO.country AS country_name,   count(F.film_id) as timesFROM customer C INNER JOIN address A ON C.address_id = A.address_idINNER JOIN city CI ON A.city_id = CI.city_idINNER JOIN country CO ON CI.country_id = CO.country_idINNER JOIN rental R ON C.customer_id = R.customer_idINNER JOIN inventory I ON R.inventory_id = I.inventory_idINNER JOIN film F ON I.film_id = F.film_idGROUP BY F.film_id, CO.country_id;

And here is the above query and results in Navicat Premium 16:

The next step is to convert the above results into a list of countries, along with the most viewed film title and the number of times it was viewed. Here's the full query with correlated subquery with an explanation to follow:

Explanation:

• Subquery: Fetches a list of movie count, grouped by country.
• GROUP_CONCAT(title ORDER BY times DESC SEPARATOR '|||') returns ALL titles in that 'row', with the most-viewed title first. The separator doesn't matter, as long as never occurs in a title.
• SUBSTRING_INDEX('...', '|||', 1) extracts the first part of the string until it finds "|||", in this case the first (and thus most-viewed) title.

Final Thoughts on Correlated Subqueries

In today's blog we learned how to write a correlated subquery using Navicat Premium 16. Be forewarned that correlated subqueries can be slow. However, with proper optimizing, their speed can be increased significantly.

As you are no doubt aware, updating text values in the database is a commonplace occurrence. Nonetheless, it is a rare database administrator (DBA) that doesn't feel some trepidation upon executing batch updates against production tables. In today's blog, we'll learn how to use the SQL REPLACE() function to replace either a complete or partial string in a table column.

A Typical Scenario

Here's a screenshot of the products table from the classicmodels sample database:

Suppose that the makers of Chef Anton products have decided to enclose their products in quotation marks (""). This would require a total of 4 steps:

1. Employ the LIKE operator to identify rows with Chef Anton products.
2. Parse out the product name.
3. Add the enclosing quotation marks.
4. Convert the SELECT QUERY to an UPDATE.

Let's go over each step.

Identify Rows with Chef Anton Products

As mentioned above, we can utilize the LIKE operator to identify rows with Chef Anton products. Each of these begins with the string "Chef Anton's ", so we can search for it. To do that, we will need to escape the single quote (') character and include the multi-character "%" wildcard. Here is the resulting query and results in Navicat Premium 16:

Parse Out the Product Name

The next step is to parse out the product name so that we can enclose it within quotation marks. To do that, we can employ the LEN() function to calculate the number of characters after the "Chef Anton's " portion of the string and supply that result to the RIGHT() function:

Add the Enclosing Quotation Marks

The last step in constructing the SELECT query is to add the quotes around the product name. Having parsed out the product name, we can provide it to the REPLACE() function as the first parameter, along with the concatenated (quoted) version as the 2nd parameter:

An alternative way to achieve the same end is to simply use the CONCAT() function and feed it each part of the string as follows:

SELECT CONCAT(         LEFT(ProductName, LENGTH('Chef Anton\'s ')),  '"', RIGHT(ProductName, LENGTH(ProductName)-LENGTH('Chef Anton\'s ')),  '"'       ) AS product_name FROM products WHERE ProductName LIKE 'Chef Anton\'s %'; 

Convert the SELECT QUERY to an UPDATE

All that's left to do now is to convert our SELECT query into an UPDATE. Having executed the query as a SELECT first, we can be confident that the UPDATE statement won't affect any other rows than the ones we're interested in. Here is the UPDATE query and results confirming that only two rows were updated:

Upon refreshing the products table, we can now see our updated values:

Final Thoughts on How to Perform a Search and Replace in SQL

In this blog, we learned how to update a string in a table column using a four step process. By building up the query as a series of SELECT statements, we can minimize the risk of inadvertently changing data that we did not intend to.

The Code Snippets feature was introduced to all "Non-Essentials" Navicat Database Administration and Development tools in version 12. Version 16 added Code Snippets to Navicat's cloud services so that users could save their Code Snippets to the cloud and share them across Navicat products. For those of you who are unfamiliar with the Code Snippets feature, it allows you to insert reusable code into your SQL statements when working in the SQL Editor. Besides gaining access to a collection of built-in snippets, you can also define your own snippets. We've talked about Code Snippets before. The March 14, 2018 blog, Using Navicat Code Snippets, provided a general overview of the Code Snippets feature. Today's blog will cover how to create your own custom Code Snippets. It's something that can make writing queries a whole lot easier!

Creating a Code Snippet from Scratch or from Selected Text

There are a couple of ways to create a brand new Code Snippet. The first is by clicking the Create Snippet button in the Code Snippet pane:

If you've got a particular piece of code that you'd like to save for future use, you can select it in the Query Editor, bring up the context menu (i.e. right-click in Windows), and select Create Snippet from the menu:

That will bring up the New Snippet dialog with the Code text field filled in for you:

From there you can choose a category from the Label drop-down. This will allow you to filter by type later when you need to bring it up. The pre-defined choices are "Comment", "DLL", and "Flow Control". If none of these apply, you can always add your own! The next time you access the New Snippet dialog, it will be included in the Label drop-down.

There's also a text area for Remarks in order to add some context to the Snippet.

Finally, don't forget to give your Code Snippet a name! Here's the New Snippet dialog with all of the fields filled in:

Once saved, the new Code Snippet will appear in the Code Snippet pane:

Setting Placeholders

Placeholders are tabbable text selections that act as arguments for Code Snippets. In fact, auto-completed function arguments are themselves always displayed as placeholders! For instance, here is the Count() function:

To create a placeholder in your Code Snippet, select the text that you'd like to be highlighted in the Code Snippet and click the Set selected text as placeholder button:

Placeholder text is highlighted by a colored box for easy identification:

You can remove a placeholder the same way using the Remove placeholder button.

Conclusion

In today's blog we learned how to create our own custom Code Snippets in any "Non-Essentials" edition of Navicat 16. It's something that can definitely make writing queries faster and easier!

Aliases temporarily rename a table or a column in such a way that does not affect the underlying table(s) or view(s). As a feature of SQL that is supported by most, if not all, relational database management systems, aliases are a great way to both simplify your queries and/or customize the column headers in your result sets. In this blog, we'll do both, using Navicat Premium 16.

Renaming Columns

A lot of database designers use abbreviations for the table column names to keep them short, for example:

• emp_no for "Employee Number"
• qty for "quantity"

The full value of abbreviated column names are not always intuitive to those who view the query results. To remedy that, you can use column aliases that give columns more descriptive names in the result set.

The syntax for column aliases is:

column_name [AS] alias_name

Note that the AS keyword is optional.

You can include whitespace in your aliases, by enclosing them within single (or double) quotes like this:

column_name AS 'Alias Name'

Here's an example query that includes a few column aliases:

Using Aliases for Expressions

You've probably noticed that, if a query contains expressions, the entire expression is utilized as the column header. For example:

Assigning a column alias to the expression makes it much more palatable:

Table Aliases

Table aliases follow the same rules as column aliases, but their purpose is different, as table aliases don't appear anywhere in the query results. In their case, the idea is to use a shorter name in order to associate columns with their table in a way that shortens your queries.

The basic syntax of a table alias is as follows:

SELECT column1, column2....FROM table_name [AS] alias_nameWHERE [condition];

A column that is associated to its table is called a qualified column name. Columns need to be qualified when two columns with the same name appear in the same SELECT statement. In fact, we saw an example of qualified column names in the column alias example above. Here's another query that contains two actor_id columns - one from the actors table and another from the film_actor table:

Although the above query is perfectly functional, we can shorten it by employing table aliases:

Notice that unambiguous columns, i.e., those which only appear in one table, do not need to be qualified.

Another way that table aliases are helpful comes into play when using modern database tools like Navicat. Thanks to the auto-suggest feature, typing the table alias causes a drop-down of suggestions. In the case of table aliases, the drop-down will contain all the table columns:

This greatly accelerates query writing, which is an important part of professional database development.

Conclusion

This blog provided an overview of column and table aliases, along with some practical examples in Navicat Premium 16.

If you're interested in learning more about Navicat Premium 16, you can try the full unrestricted version out for 14 days completely free of charge!

Tablespace Management

This 3rd and final part of the Navicat 16 and Tablespaces series will focus on how to manage tablespaces in MySQL using Navicat Premium 16. Recall that Part 1 presented some advantages offered by tablespaces, including Recoverability, Ease of Adding More Tables, Automatic Storage Management, and the Ability to Isolate Data in Buffer Pools for Improved Performance or Memory Utilization. The second installment provided more information on what tablespaces are, how they work and the types of default tablespaces you'll find in the various relational database products.

Creating a Tablespace

Just as Navicat provides Table and SQL Designers, there are also facilities for working with Tablespaces. To open the Tablespace Designer, click on Others -> Tablespace from the Main Toolbar:

In the Designer, click on the New Tablespace button in the Toolbar:

The fields shown in the Designer will depend on the type of database that your working with. In the case of MySQL, you'll see the following fields:

• Engine drop-down: For standard MySQL 5.7 releases, only the InnoDB engine supports tablespaces, so it's the only option in the drop-down. MySQL NDB Cluster 7.5 also supports tablespaces using the NDB storage engine.
• Path textbox: Specifies the path of the datafile/tempfile. Note that you have to include the ".ibd" file extension.
• Block Size drop-down: The block size for the tablespace. MySQL only supports a block size of 1024, or 1 MB, so be sure to select that option from the drop-down.
• Block Size Unit: The size of one data block. As mentioned above, MySQL only supports a block size of 1024, or 1 MB; for other database types, you may choose K, M, G, T, P or E to specify the size in kilobytes, megabytes, gigabytes, terabytes, petabytes, or exabytes.

You can see the generated SQL statement by clicking on the SQL Preview tab:

Navicat will issue the CREATE TABLESPACE statement upon clicking the Save button. Here are the New Tablespace form fields after a successful Save operation:

Before saving the tablespace, Navicat will present a dialog for entering the tablespace name that will be utilized to display the tablespace in the Tablespaces Objects list:

Hence, entering a name of "classicmodels" will add it as seen below:

Altering a Tablespace

Selecting a tablespace from the Tablespaces Objects list will enable the Design Tablespace button in the toolbar for editing. If the database in question does not allow tablespace editing, as in the case of MySQL, the form fields will be disabled:

Otherwise, data can be modified and re-saved.

Deleting a Tablespace

Selecting a tablespace from the Tablespaces Objects list also enables the Delete Tablespace button in the toolbar. Clicking it will bring up a confirmation dialog that requires the user to check a box indicating that the Delete action is permanent and cannot be undone:

The user may also click the Cancel button to close the dialog without deleting the tablespace.

Conclusion of Navicat 16 and Tablespaces Series

Tablespaces allow database administrators to better control the physical storage layout by putting some tables on faster or more redundant disks, or to stripe tables across disks. This series covered both the theoretical side as well as more practical matters of tablespace management, from their creation to deletion using Navicat Premium 16.

How They Work

"What is it? It's it" - Epic, Faith No More

Welcome back to this series on working with tablespaces in Navicat 16. Part 1 presented some advantages offered by tablespaces, including Recoverability, Ease of Adding More Tables, Automatic Storage Management, and the Ability to Isolate Data in Buffer Pools for Improved Performance or Memory Utilization. This second instalment will provide more information on what tablespaces are, how they work and the types of default tablespaces you'll find in the various relational database products. The next and final part of the series will focus on how to manage tablespaces in Navicat 16.

Tablespaces As Containers

You can think of tablespaces as containers. These can be a directory name, a device name, or a file name. A single tablespace can have several containers. And, although it is possible for multiple containers (from one or more tablespaces) to be created on the same physical storage device, you will get the best performance if each container you create utilizes a different storage device. The figure below illustrates the relationship between tables and tablespaces within a database:

Tablespaces and the Database Manager

The database manager's role is to balance the data load across containers. As a result, all containers are used to store data to a lesser of greater degree. At the same time, the database manager does not always start storing table data in the first container. The number of pages that the database manager writes to a container before using a different container is called the "extent size".

The figure below shows the components of a tablespace, including the extent size:

Default Tablespaces

Most relational databases come with their own built-in tablespaces. Here are a few examples:

Oracle

Oracle comes with the following default tablespaces: SYSTEM, SYSAUX, USERS, UNDOTBS1, and TEMP:

• The SYSTEM and SYSAUX tablespaces store system-generated objects such as data dictionary tables. You should not store any objects in these tablespaces.
• The USERS tablespace is helpful for ad-hoc users.
• The UNDOTBS1 holds the undo data.
• The TEMP is the temporary tablespace which is used for storing intermediate results of sorting, hashing, and large object processing operations.

MySQL

Only the InnoDB engine supports tablespaces, as follows:

• The System Tablespace
• File-Per-Table Tablespaces
• Undo Tablespaces

DB2

When you create a new database, the database manager creates some default tablespaces for the database. These tablespaces are utilized as a storage for user and temporary data. Each database must contain at least three tablespaces as given here:

• Catalog tablespace
• User tablespace
• Temporary tablespace

Going Forward

That concludes the second instalment on tablespaces. This instalment provided some information on what tablespaces are, how they work and the types of default tablespaces you'll find in the various relational database products. The next and final part of the series will focus on how to manage tablespaces in Navicat 16.

The Advantages

Did you know that Navicat 16 supports tablespaces? A table space is a storage structure for tables (as well as indexes, large objects, and long data) that organizes database data into logical storage groupings that relate to where data is stored on the filesystem. It's main function is to link the physical storage layer and the logical storage layer. By assigning tables to a tablespace you can control the physical storage layout by putting some tables on faster or more redundant disks, or to stripe tables across disks. This series is split into two parts: in the first couple of blogs, we'll cover the theoretical side, specifically, what sort of advantages tablespaces offer, as well as how they work and. The second part will focus on more practical matters, i.e., how to manage tablespaces in Navicat 16.

Some Advantages Offered by Tablespaces

Besides the advantages mentioned above, tablespaces offer a few other benefits:

Recoverability

Putting objects into the same tablespace makes backing up and restoring the database easier, since you can backup or restore all the objects within a tablespace with a single command. Moreover, if you have partitioned tables and indexes that are distributed across tablespaces, you have the option of backing up and/or restoring only the data and index partitions that reside in a given tablespace.

Easy to Add More Tables

Although there are limits to the number of tables that can be stored in any one tablespace, should you have a need to store more tables than can be accommodated within a single tablespace, you can easily create additional tablespaces for them using the CREATE TABLESPACE command:

CREATE TABLESPACE tbs1    DATAFILE 'tbs1_data.dbf'    SIZE 1m;

Automatic Storage Management

Usually, you need to define and manage the tablespace containers yourself. However, certain databases, such as DB2, support automatic storage tablespaces, whereby storage is managed automatically. Creating a tablespace with the automatic storage tablespace option delegates the creation and management of containers to the database manager.

Ability to Isolate Data in Buffer Pools for Improved Performance or Memory Utilization

If you have a set of objects (for example, tables and indexes) that are queried frequently, you can assign the tablespace in which they reside to a buffer pool with a single CREATE or ALTER TABLESPACE statement. Temporary tablespaces can also be assigned to their own buffer pool to increase the performance of certain operations such as sorts or joins. For seldom-accessed data, or for applications that require very random access into a very large table, it might make sense to define smaller buffer pools; the data can be kept in the buffer pool for no longer than a single query.

Going Forward

This first instalment of the Navicat 16 and Tablespaces series presented several advantages offered by tablespaces. In the next blog, we'll learn more about how tablespaces work. Finally, we'll move on to working with tablespaces in Navicat 16.

As you well know, multiple server hits can slow down an application. For that reason, developers are keen to find the most efficient ways to update data using as few statements as possible. As it turns out, the SQL UPDATE statement does support the setting of fields from multiple tables using this syntax:

UPDATE table1, table2, ...    SET column1 = value1,        column2 = value2,        ...[WHERE conditions]

The syntax gets formed by the combination of various keywords that helps in the grouping of two or more tables, like the join keyword.

Today's blog will present an overview of the multi-table UPDATE statement along with an example using MySQL 8 and Navicat Premium 16.

Some Caveats

Combining two table updates into one statement is not without limitations and quirks. Here are some points to keep in mind:

• In the multi-table UPDATE query, each record satisfying a condition gets updated. Even if the criteria are matched multiple times, the row is updated only once.
• The syntax of updating multiple tables cannot be used with the ORDER BY and LIMIT keywords.

So, while the multi-table UPDATE statement is quite efficient, it is not ideal for every situation.

A Practical Example

To give the multi-table UPDATE statement a try, we'll create two tables named "library" and "book" and consider the case when one or more books are borrowed from the library. Doing so increases the count of books while decreasing the count of the books. As it turns out, that's the ideal scenario to combine two separate statements into one UPDATE query. This will avoid separate calls to the server, making it a very efficient operation.

Here are the definitions and contents of each table:

The library Table

The book Table

Here is the query that will update both tables:

UPDATE library l, book b    SET l.book_count = l.book_count - 2,        b.book_count = b.book_count + 2WHERE l.id = b.book_idAND b.id = '1AG';

In the above query, the l.id = b.book_id condition acts as an inner join which combines the two tables and operates on the combined table after checking the table constraints. Meanwhile, the b.id = '1AG' condition further reduces the target rows to those which pertain to user '1AG'.

Other join types like outer join and right outer join may be employed as well; the only mitigating factor is that the two tables getting grouped must have a similar/matching attribute.

As with the regular (single table) UPDATE statement, the SET keyword is used along with the UPDATE keyword to set the new values in existing rows. It causes the older values to be overwritten with the new data. We can observe the query results in Navicat below:

As expected, the book counts for user '1AG' and book 103 have been updated in both tables:

Conclusion

Today's blog presented an overview of the multi-table UPDATE statement along with an example using MySQL 8 and Navicat Premium 16. The lesson here is that the multi-table UPDATE statement works best for applying mathematical operations such as incrementing and decrementing on related table columns.

In the Joins versus Subqueries: Which Is Faster? blog article we learned that joins tend to execute faster than subqueries. Having said that, it's not a universal rule, so you may not want to automatically assume that a join will be preferable. As mentioned in that article, if you need to add many joins to a query, the database server has to do more work, which can translate to slower data retrieval times. This article will present a couple of quick tests you can perform to compare a query that employs joins to one that contains subqueries so that you can choose which performs best.

Two Queries, Same Result

Most of the time, a query can be written using joins or subqueries. To illustrate, here is a query that selects countries, along with their associated cities and addresses from the MySQL Sakila Sample Database. The first SELECT statement uses joins while the second one fetches the exact same data using subqueries:

SELECT    co.Country,    COUNT(DISTINCT ci.city_id) AS city_cnt,    COUNT(a.city_id)           AS address_cntFROM country coINNER JOIN city ci    ON co.country_id = ci.country_idINNER JOIN address a    ON ci.city_id = a.city_idGROUP BY    co.country_id;SELECT     Co.Country,    (Select COUNT(1)  FROM City Ci  WHERE Ci.country_id=co.country_id) AS city_cnt,    (Select COUNT(1)  FROM Address A    INNER JOIN city c on a.city_id=c.city_id  WHERE C.country_id=co.country_id) AS address_cntFrom Country Co;

We can easily compare the results in Navicat, because it can run multiple queries simultaneously. Each result set is shown in its own tab below the SQL Editor. In the image below, the contents of the Result 2 tab is shown next to Result 1 for quick comparison:

Query Execution Time

Having verified that both statements are equivalent, we can now compare their execution times.

To do that, we can select an individual statement, and click the Run button, whose label changes to Run Selected whenever text is selected in the editor. An Elapsed Time of 0.020s can be seen at the bottom of the screen:

Doing the same with the second statement yields an Elapsed Time of 0.021s. A minute difference, but one that would grow as the volume of data increases:

Comparing Execution Plans

A query's Execution Plan can reveal a lot of information about how quickly it will execute. In Navicat, we can view the Execution Plan by clicking the Explain button. While it takes some practice to become adept at interpreting the results of Explain, doing so can pay dividends when trying to ascertain a query's efficiency.

The Explain1 tab shows the Execution Plan for the first (join) query. We can see at a glance that it involves 3 SIMPLE selects:

Meanwhile, the Explain2 tab lists one PRIMARY select, followed by three DEPENDENT SUBQUERIES. Even without digging deeper, we can already see that there is an additional step required to execute the second (subquery) statement:

Conclusion

While this blog seems to confirm the conclusion reached by Joins versus Subqueries: Which Is Faster? article, it can be a worthwhile exercise to compare both a join and subquery approach. In any event, there are still times that a subquery is advantageous over joins, such as when you have to calculate an aggregate value on-the-fly and use it in the outer query for comparison.

Back in 2020, we learned about The NULL Value and its Purpose in Relational Database Systems. As stated in that article, the value NULL has become a special marker to mean that no value exists. You could also say that NULL values may indicate that a column could have a value, but you don't know what that value should be yet. In that context, they act as a placeholder until you finally collect the data needed to fill the table field with a real value.

Moreover, when you consider that all major database vendors support NULLs as default values, it only makes sense to use them, doesn't it? Well, not so fast. There are database designers who avoid using NULLs unless absolutely necessary. Do they know something that the rest of us don't? Read on to find out!

Space Considerations

Although NULL values represent "nothing" or "no value", they are treated as a value by the database. As such, they take up space on the hard drive. So, if you think that you are saving hard drive space by employing NULL values, you could be mistaken. In fact, NULL is considered to be a variable-length value, meaning that it could be a few bytes or several bytes, depending on the column type. The database leaves room for extra bytes should the value be larger than what is stored in the field, the result being that your database might take up more hard drive space than if you had used regular values.

Don't Create a Record with Missing Information

Some database administrators argue that if all the columns of a record can't be filled, then a record shouldn't be created. This argument obviously doesn't apply to all use cases, but the idea behind it is that a record should only be created when all fields have actual values without any placeholders. For example, in a banking application, you wouldn't proceed with a transaction if you didn't know the amount of the transaction. Fair enough, but this type of rigorous standard doesn't work so well in other industries such as e-commerce or websites that collect user data.

Complex SQL

Another disadvantage affects your database stored procedures. While most databases provide functions to detect NULL values, special care must still be taken to distinguish NULLs from other values. This means that your SQL procedures might be much longer than necessary, and they can become complex to read as well. A database administrator may reject code changes if the procedures are too convoluted and/or unintelligible.

Case in point, here's a small table in Navicat Premium 16 that contains a combination of values, empty strings, and NULLs:

In Navicat, it's easy to insert an empty string or NULL via the Edit menu.

Now here's a query that counts the number of names based on a variety of criteria:

We were looking for a count of 5 as records 4, 5, 7, 8, and 10 do not have values in them. However, only the combo_count returned 5. This is because while a NULL value does NOT have a length, so NULLs are not picked up by the length() function.

From this example, we can conclude that allowing NULL values can make you work extra hard to get at the kind of data you are looking for. Moreover, allowing NULL values may reduce your confidence regarding the data in your database, as you can never quite be sure whether a value exists or not.

Conclusion

Most database practitioners choose to allow some NULL values in their database tables, as they are the default value in just about any well known database and function well as a placeholder for missing data. On the other hand, we saw here that some DBAs don't feel that NULLs are worth the extra trouble they entail. The moral of this story is that you should consider your own business processes before designing your database(s) and choose a structure that best suits your data.

Although there are few database administrators (DBAs) who do not believe in performing regular database backups, there are many opinions on how best to do so. Whichever approach you espouse, there are many good reasons to keep a copy of the database schema. In the event of data loss, you can restore the database structure from the schema, and then populate it with the latest data backup.

Some database vendors, such as MySQL, offer free utilities (i.e. mysqldump) for backing up the database structure on its own, while others require a specific administration tool to do so. If you're a Navicat user, there's no need for external tools. While data backups may be performed using the Backup Wizard, the schema can be copied using the Data Transfer Tool. In this blog, we'll learn how!

About the Data Transfer Tool

The Navicat Data Transfer Tool is a wizard-driven process that helps you to transfer tables, collections or other objects from one database/schema to another, or to a SQL/script file via a series of screens. The target database/schema can reside on the same server or on a remote server. In Navicat Premium, you can also transfer objects across server types, e.g. from MySQL to SQL Server. Only MongoDB does not support transferring to other server types, due to it being a NoSQL document database, as opposed to a traditional relational database.

You'll find the command to launch the Data Transfer Tool under Tools > Data Transfer in the Main Menu:

Source and Target Screen

The first screen is where you provide the Source Connection and Database/Schema and Target. The Target may be another connection or an SQL File that you can execute to rebuild the database schema later.

We'll specify the File option, and choose a location and name for the SQL/script file:

Options Screen

At the bottom of the Source and Target Screen, you'll see a button for choosing various options, including Table, Record, and Other options.

To backup the database structure only, we simply need to uncheck the Create records option as shown in the image below:

Database Objects Screen

On the Database Objects Screen, we can choose which tables, views, procedures/functions, and events to backup. If we do not select anything here, an empty database will be backed up, without any objects.

Summary Screen

The last screen in the process provides a summary of your choices along the way, so that you can verify them before clicking the Start button. Should you change your mind about anything, you can click the Back button to return to the relevant screen.

You'll also find a couple of common options there for quick selection:

Progress Screen

The Progress Screen displays every step of the backup along with a summary of transferred objects, errors, and the elapsed time:

Conclusion

Keeping a copy of the database schema is always a good idea so that, you can restore the database structure from the schema, and then populate it with the latest data backup, in the event of data loss. Although some database vendors, such as MySQL, offer free utilities (i.e. mysqldump) for backing up the database structure on its own, an even easier option is to use Navicat 's Data Transfer Tool. It can transfer tables, collections or other objects from one database/schema to another, or to a SQL/script file via a series of screens!

Last week's article shed some light on the Outer Joins in SELECT queries. It's a JOIN type that returns both matched and unmatched rows from related tables. Unfortunately, it is not supported by all database (DB) vendors, including MySQL. But that's OK, because Outer Joins can be emulated by combining three other JOIN types, namely LEFT, INNER, and RIGHT joins. In this article, we'll learn more about LEFT and RIGHT joins and how, when combined with an INNER JOIN, they create an OUTER JOIN.

The LEFT Join

The LEFT JOIN returns all rows from the left table and the matching rows from the right table. If no matching rows are found in the right table, a NULL is returned. Here's the syntax:

SELECT    select_listFROM    T1LEFT JOIN T2 ON    join_predicate;

The following VEN diagram illustrates what data is fetched from two tables T1 and T2 using the LEFT JOIN clause:

The RIGHT Join

The RIGHT JOIN returns all rows from the right table and the matching rows from the left table. If no matching rows are found in the left table, a NULL is returned. Here's the syntax for that:

SELECT     select_listFROM     T1RIGHT JOIN T2 ON join_predicate;

The following VEN diagram illustrates what data is fetched from two tables T1 and T2 using the RIGHT JOIN clause:

Combining Joins to Emulate an OUTER JOIN

It is common knowledge throughout the database community that MySQL lacks support for FULL OUTER JOIN. One common workaround to this short-coming is to use a UNION ALL to combine three result sets from a LEFT JOIN, an INNER JOIN, and a RIGHT JOIN of two tables, where a join_column IS NULL condition is added to the LEFT and RIGHT joins.

To demonstrate how to emulate an OUTER JOIN as described above, we'll write a query against the same Project Management database as last week in the Understanding SQL Outer Joins article, but in MySQL this time.

Finding Unmatched Records in the Left Table

This first query will return rows that are found only in the left table. The query below achieves this effect by using LEFT join with a WHERE clause that specifies that the common (joining) column in the right table is null:

Finding Unmatched Records in the Second Table

The second query will return rows that are found only in the right table. To do that, we'll use a RIGHT join with a WHERE clause that specifies that the common (joining) column in the left table is null:

Finding Matched Records in Both Tables

To find records that appear in both tables, we can use a standard (INNER) JOIN like the following:

When combined using UNION ALL, the three separate queries produce the same results as an OUTER JOIN:

Conclusion

In this article, we learned more about LEFT and RIGHT joins and how, when combined with an INNER JOIN, they create an OUTER JOIN. Like last week, there is again a caveat. This technique can be quite inefficient on large tables when used with ORDER BY and/or LIMIT queries, as these utilize a filesort. In such cases, you may want to employ another approach.

The Outer Join is the least understood of all the SQL Join types. Perhaps it's because Outer Joins are required somewhat less often than other join types. In any case, there is nothing inherently strange about Outer Joins. As we'll see in this blog article, a few examples of the Outer Join in action should be enough to clarify any misapprehensions and/or confusion you may have about them.

This blog will first describe the syntax and purpose of the Outer Join statement, which will then be followed by some illustrative examples.

OUTER JOIN Syntax

The OUTER JOIN (or FULL OUTER JOIN if you like) keyword returns all records of two joined tables when there is a match in either the left (table A) or right (table B) table records. The following VEN diagram depicts the potential matches and OUTER JOIN syntax:

Hence, FULL OUTER JOIN returns unmatched rows from both tables, as well as matched rows in both tables. In other words, rows are returned by the query regardless of whether or not the join field (Clave) value is matched across both tables.

Still confused. Don't worry, we'll go over some examples in the next section to clear things up.

OUTER JOINs In Practice

In this tutorial we will use the well-known Northwind sample database.

The following SQL statement selects all customers, and all orders:

SELECT Customers.CustomerName, Orders.OrderIDFROM CustomersFULL OUTER JOIN Orders ON Customers.CustomerID=Orders.CustomerIDORDER BY Customers.CustomerName;

One of the hallmarks of a result set produced by an OUTER JOIN query is that you will see Null values in either joined columns as one may appear in one table, but not the other. We can observe that here in this screen capture of the above query and results in Navicat Premium 16:

Or course, you will never see Nulls in both table columns because a value must appear in at least one table. It should also be noted that the present of a Null in the ContactName columns is problematic because it means that an order was placed that is not associated to an existing customer. This would point to a flaw in the database design, most likely a missing Foreign Key Constraint.

Our second example fetches data from a Project Management database, namely project managers and projects. Here's the SQL:

SELECT     m.name member,     p.title projectFROM     pm.members m    FULL OUTER JOIN pm.projects p         ON p.id = m.project_id;

Again, we can see Null values (at least one Null)

In this case, the results indicate that Jack Daniel has no projects at the moment. Whether or not this represents an issue would depend on that organization's particular operations. It may be perfectly reasonable for a Project Manager to be without a project, or for projects to be unnasigned, at any given time.

Conclusion

Hopefully today's blog helped shed some light on the purpose and usage of Outer Joins in your queries. One final word of warning: Outer Joins can result in very large result sets, so use them sparingly, and include filtering clauses such as WHERE to minimize the number of rows returned.

In the realm of Information Technology, or IT as it's more commonly known, an enum is a special data type that encapsulates a set of predefined constants. As such, the variable may only hold one of the values that have been predefined for it. Common examples include compass directions of NORTH, SOUTH, EAST, and WEST or the days of the week.

One of the complicating factors when storing enums in a database table is that their values may be numeric or alphabetic (i.e. strings). Moreover, you'll want to prevent users from adding any values to the table that are not part of the permissible enum set. We'll be addressing both of these issues in today's blog.

Enum Values Explored

The most basic enums contain a set of zero-based ordinal values each represented by a constant, seen below in Java:

public enum Day {    SUNDAY, MONDAY, TUESDAY, WEDNESDAY,    THURSDAY, FRIDAY, SATURDAY }

More complex enums may also contains other types; strings are the most common, but more complex objects are also supported. Here is an enum for representing different environment URLs (also in Java):

public enum Environment {    PROD("https://prod.domain.com:1088/"),     SIT("https://sit.domain.com:2019/"),     CIT("https://cit.domain.com:8080/"),     DEV("https://dev.domain.com:21323/");     private String url;     Environment(String envUrl) {        this.url = envUrl;    }     public String getUrl() {        return url;    }}

Generally, it is considered bad practice to store enumerations as numerical ordinal values, as it makes debugging and support difficult. It's usually preferable to store the actual enumeration value converted to string. To illustrate, imagine that we had an enum of card suits:

public enum Suit {   Spade,   Heart,   Diamond,   Club }

Now imagine that you are a database practitioner trying to decipher either of the following query results:

Name          Suit------------  ----John Smith    2Ian Boyd      1Name          Suit------------  -------John Smith    DiamondIan Boyd      Heart

I think that you will agree that the latter is much easier to interpret as the first option requires getting at the source code and finding the numerical values that were assigned to each enumeration member.

Although storing strings takes more disk space, enumeration member names tend to be short, and hard drives are cheap, making the trade-off worth while to make your day-to-day job easier.

Another problem with using numerical values is that they are difficult to update; you cannot easily insert or rearrange the members without having to force the old numerical values. For example, adding a value of Unknown to the Suit enumeration would require you to update it to:

...in order to maintain the legacy numerical values already stored in the database.

Validating Enum Values In the Database

Many modern databases, including MySQL and SQL Server, support the ENUM data type. Specified as strings, ENUM values are automatically encoded as numbers when stored for compact storage.

Here are the MySQL statements in Navicat for MySQL to create and populate a table with shirts and their sizes, as well as a SELECT query that fetches medium sized shirts:

If we now try to insert an invalid ENUM value, we get the following error:

Although the message states the the value was truncated, the data was not actually inserted.

Conclusion

In today's blog, we explored how to with enum values in the database, including how to store, validate, insert, and retrieve them.

Interested in trying Navicat for MySQL? You can use it for 14 days for free!

Strings as Primary Keys

In this third and final installment of this series on choosing a Primary Key for relational databases we'll be examining some of the reasons for employing string data as a Primary Key (PK). Recall that, in Part 1, we covered Natural and Surrogate Primary Keys and considered why one might choose one over the other. Part 2 explored String and Numeric data types as Primary Keys in an effort to ascertain whether one is preferable to the other. Now it's time to set the record straight and conclude whether or not string - or alphabetic - data can make a suitable PK.

Ready-made Keys

There is often a sensible natural primary key for your data which has a universal meaning and may not be an integer. If so, then adding an artificial key just for the sake of an integer type adds nothing but redundancy. Performance may be hampered slightly, but it's slightly less important than correctness, integrity, and appropriate modelling, in the estimation of many database developers.

A non-obvious benefit of alphabetic keys is that a short symbolic string can simplify debugging by being immediately human-readable in data dumps (without additional joins). For example, US states have an alphabetic code which is unique and is meaningful as a key outside the schema. Also, countries have alphabetic ISO codes. And there are an endless number of other examples, such as vehicle VIN numbers, invoice IDs, etc.

Using GUIDs as Primary Keys

If you have more than one database, auto-incrementing keys can cause redundant records. One way around this problem is to use GUIDs. Short for "Globally Unique IDentifier", GUID is a 16 byte binary data type that is guaranteed to be unique across tables, databases, and even servers.

How you create a GUID varies across different databases, but, in SQL Server, the NEWID() function is used as shown below:

SELECT NEWID()

Here's a statement for creating a table with the UNIQUEIDENTIFIER data type. To set a default value for the column we will use the default keyword and set the default value as the value returned by the NEWID() function:

USE EngDBGO CREATE TABLE EnglishStudents1(Id UNIQUEIDENTIFIER PRIMARY KEY default NEWID(),StudentName VARCHAR (50) )GO INSERT INTO EnglishStudents1 VALUES (default,'Shane')INSERT INTO EnglishStudents1 VALUES (default,'Jonny')

This will ensure that whenever a new record is inserted in the EngStudents1 table, by default, the NEWID() function generates a unique value for the Id column. When inserting the records, we simply have to specify "default" as value for the first column. This will insert a default unique value to the Id column.

GUIDs vs. UUIDs

While GUIDs (as used by Microsoft) and UUIDs (as defined by RFC4122) look similar and serve similar purposes, there are subtle-but-occasionally-important differences. First, let's establish what UUIDs are.

UUIDs are 128 bits values, with textual representation in hex digits. It bears repeating, because many people think that UUIDs are stored as text. UUIDv4 values being random, you don't have a guaranteed uniqueness. However, the probability of a collision is rather small. Also, keep in mind that in the extremely unlikely case of colliding UUIDs, it will be caught by the DB thanks to the primary key constraint. You'll also be happy to know that UUIDv4 are perfectly well indexed by most relational DBs.

Some Microsoft GUID docs allow GUIDs to contain any hex digit in any position, while RFC4122 requires certain values for the version and variant fields. Also, GUIDs should be all-upper case, whereas UUIDs should be "output as lower case characters and are case insensitive on input". This can lead to incompatibilities between code libraries.

You could say that GUID is Microsoft's implementation of the UUID standard. Treat them as a 16 byte (128 bits) value that is used as a unique value. In Microsoft-speak they are called GUIDs, but call them UUIDs when not using Microsoft-speak.

The Verdict

In this third and final installment of this series on choosing a Primary Key for relational databases, we set out to decisively conclude whether or not string - or alphabetic - data can make a suitable PK. The answer is YES, but with some caveats. You should expect to take a slight performance hit. If that prospect does not deter you, then you've got some options to work with, from short symbolic strings to GUIDS and UUIDs. Finally, try to use fixed length strings rather than varchars, as fixed length strings perform better.

String vs. Numeric Data Types as Primary Keys

Welcome back to this series on choosing a Primary Key for relational databases. In Part 1, we covered Natural and Surrogate Primary Keys and considered why one might choose one over the other. Today's instalment will explore String and Numeric data types as Primary Keys in an effort to ascertain whether one is preferable to the other.

String and Numeric Data Types in Relational Databases

Both string and numeric nomenclatures are actually umbrella terms that encapsulate several different data types. For starters, the string data type is a generic IT term that traditionally refers a sequence of characters, either as a literal constant or as some kind of variable. With regards to databases, single characters, represented by the CHAR type, are also grouped with Strings. Other DB string data types include VARCHAR, BINARY, VARBINARY, BLOB, TEXT, ENUM, and SET. Numeric data types include both exact numeric data types such as INTEGER, SMALLINT, DECIMAL, and NUMERIC, as well as the approximate numeric data types like FLOAT, REAL, and DOUBLE PRECISION.

The Great Debate

Advice on what data type works best for primary keys (PKs) abounds on the Internet. Some sites state outright that numeric keys are almost always superior to character-based ones, while an equal number of sites promote the use of string types. Meanwhile, DB vendors themselves don't suggest one type over the other. What they do offer, are instructions regarding the PRIMARY KEY Constraint. It uniquely identifies each record in a table and posits that:

• Primary keys must contain UNIQUE values, and cannot contain NULL values.
• A table can have only ONE primary key; and in the table, this primary key can consist of single or multiple columns (fields).
• PK values should not be changed over time.

So long as your PK satisfies the above criteria, then you're good to go, as far as DB vendors are concerned. But that doesn't mean that one type may offer some advantages over the other. Let's dive into those now.

Say Aye for Numeric Types

Back when I was first learning about database development, I was instructed that numeric types are best for PKs because they are both faster and memory efficient. This opinion was reinforced by my first employer, the Federal Government, who utilized numeric PKs, even if that meant adding a surrogate key.

There are plenty of reputable reference sites who echo that sentiment. Speaking about MySQL, Mysqltutorial.org states:

Because MySQL works faster with integers, the data type of the primary key column should be the integer e.g., INT, BIGINT. And you should ensure sure that value ranges of the integer type for the primary key are sufficient for storing all possible rows that the table may have.

MySQL is far from unique on its handling of numeric data; another page on Primary Keys in Oracle states that "primary keys typically are numeric because Oracle typically processes numbers faster than any other data types."

They even go so far as to say that PK data should be "meaningless":

Sometimes, you may want use meaningful data, which considers being unique, for the primary keys e.g., social security number (SSN), vehicle identification number (VIN), email, and phone number. However, you dont know when the email or phone number changes or is reused by another person. In such cases, it will create many data problems. In the database world, the artificial keys are known as surrogate keys which are as opposed to natural primary keys.

Coming Up Next Week...

So far, it would seem that Numeric primary keys are best. However, we have not yet heard from the pro-string side. Perhaps they can offer some very good reasons for using strings instead.

Natural vs. Surrogate Keys

One of the first decisions you'll be faced with as a database designer is what kind of Primary Key (PK) to use on your tables. If you ask anyone who works with databases on a daily basis, whether database administrator, developer, or tester, you'll get a myriad of opinions and justifications to go along with them. Compounding the impediments to coming up with an answer is that there is no one size fits all solution. With that in mind, this series will present some reasons both for and against different types of PKs. Somewhere in all those ideas, there will be a few that will steer you towards the best type of PK to use for your organizational needs. In this first instalment, we'll compare the two basic types of PKs: Natural and Surrogate Keys. Later, we'll cover the questions of whether or not to use the database Auto Increment feature as well as which data type(s) - if any - make the best PKs.

Natural Keys

A natural key is one made up of one or more columns that already exist in the table (e.g. they are attributes of the entity within the data model) that uniquely identify a record in the table. Since these columns are attributes of the entity they inherently possess business meaning. The following is an example of a table with a natural key in Navicat Premium 16's Table Designer. We can easily identify the Primary Key by the key icon in the Key column:

Looking at the data, we can see that the productCode has business meaning:

Surrogate Keys

A surrogate key (or synthetic key, pseudokey, entity identifier, factless key, technical key, etc!) is a system generated (GUID, sequence, unique identifier, etc.) value with no business meaning that is used to uniquely identify a record in a table. The key itself could be made up of one or multiple columns (i.e. Composite Key) as well. We can see a surrogate key in a table from the same database, which defines a customerNumber column as its PK:

Although not Auto Incrementing, it's a numeric field that is unrelated to the customer entity:

Making a Decision

So why does one table employ a Natural Key while the other utilizes a Surrogate Key?

It's quite common for products to have some sort of unique inventory number, which makes an ideal PK. Adding an additional numeric key would simply be a waste of disk space and would almost certainly require an additional index on the productCode column for searching. On the other hand, customers don't typically come with unique identifiers. Speaking as someone who has had to uniquely identify persons in a database, it takes a surprisingly long list of columns to do so. Hence, it's usually much easier to assign a numeric Surrogate Key that index every column in the table.

Conclusion to Natural vs. Surrogate Keys

In this first instalment on Choosing a Primary Key, we explored Natural and Surrogate Primary Keys and considered why one might choose one over the other. It's important to decide between using a Natural or Surrogate first because which you choose will help answer some of the follow-up questions as well - especially in the case of a surrogate key.

If you'd like to give Navicat 16 a test drive, you can download a 14 day trial here.

Application developers have long held the belief that housing database operations within stored procedured yielded optimum performance and guarded against SQL Injection attacks. It was also thought that these advantages were worth the extra costs associated with maintenance, testing, and migration of database logic to a different vendor. In recent years, the tide has been turning away from stored procedures - or procs - towards Object-relational Mappers (ORM) such as Hibernate or Entity Framework as developers have begin to question these long-standing assumptions.

The Are Stored Procedures an Outdated Tool? article highlighted a few reasons for eschewing stored procedures in favor of application code and ORMs. This week, we'll explore the two myths introduced above and see if they still stand up to scrutiny today.

Performance Advantages

In the early days of the Internet, it was common practice to minimize network traffic in order to boost performance. Stored procedures helped reduce network traffic by requiring only the proc name and parameters to be transferred over to the server rather than the full SQL statement. Considering the complexity and length of some production queries, these gains could sometimes be substantial. Today, whatever gains you may garner from this approach are easily offset by the fact that you are all too likely to end up calling the same procedure two or three times with the same parameters in the same request. Meanwhile, an ORM would look in its Identity Map and recognize that it already retrieved that result set, so there's no need to do another round trip. Moreover, it should be noted that the claim that stored procedures are cached on the server, whereas ad-hoc SQL is not, is a myth that was busted by Frans Bouma in his blog post, Stored Procedures are bad, m'kay?.

Stored Procedures and SQL Injection

It has often been said that stored procedures offer natural protection against SQL injection because they separate data from instructions. This is true, as long as the developer doesn't use dynamic SQL within the stored procedure where a raw string is passed via the input parameter that replaces the placeholder. Here's a badly written proc that shows exactly how it could open up the database to SQL injection:

create procedure GetStudents(@School nvarchar(50))asbegin    declare @sql nvarchar(100)    set @sql = 'SELECT STUDENT FROM SCHOOL WHERE SCHOOL LIKE ' + @School    exec @sqlend

You can write SQL that eliminates SQL injection vulnerabilities by using parameterized queries. Written in a programmatic language such as python, TypeScript, or Java, a prepared statement like the one below can sanitize user input so that it's safe to use in your queries:

String sql = "SELECT STUDENT FROM SCHOOL WHERE SCHOOL LIKE ? ";PreparedStatement prepStmt = conn.prepareStatement(sql);prepStmt.setString(1, "Waterloo%");ResultSet rs = prepStmt.executeQuery();

The lesson here is that protection against SQL injection is not a benefit of stored procedures themselves, but rather the convention of not concatenating SQL strings together.

Going Forward

This blog explored a couple of long-held assumptions about stored procedures that don't quite hold true today. While not by themselves sufficient reason to hop off the stored procedure bandwagon, they do strongly suggest that it may be time to reevaluate your application architecture.

Data Synchronization

You are now able to choose not to preview the results and deploy directly. Navicat now offers two buttons:

• Compare & Preview: preview the comparison results.
• Compare & Deploy: skip the preview and deploy immediately.

Dump SQL File

Weve enhanced our Dump SQL File feature. The file created from Dump SQL File can now be opened in three ways:

• Open: open the file using Query Editor.
• Open with External Editor: open the file with different editors.
• Open Containing Folder: open the folder where the file located.

Stored procedures have been falling out of favour with some organizations for several years now. The preferred approach of these businesses for accessing their database(s) is to employ an Object-relational Mapper (ORM) such as NHibernate or Entity Framework. Over the next couple of blog articles, we'll explore their reasons for doing so, and whether this paradigm shift points to the eventual obsolescence of Stored Procedures.

Stored Procedure Basics

As expressed in the Understanding Stored Procedures and Functions in Relational Databases:

A stored procedure - or "proc" for short - is a set of Structured Query Language (SQL) statements with an assigned name, which are stored in a relational database management system as a group, so it can be reused and shared by multiple programs. Stored procedures can access or modify data in a database, but it is not tied to a specific database or object. This loose coupling is advantageous because it's easy to reappropriate a proc for a different but similar purpose.

Sounds like a useful tool so far, but, as we'll see in the next section, not everyone is convinced.

Drawbacks of Stored Procedures

Despite their long-established advantages, opponents of Stored Procedures point out their many disadvantages, such as:

• Bug Prone: Since stored procedures encapsulate application logic, it should be moved into the application code, where it can be better managed and tested. Due to the inherent challenges in testing stored procs, they can be the cause of some really nasty bugs.
• Implementation Differences: Stored procedure implementations vary from vendor to vendor. While many DB developers consider Oracle's stored procedures to be of the highest quality, other products' procedures, such as those of MySQL, are less well conceived.
• Changing Requirements: One of the original use cases for stored procedures was to reduce network traffic. However, with today's lightning fast network speeds, this isn't nearly as big an issue as it once was. As such, dropping application logic into stored procedures can be a case of premature optimization.
• Difficult to Maintain: Stored procedures tend to require much more work to develop and maintain than application code. For starters, you need to have individual stored procedures to execute create, retrieve, update and delete operations for each table, plus a separate stored procedure for each different query that you wish to make. Then, you need to implement classes and/or methods in your code to call each stored procedure. Compare that with an O/R mapper, where all that's needed are class definitions, database table, and mapping file. In fact, modern ORMs use a convention-based approach that eliminates the need for a separate mapping definition.
• Code Duplicaton: Stored procedures require you to violate DRY (Don't Repeat Yourself) principle, since you have to reference database table columns half a dozen times or more. Moreover, it isn't possible to pass an object as a parameter to most stored procedures - only simple types like string, integer, date/time, etc. - making it virtually impossible to avoid huge parameter lists (a dozen or more is common!).

Even the most staunch opponents of stored procedures still use them in some circumstances. For example, stored procs are great for database housekeeping or reporting. Otherwise, developers should have very good reasons to integrate them into their applications.

Going Forward

Having heard a few reasons for eschewing stored procedures in favor of application code and Object-relational Mappers (ORMs) such as NHibernate or Entity Framework, you may be convinced that this is the way to go. Well, don't make the switch just yet; in the next installment, we'll consider a few more motivations for both abandoning and staying with stored procs. Then, if you still want to make the change, at least you'll be armed with a more complete understanding of all the issues involved.

Charts

Trend Line supported

Starting with Navicat 16.1, trend line can be added in Vertical/Horizontal Bar Chart, Line Chart, Area Chart, Bar and Line Chart, and Scatter Chart. To make your chart even more understandable and easily interpreted, you can change the default appearance of a trendline.For this, just simply click on Trend Line tab in Properties pane.

Better properties pane

The Properties pane allows you to customize your charts. Before, all properties were displayed under one panel only. Now, properties are categorized into several tabs - General, Data, Axis and Trend Line. Each tab regroups the properties and settings, ensuring that you can easily look for the right properties settings when you need them.

Support for additional month format

Weve added the ability to abbreviate the name of the month to one-letter. For example, January appears as J.

Changing chart color

Previously, it was time-consuming to look for the color settings in Properties pane. Its now possible to change the chart color on right-click context menu.

Other improvements

• Weve enhanced our filter function which now dividedinto two filters. The Data Filter is for filtering the source data in the current chart; The Display Filter is applying on the display data.
• Added smoothed line in Line Chart.
• Added "Restore Defaults (Style Only)" on each Properties tab for restoring the chart style only.
• Added hint when resizing/moving objects in dashboard.

Data Viewer

Better view for your data

We think you will particularly appreciate the improvement for highlighting the selected row and active cell. You can now easier to see your current place in Grid View.

Weve improved the data alignment in Grid View. Previously,this made it difficult to view data between number column and text column, but now Navicat handles this situation well.

Show and hide Columns

Weve made it easier to show/hide columns adding a Columns button on the toolbar for quick access.

Table Designer

Heres what we improved for Table Designer:

• Added column CHECK support for MariaDB 10.2.1 or above.
• Added support for PostgreSQL partitioned hash.

Query

Completion in MongoDB

Logical operator is now available in code completion for MongoDB.

Copy code snippets to cloud

When writing SQL, you might sometimes insert reusable code into editor. If you want to share your own snippet in cloud, use a new action, Copy Snippet To, under the Code Snippet pane.

General

Connection tree

• Pressing Ctrl+F in connection tree will now open the Search box in tree. It activated another Search box before.
• Adding shortcut in submenu to have quick access to create a new connection profile.

Information Pane

The behavior of the Information Pane is now more straightforward:

• Highlighting and copying the object title directly.
• Adding the Copy button - make it easierto copy the value to clipboard.
• Showing the related information that you require. Before, SSH/HTTP still displayed even they werent being set.
• Finding in DDL tab - if you are looking for a particular information in DDL, you can use text search for help!

Open currenttabinnew window

You can now right-click on a tab and open it to new window.

Connection

Support OceanBase Community Edition

We announce that OceanBase has been added in our Navicat family. Together with our efficient technical support, Navicat is exactly what you need.

Upgrade for MongoDB Driver

The following points have been improved for MongoDB connection:

• MongoDB driver upgraded from 1.16.2 to 1.21.1.
• Support MongoDB Atlas Serverless.
• You can now choose either driver 1.21.1 (Default) or 1.16.2 (Legacy).

Show Hidden Passwords Behind Dots

We hide the password field with dots for security purposes. You can now click on the eye symbol to reveal the masked password.

Part of knowing your business is tracking sales metrics such as units sold and identifying your best customers. To that end, you'll probably want to begin with fetching data about customers who've made the most purchases throughout the month, quarter, year, or other time period. This data will allow you to analyze their buying patterns and identify trends. This blog will present a few sample queries to do that by combining the mighty Count() function with the GROUP BY and HAVING clauses.

A Basic Query

We'll be executing our queries against the Sakila Sample Database. It's a nicely normalized schema modeling a DVD rental store, featuring things like films, actors, film-actor relationships, and a central inventory table that connects films, stores, and rentals. Hence, its customers are not buying movies, but rather, renting them. Nevertheless, the criteria for selecting the data remains the same, which is to count the rows of the main rental table and group results by customer_id. Here is a basic query in Navicat Premium 16 that limits results to those customers who rented more than 20 movies in total:

That orders results by customer_id. Later on, we'll sort results by num_of_films_rented.

Fetching Additional Customer Details

While the above query is sufficient to identify those customers who rented many movies, it does not provide any customer details other than their IDs. To include more customer data, we need to join the customer table. It should be LEFT JOINed so that only customers who have rented movies are joined to the main query. Here are the results with customer names added:

Filtering Results

So far, we've been casting a very wide net, including results for all films and time periods. We could get more specific by targeting films by category as well as time period. To do that, we'll need to add a few more tables. If you're ever unsure on how to JOIN tables to a query, in Navicat, you can select the tables in the Object pane and run the Reverse Tables to Model... command:

That will add them to a schema diagram in the Modeling Tool so that you can view their relationships:

In the revised query, we'll limit results to comedies that were rented throughout 2005:

Notice that the minimum film count was lowered to 5 or more because there are less rentals for a single category.

Sorting By Count

Perhaps you'd rather view records by rental counts. All that's required to make that happen is to include an ORDER BY clause. Here is the final query, sorted by num_of_films_rented in DESCending order, so that the customer who rented the most comedies in 2005 appears at the top of the results:

Conclusion

In today's blog we learned how to combine the Count() function with the GROUP BY and HAVING clauses to gain valuable insight into our customers' spending habits. As you can imagine, the same query structure can be utilized to discover all sorts of trends and patterns related to product sales and/or rentals. Insights gleaned can be of tremendous benefit in guiding organizational decisions.

Having to select only odd or even rows from a table sounds like something that you'd never have to do, that is until you do. A quick Google search confirms that it's something that is done often enough, but, with few database practitioners knowing how, they invariably turn to online database communities in search of answers. As a reader of this blog, you can save yourself the trouble of scouring database forums for a solution, as we'll set the record straight right here today.

Picking a Suitable Target Column

Before we can speak of "even or odd rows" we have to order the rows by the column whose data we're interested in splitting. Ideally, its data should be numeric, unique, and sorted in ascending order. Hence, auto-increment columns like those of a primary key make perfect candidates. Otherwise, you may need to write a subquery with an ORDER BY clause and then select from it.

As an example, let's open the orders table of the classicmodels sample database in Navicat Premium 16's Table Designer. We can see that its PK (the orderNumber column) is not auto-incrementing, as evidenced by the unchecked "Auto Increment" checkbox:

However, opening the table in Grid View shows that orderNumber values are clearly sorted in ascending order:

Hence, we can write a query directly against the table.

Solutions By Database

The simplest way to find the records with odd or even values is to check the remainder when we divide the column value by 2. A remainder of 0 indicates an even number, while an odd number points to an odd number. However, like so many database tasks, how you go about determining the remainder depends on what type of database you're working with.

In PostgreSQL, MySQL, and Oracle, we can use the MOD() function to check the remainder:

Here's the general query syntax to find rows where a specified column has even values:

SELECT * FROM table_name WHERE mod(column_name,2) = 0;

This syntax will find rows where our target column has odd values:

SELECT * FROM table_name WHERE mod(column_name,2) <> 0;

SQL Server does not have a MOD function. Instead, it provides the % modulus operator.

Here's the general query syntax to find rows where a specified column has even values:

SELECT *FROM table_name where column_name % 2 = 0;

This syntax will find rows where our target column has odd values:

SELECT *FROM table_name where column_name % 2 <> 0;

Some Examples

Let's give each of the above statements a try against the orders table of the classicmodels sample database, first in SQL, then in SQL Server.

First, we'll retrieve even rows:

Next, we'll fetch odd rows only:

As mentioned previously, SQL Server does not have a MOD function, so well employ the % modulus operator instead.

Even rows:

Odd rows:

Conclusion

This blog presented an easy way to retrieve odd or even numbered rows from various databases by checking the remainder after dividing the target column value by 2 - a solution that is both simple and effective.

Supported Platforms and Databases, Plus SQL Editing

In this three part series, we've been comparing HeidiSQL, a free database client, with Navicat Premium. So far we've done a quick visual comparison and looked at both tools' pros and cons. In this final instalment, we'll be examining specific features, such as supported platforms and databases, SQL Editing, and more!

Supported Platforms

HeidiSQL was built for the Windows platform, and still only works on Windows. On the download page, there is a 32/64 bit combined (SHA1 checksum) Installer, Portable (zipped) versions for 32 bit and 64 bit, as well as the full source code. You can run HeidiSQL on Wine. Short for "Wine Is Not an Emulator", Wine is a compatibility layer for running Windows applications on several POSIX-compliant operating systems, such as Linux, macOS,and BSD. On Wine, HeidiSQL runs fine on Windows 8 and 10, but has minor issues on both Windows 7 and 11. Moreover, running HeidiSQL on any Wine release newer than 4.0 is currently quite unstable.

Navicat Premium is available for Windows (32 and 64 bit), macOS (64 bit), and Linux (64 bit). As such, each version of Navicat is optimized for that O/S. As a commercial product, Navicat comes with customer support. Users can also submit a ticket in the rare instance that they encounter a bug, to receive assistance in resolving it and have it fixed in the next patch or minor release.

Supported Databases

Initially, HeidiSQL offered support for MySQL and MariaDB, then added MS SQL Server. Now it includes PostgreSQL support as well.

Meanwhile, Navicat Premium is a Universal Database Tool, which means that it supports all popular databases, including MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite. It is also compatible with cloud databases, such as Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas.

SQL Editing

Both Navicat and HeidiSQL's Query Editors are similar in terms of functionality. Both offer auto-completion and customizable Code Snippets that strip the repetition from coding. Here's a side-by-side comparison with HeidiSQL on the left and Navicat Premium on the right:

Each tool includes common SQL statements, functions, code snippets in the right-hand pane. Here they are with HeidiSQL again on the left and Navicat Premium on the right:

One Navicat tool that is absent in HeidiSQL is the Visual Query Builder. It allows anyone to create and edit queries with only a cursory knowledge of SQL:

A Visual Query Builder is a must-have for many users, as is evidenced by a forum thread about that very subject on the HeidiSQL site.

Conclusion

In part 3 of this series on HeidiSQL vs. Navicat Premium, we examined specific features of both products, such as supported platforms and databases, as well as SQL Editing. While both share many similarities, there is no question that Navicat offers a more comprehensive array of tools and features, from the Visual Query Builder, to Navicat Cloud, to a dedicated support team. As we saw when we previously compared DBeaver to Navicat, Navicat Premium perfectly exemplifies the age old adage that you get what you pay for.

General Comparison

While HeidiSQL and Navicat Premium share many similarities, they are in fact different product types. Navicat Premium is what's known as a Universal Database Tool, which means that it supports ALL popular databases, including MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite. Moreover, Navicat is compatible with cloud databases as well, such as Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas. HeidiSQL began as a MySQL/MariaDB client and evolved to suport a few additional database types. That being said, the two products are homogenous enough to warrant a side by side comparison. In this installment we'll be taking a high level inventory of pros and cons, while the next part will focus on specific features.

Visual Interface

A quick look at an application's visual interface, or UI/UX, as it's more commonly known, can instantly give us some idea about how easy or difficult the application might be to work with. With that in mind, here's a screen capture of HeidiSQL's Data view:

There is no question that the HeidiSQL UI is chock full of information. Perhaps a little too much, as some elements run out of room at smaller viewport sizes:

Aside from minor glitches, the overall design is excellent; there is a main toolbar for accessing common functionality, and the bottom pane shows all database commands in real time.

Navicat 16 saw a lot of changes to the GUI. In fact, it was completely revamped with the goal of improving the usability and accessibility, allowing the user to accomplish complex tasks faster than ever before:

Navicat shows the latest database command at the bottom of the screen, and also includes additional table, column, and DDL information in the right pane.

Some Quick Pros and Cons

Now, let's run through some pros and cons of each product. First, HeidiSQL:

Pros:

• It's lightweight.
• Connects to multiple servers in one window.
• Free to use Licensed under GNU GPL. The source code is also available.
• Available in a portable version.
• Full database user roles and privileges management.
• Write queries with customizable syntax-highlighting and code-completion.
• Data synchronization. HeidiSQL can compare and synchronize your data and structure between local and remote databases.
• SSH tunneling support

Cons:

• Low stability. HeidiSQL is known to have a lot of bugs that result in frequent crashes.
• It's only available for Windows and it doesn't look like a cross-platform is coming anytime soon.
• No built-in debugger included
• Low DPI display. (DPI stands for dots per inch and it determines clarity and crispness to a display.) The author attempted to add high DPI but wound up dropping it.

And now, Navicat Premium:

Pros:

• It's cross-platform and supports multiple drivers.
• Data and structure synchronization.
• Visual query builder and report builder.
• Excellent import/export capabilities.
• SSH tunneling and SSL (Secure Sockets Layer) support
• Supports many languages, including Polish, Russian, Japanese, Portuguese, Korean, Simplified Chinese, Traditional Chinese, Spanish, French, and English.
• Compatible with other Navicat products, including Navicat Monitor, Navicat Data Modeler, Navicat Report Viewer, and Navicat Data Modeler Essentials.

Cons:

• It's a commercial product. Perhaps a con if you're on a very limited budget and need to work with multiple database types, such as PostgreSQL, SQL Server, or SQLite. In that case, you would have to purchase the Navicat premium package.
• It's somewhat resource-intensive as it requires fairly high memory while running.

Coming Up...

In part 3, we'll be examining specific features, such as supported platforms and databases, as well as SQL Editing.

Meet the Contestants

There is no shortage of either free or commercial relational database clients. Some provide basic functionality, while others offer advanced tools that help professionals achieve their many day-to-day activities in an efficient manner. While there is some correlation between cost and the number of features offered, each product needs to be evaluated on its own merits when deciding which product(s) to use yourself.

Back in June of last year, we compared DBeaver, a popular free tool, to Navicat Premium 15. Now, it's high time that we did it again. In this installment of Database Tools Showdown, we'll be taking a look at another freebie called HeidiSQL, and see how it stacks up against Navicat Premium 16.

Some Product Background

First, the challenger:

Invented by Ansgar Becker in 2002, his aim was to create software that was easy to learn. Becker chose the name "HeidiSQL" when it was suggested to him by a friend as a tribute to Heidi Klum. The name also reflected Becker's own nostalgia for Heidi, Girl of the Alps. "Heidi", as it's affectionately known, lets you view and edit both the data and structures of MariaDB, MySQL, Microsoft SQL, PostgreSQL and SQLite.

HeidiSQL began as a MySQL front-end in 1999 under the project name "MySQL-Front". In 2004, during a period of inactivity, Becker sold the MySQL-Front branding to his business partner, Nils Hoyer. In April 2006, Becker open-sourced the application on SourceForge, renaming the project "HeidiSQL". Later, he added support for other database servers as follows:

• Microsoft SQL Server support was added in March 2011 for the 7.0 release.
• PostgreSQL was introduced in March 2014 for the 9.0 release.
• SQLite support was introduced in March 2020 for the 11.0 release.

Today, HeidiSQL is routinely ranked among the most popular tools for MariaDB and MySQL worldwide. Since the 8.0 release, HeidiSQL offers its GUI in about 22 languages other than English.

About the reigning champion:

Navicat Premium is Navicat's flagship product. It's a commercial database development and design tool that allows users to simultaneously connect to multiple local and/or cloud databases from a single application. Navicat Premium was designed to meet the needs of a variety of audiences, from database administrators and programmers to various businesses/companies that serve clients and share information with partners.

The initial version of Navicat was developed by Mr. Ken Lin in 2001. The main goal of the initial version was to simplify the management of MySQL instances. In 2008, Navicat for MySQL was the winner of the Hong Kong ICT 2008 Award of the Year, Best Business Grand Award and Best Business (Product) Gold Award. Navicat Premium was launched in 2009. It combined all previous Navicat versions into a single product and could connect to all popular database types simultaneously, giving users the ability to perform data migration between different (heterogeneous) database types.

Going Forward

Now that we've properly introduced our contestants, the next installment(s) will delve into each tool's feature set, and compare them for usability, performance, user ratings, and more!

A CSV is a Comma-Separated Values file, which allows data to be saved in a tabular format. It's long been the preferred format for transferring data between databases. More recently, Internet-driven formats such as XML and JSON have also gained much traction. CSV files are well suited to databases because they represent table data exceptionally well and can be used with just about any spreadsheet program, such as Microsoft Excel or Google Spreadsheets. In today's blog, we'll be taking a look at a few ways to export table data to CSV in MySQL.

Using the Command Line

Most relational databases, MySQL included, provide commands to export and import to and from CSV.

Make sure that you start your MySQL server instance with the secure-file-priv option. It sets the directory where MySQL imports and exports data using statements such as LOAD DATA and SELECT INTO FILE. You can see the current setting using the command:

SHOW VARIABLES LIKE "secure_file_priv"  

All that's left to do now is select the data and specify the location of the output file. Here's a statement that outputs an entire table:

TABLE tableName INTO OUTFILE 'path/outputFile.csv'FIELDS TERMINATED BY ','OPTIONALLY ENCLOSED BY '"'ESCAPED BY ''LINES TERMINATED BY '\n';

You can also filter the data as you would in any SELECT query. Here's an example that filters both columns and values:

SELECT columnName, ...FROM tableNameWHERE columnName = 'value'LIMIT 1000INTO OUTFILE 'path/outputFile.csv'FIELDS TERMINATED BY ','OPTIONALLY ENCLOSED BY '"'ESCAPED BY ''LINES TERMINATED BY '\n';

Want to include column headers? That's easily done using the UNION statement:

(SELECT 'columnHeading', ...)UNION(SELECT column, ...FROM tableNameINTO OUTFILE 'path-to-file/outputFile.csv'FIELDS ENCLOSED BY '"' TERMINATED BY ','ESCAPED BY '"'LINES TERMINATED BY '\n')

Using mysqldump

mysqldump is a command line utility provided by MySQL for exporting tables, databases, and entire servers. Moreover, it can also be utilized for backup and recovery. Issue the following command in a command prompt/terminal to export a table:

mysqldump -u [username] -p -t -T/path/to/directory [database] [tableName] --fields-terminated-by=,

Using Navicat's Export Wizard

Navicat 16 for MySQL comes with a very powerful export (and import) wizard, which can export data in multiple formats, including .xlsx, .json, and .sql. To start the export wizard, select the corresponding table, right-click > Export Wizard, and select the format:

You can choose to export one table, the entire database, or anything in between:

You can also select exactly which fields you want, if you're not interested in all the columns:

Navicat supports a wealth of options, such as including headers, delimiters, error handlers, and more:

Conclusion

CSV is not the perfect format, and does have limitations. For example, you cannot save data types or formulas in this format. That being said, CSV is still a very important data transfer format; one that every DBA should be familiar with.

In some cases, running a well crafted UPDATE statement in production can save the day. Other times, a botched UPDATE can cause more harm than the initial issue. You can always execute your Data Manipulation Language (DML) statements on a development or test database, but due to differences in the data, this approach makes determining the statement's effects on the production data a craps shoot at best.

So what are some options to accurately predict what the result of an INSERT, UPDATE, or DELETE statement will be on production data before running it? Well, that depends on the database vendor and product, at least in part. There are also some solutions that enjoy widespread support. We'll be taking a look at both options in this blog.

Syntax Check

The process of testing your statements can be split into two stages. The first is to verify that the statement is syntactically valid, I.E. it will execute. The next step is to ascertain whether or not it produces the result that you intended.

One way to validate your syntax is to ask your database (DB) for the query plan. This tells you two things:

• Whether there are any syntax errors in the query; if so the query plan command itself will fail.
• How the DB is planning to execute the query, e.g. what indexes it will use.

In most relational DBs the query plan command is "explain" or "describe", as in:

explain update ...;

In Navicat's database administration and development tools, you can run the EXPLAIN command with the click of a button. If the statement fails, you'll get an error message similar to the following:

Otherwise, the query plan will be displayed in a tabular format:

Statement Testing

You can parse a statement to see if it's syntactically valid, but that doesn't mean it will produce the correct results. To see what your query will actually do, you've got a few options.

Turn Off Autocommit

Most relational DBs provide a way to disable autocommit mode so that you must issue the COMMIT statement to store your changes to disk or ROLLBACK to ignore the changes.

In MySQL the command to disable autocommit mode is:

SET autocommit=0OrSET autocommit = OFF

In SQL Server, the command is:

SET IMPLICIT_TRANSACTIONS OFF

With autocommit turned off, you are now ready to give your statement(s) a try, by running it within a transaction:

-- 1. start a new transactionSTART TRANSACTION;-- 2. insert a new order for customer 145INSERT INTO orders(orderNumber,                   orderDate,                   requiredDate,                   shippedDate,                   status,                   customerNumber)VALUES(@orderNumber,       '2005-05-31',       '2005-06-10',       '2005-06-11',       'In Process',        145);        -- 3. then, after evaluating the results,--    rollback the changesROLLBACK;

That will leave your DB in exactly the same state as it was before you ran your statement.

Convert Your Statement Into a SELECT

A decidedly low tech approach to testing DML statements is to convert them to SELECTs. As long as you don't expect them to retrieve the entire database, running them as SELECTs is a good way to see exactly which records will be affected. All you need to do is replace the action word with SELECT:

INSERT INTO orders...BECOMESSELECT * FROM ORDERS...

Conclusion

There are few things scarier than executing DML statements in a production environment. Thankfully, there are ways to minimize the risk so that you don't have to cross your fingers or recite the Hail Mary.

If you'd like to give Navicat 16 a test drive, you can download a 14 day trial here.

Just when you thought you knew every type of join, here comes another! Perhaps you've heard of nested joins, or even nested-loop query plans, and wondered what they were. Well, wonder no more. Today's blog will settle the mystery once and for all!

A Case of Terminology

In the world of relational databases, there can be many different names for the same thing. Joins are no exception to this rule. In fact, when it comes to Nested Joins, database practitioners vary as per their opinions on them. Some say that there is no such thing; others are more pragmatic and acknowledge that they are simply an alternative term for multi-table joins.

In all likelihood, the term came about when referring to nested-loop query plans. These are often used by the query engine to answer joins. In its crudest form, a nested loop goes something like this:

for all the rows in outer table   for all the rows in the inner table     if outer_row and inner row satisfy the join condition       emit the rows   next inner next outer

This is the simplest, but also the slowest type of nested-loop. Meanwhile, multi-table nested-loop joins perform even worse because they scale out horrendously as the product of the number of rows in all the tables involved grows.

A more efficient form of nested loop is nested-loop-over-index:

for all the rows that pass the filter from the outer table   use join qualifier from outer table row on index on inner table     if row found using index lookup       emit the rows next outer 

Nested Join Syntax

Now that we've established that the term "Nested Joins" simply refers to joins between more than two tables, let's take a quick look at their syntax.

Typically, when we need to join multiple tables and/or views, we would list them one by one, using this generic format:

FROM  Table1 [ join type ] JOIN Table2                 ON condition2 [ join type ] JOIN Table3                 ON condition3 

But this is not the only way. The official ANSI syntax standard for SQL proposes another valid way to write the above join:

FROM  Table1 [ join type ] JOIN Table2 [ join type ] JOIN Table3                 ON condition3                 ON condition2 

To make the above join style more human readable, we can add parentheses and indentation to make the meaning clearer:

FROM  Table1 [ join type ] JOIN ( Table2                     [ join type ] JOIN Table3                                     ON condition3 )                 ON condition2 

Now it's easier to see that the join between Table2 and Table3 is specified first and has to be done first, before joining to Table1. This query style also positions the join between Table2 and Table3 in such a way that they appear to be nested. In fact, we could consider the join between Table2 and Table3 to be nested.

Here is the same query in Navicat Premium written in both styles:

Conclusion

Today's blog shed some light on the terms "Nested Joins" and "Nested-loop Query Plan". So, next time you hear them, realize that they are merely referring to the joining of multiple tables or views, regardless of which syntax is employed in doing so.

Having completed our series on Top SQL Query Mistakes last week, it's time to take a page from the Monty Python playbook and move on to something completely different. And that something is why database developers and administrators should consider using third-party database administration tools (DBMT) to fill the gaps left by the major database manufacturers. Regardless of price, all 3rd party DBMT provide functionality that fulfills the needs of the general DBA community by either complimenting or replacing database manufacturers' tool sets. Today's blog will highlight just a few of the benefits provided by 3rd party DBMT.

Heterogeneous DBMS Support

It's a fairly rare occurrence to find an IT organization that supports a single DBMS platform these days. Most businesses utilize several different database types - both locally hosted and in the Cloud. For example, my own employer has some local PostgreSQL databases as well as some of Amazon's online DB services. The local DB instances are well suited to development, testing, and certain production uses. Meanwhile, Amazon database services pair well with other Amazon services, such as AI, batch processing, etc...

A growing class of third-party tool providers is capitalizing on this DBMS proliferation by providing tools that are purposely designed to work with a variety of database types. Their value stems from their ability to manage multiple DBMS from a single application interface. Some products, such as Navicat Premium, can even connect to multiple heterogeneous DBMS simultaneously, allowing admins to transfer data between databases as easily as copying files on a PC desktop.

Increased Data Security

An organization's data is invariably its greatest asset. As such, most businesses place a high premium on security and choose 3rd party DBMT that provide more secure connection options than those offered by individual DB vendors' own administration and development tools.

One popular feature is SSH tunneling. It's a method of transporting data that employs an encrypted SSH connection. SSH tunnels allow connections made to a local port to be shuttled to a remote machine via a secure channel.

3rd party DBMT also add value by supporting multiple authentication methods such as PAM authentication for MySQL and MariaDB, Kerberos and X.509 authentication for MongoDB, and GSSAPI authentication for PostgreSQL. High end products like Navicat provide more authentication mechanisms and high-performance environments so you never have to worry about connecting over an insecure network.

Collaboration Support

One feature that is almost universally missing from DB vendor administration and development tools is the ability to share queries and such with team mates. Navicat's main collaboration tool is Navicat Cloud. It uses Amazon Simple Storage Service (Amazon S3) to store (256-bit AES) encrypted connection settings, queries, models, snippets, virtual group information, and even chart workspaces. These may be shared as well as synchronized across all your devices including Windows, macOS, Linux, and iOS. Files stored in Navicat Cloud automatically show up in Navicat so that you can get real-time access at anytime and from anywhere.

Conclusion

This blog presented three ways that 3rd party Database Management Tools such as Navicat Premium provide value to organizations by either complimenting or replacing database manufacturers' tool sets. In doing so, they can greatly simplify everyday tasks and promote increased productivity.

Predicate Evaluation Order

Just before Part 3 of this series, we took a brief pause to talk about Predicates in SQL, as they factored into mistakes related to Outer Joins. In this final installment of this series on Top SQL Query Mistakes, predicates will once again enter the picture, as we examine how predicate evaluation order can cause seemingly well constructed queries to fail with errors.

A Quick Review of Predicate Processing Order

In terms of logical query processing order, queries are executed in the following sequence:

• FROM
• WHERE
• GROUP BY
• HAVING
• SELECT

Hence, logically speaking, the FROM clause is processed first to define the source data set. Next, the WHERE predicates are applied in order to whittle down the result set, followed by GROUP BY, and so on.

In practice, predicate evaluation and processing order is far less rigid, as the query optimizer may move expressions in the query in order to produce the most efficient plan for retrieving the data. As a result, a filter in the WHERE clause may not be applied before the next clauses are processed. In fact, a predicate may be applied much later in the physical execution plan than you might expect.

Another common source of confusion and frustration to database developers is that, unlike with most programming languages, predicates are not always executed from left to right. This means that, if you have a WHERE clause containing the filters "WHERE a=1 AND b=2", there is no guarantee that "a=1" will be evaluated first. In fact, there is no easy way to tell which order filters will be executed simply by looking at the query.

A Practical Example

To better understand predicate evaluation order, we'll write a SELECT query against the following accounts table, seen in Navicat 16's Table Designer:

Here is some sample data that we'll be querying against:

In the account_number column, business accounts are assigned a numeric identifier, while personal accounts are given an identifier made up of characters. This is not great table design, as the account_number column should be represented by two different fields, where each account type is given the correct data type and not share the same table. Nonetheless, altering the design is not always possible, so we must deal with the table as is.

So, with this in mind, let's devise a query to retrieve all business type accounts with an account_number that is greater than 50. The resulting query might look like this one:

In some databases, the query produces an error:

Conversion failed when converting the varchar value 'ACFB' to data type int

The query will fail any time that the query optimizer decides to prioritize the "CAST(account_number AS UNSIGNED INTEGER) > 50" predicate over the "account_type LIKE 'Business%'". The safest be for avoiding errors like the one above is to either:

• Design the table correctly and avoid storing mixed data in a single column.

  OR

• Use a CASE expression to guarantee that only valid numeric values will be converted to INTEGER data type, like this:

  

Conclusion

In this series on Top SQL Query Mistakes, we explored how seemingly intuitive ways of constructing SQL queries can result in anti-patterns that lead to erroneous results and/or performance degradation. Be especially wary of predicate placement and evaluation order as these contribute to many unexpected issues.

Breaking Subqueries

In this series on Top SQL Query Mistakes, we've seen several examples of SQL queries that look perfectly solid on first inspection, but can lead to erroneous results and/or performance degradation. Last week, learned how the placement of predicates can adversely affect query execution - particularly in outer joins. Today's installment will focus on subqueries, and how they can break an SQL statement when changes are made to any of its underlying tables.

Single vs. Multiple Value Subqueries

Even before we compare single and multiple value subqueries, we should briefly cover what a subquery is. A subquery is a complete SQL query that is nested inside a larger query. A subquery may be placed in the SELECT, FROM, and WHERE clauses.

Now that we know what a subquery is and where it can go in a query, it should be noted that, like any SELECT query, a subquery may return one or more rows. This distinction is quite important, because it affects how you would write your query statement. For example, here's a query against the Sakila Sample Database in Navicat Premium 16 that fetches all of the actors who appeared in the film "ALONE TRIP":

Since there should only be one film named "ALONE TRIP", we can use the equals (=) operator to match the film_ids against.

Contrast the above query to the following one:

In this case, the subquery selects all of the actors who appeared in the movie. Naturally, this subquery would return multiple rows. In that case, we should employ the IN() function to match actor_ids against.

How Single Row Subqueries Break

As mentioned earlier, a subquery can be placed in the SELECT clause to fetch a column that is in some way correlated to the main query table. For example, consider these two related products and factories tables, shown in the Navicat Data Modeler:

The products and factories tables are linked using the common sku field.

Now, let's write a query to extract the factory_id for each product. One way to do that would be to write the query using correlated subquery to retrieve the product factory_id:

Note that the point here is to illustrate a technique; there are more efficient ways to retrieve the same information. In any event, we do get the correct result set, and all is well.

The query will continue to work perfectly well until the day arrives that the company decides to build a new factory as sales increase:

The extra row in the factories table causes our query to generate an error now:

The error is telling us that the outer query expected a scalar value, but our subquery returned a result set. We can fix the issue and list all factories that manufacture each product by using a JOIN:

One More Thing...

Be aware that the same error can occur in any clause where a column or expression is tested against a subquery, for example "column = (SELECT value FROM Table)". In that case, the solution is to use the IN() function instead of the equality (=) operator.

Outer Joins and Cartesian Products

In this series on Top SQL Query Mistakes, we've been exploring how seemingly intuitive ways of constructing SQL queries can result in anti-patterns that lead to erroneous results and/or performance degradation. Last week, we took a break from the series to talk about Predicates in SQL. In this installment, we'll be learning how their placement can adversely affect query execution - particularly in outer joins.

What Are Outer Joins?

There are four basic join types employed in linking related tables and views: inner, left, right, and outer. With an inner join, rows from either table that are unmatched in the other table are not returned. In an outer join, unmatched rows in one or both tables can be returned. The last three join types are all instances of outer joins, whereby:

• LEFT JOIN returns only unmatched rows from the left table.
• RIGHT JOIN returns only unmatched rows from the right table.
• FULL OUTER JOIN returns unmatched rows from both tables.

How Outer Joins Go Wrong

While outer joins certainly have their place in the database practitioner's arsenal, developers have a tendency of using them even in situations where they are not needed. Moreover, an outer join query can produce completely different results depending on how you construct it, and where you place predicates in your query. To illustrate, let's look at an example.

We would like to retrieve a list of ALL customers (whether they placed any orders or not), along with the total number of orders that they placed since the beginning of June, 2005. Do do so, we would employ an outer join to link the customers and orders tables as follows:

SELECT C.customerName, count(O.customerNumber) AS 2005_ordersFROM customers AS CLEFT OUTER JOIN orders AS O  ON C.customerNumber = O.customerNumberWHERE O.orderDate >= '2005-05-01'GROUP BY C.customerNameORDER BY 2005_orders DESC;

The result should contain every possible combination of rows from the first and second table, also known as a Cartesian product. Unfortunately, when we run the query in Navicat Premium 16, only 13 rows are returned, even though there are 122 unique customers in the table (not shown):

To understand where we went wrong, let's rebuild the query one step at a time, starting with only the columns and outer join:

Now we are getting all of the customers. Those who have not placed any orders have NULL customerNumbers, since they are coming from the orders table.

Now, let's apply the WHERE clause predicate:

All of sudden, we've lost many customers! The problem is that the predicate in the WHERE clause turned the outer join into an inner join.

To correct the issue, we need to add the WHERE predicate to the join condition:

We can now adjust our original query to fetch all customers:

The Moral of the Story

Always be careful of where you filter out rows. In the above example, the WHERE clause was the issue; in a more complex example, where multiple joins occur, the incorrect filtering may happen on a subsequent table operator (like join to another table) instead in the WHERE clause.

This week, we're going to briefly hit the Pause button from the Some Top SQL Query Mistakes series in order to talk about Predicates in SQL. The reason is that Predicates will factor into Part 3 of the Top SQL Query Mistakes series.

What Is a Predicate?

A predicate is simply an expression that evaluates to TRUE, FALSE, or UNKNOWN. Predicates are typically employed in the search condition of WHERE and HAVING clauses, the join conditions of FROM clauses, as well as any other part of a query where a boolean value is required.

There are numerous types of predicates, including:

• Comparison
• LIKE
• BETWEEN
• IN
• EXISTS
• IS NULL (/INTEGER/DECIMAL/FLOAT...)

In the remainder of this article, we'll examine a few examples of the above Predicate types.

Comparison Predicates

Any time that we use a comparison operator in an expression, such as WHERE employee_salary > 100000, we are constructing a Predicate that evaluates to TRUE, FALSE, or UNKNOWN. Comparison operators include:

• =Equal to
• >Greater than
• < Less than
• >= Greater than or equal to
• <= Less than or equal to
• <> Not equal to

Hence, a comparison predicate takes the form of:

expression_1 comparison_operator expression_2

In a comparison predicate, expression2 can also be a subquery. If the subquery does not return any rows, the comparison predicate evaluates to FALSE.

LIKE Predicate

IN SQL, the number one pattern-matching predicate is the LIKE operator, as it compares column values with a specified pattern. Like works with any character or date data type. Here's an example:

BETWEEN Predicate

The BETWEEN operator specifies a range, which determines the lower and upper bounds of qualifying values. For instance, in the Predicate income BETWEEN 5000 AND 20000 the selected data is a range of greater then or equal to 5000 and less then or equal to 20000. The Between operator can be used with numeric, text and date data types. Here's an example:

IN Predicate

An IN operator allows the specification of two or more expressions to be used for a query search. The result of the condition is TRUE if the value of the corresponding column equals one of the expressions specified by the IN predicate:

EXISTS Predicate

The EXISTS Predicate accepts a subquery as an argument. It returns TRUE if the subquery returns one or more rows, and returns FALSE if it returns zero rows.

Here's an example:

IS NULL Predicate

Use IS NULL to determine whether an expression is null, because you cannot test for null by using the = comparison operator. When applied to row value expressions, all elements must test the same.

The IS NULL predicate takes the following form:

IS [NOT] NULL

For example, the expression x IS NULL is TRUE if x is a null.

IS UNKNOWN is a synonym for IS NULL when the expression is of the BOOLEAN type.

Here's a query that uses the IS NOT NULL Predicate to fetch all actors whose last name is a non-NULL value:

Conclusion

In this blog, we interrupted the regularly scheduled blog to bring you this important lesson on SQL Predicates. Typically employed in the search condition of WHERE and HAVING clauses, the join conditions of FROM clauses, Predicates are expressions that evaluate to TRUE, FALSE, or UNKNOWN. We'll be seeing Predicates again in next weeks continuation of the Top SQL Query Mistakes series.

Part 2: Non-SARGable Query Conditions

Like most programmers, database developers tend to write code that is more or less a direct translation of a given request. The fact that most programming languages - SQL included - are designed to be human readable, also contributes to this problem. Why is this a concern? All programming languages execute certain operations faster than others. In relational databases, the query optimizer analyzes SQL queries and determines the efficient execution mechanisms called query plans. The optimizer generates one or more query plans for each query, each of which represent one possible way to run a query. The most efficient query plan is then selected and utilized to run the query. As it turns out, SQL that mimics the language of a request is seldom the most efficient approach.

In this installment of the Top SQL Query Mistakes series, we'll explore one example of a poorly written SQL statement and rewrite it in a way that increases efficiency.

Passing Indexed Columns to Functions

One faux pas that comes up over and over again in database developers' code is the passing of index columns to functions. To illustrate, let's execute a query against this table, which has an index on the varchar customerName column:

When asked to retrieve all customers whose name starts with the letter "R", one might be inclined to use the LEFT() function to return the first character of the customerName column:

Unfortunately, by passing the indexed customerName column to a function, the query engine must evaluate its result for every row in the table!

SARGable vs. Non-SARGable Queries

In relational databases, there is a term that is derived from a contraction of Search ARGument ABLE, aka, SARGable. A condition (or predicate) in a query is said to be SARGable if the DBMS engine can take advantage of an index to speed up the execution of the query. On the other side of the coin, a query that fails to be SARGable is known as a non-SARGable query. The effect is similar to searching for a specific term in a book that has no index, beginning at page one each time, instead of jumping to a list of specific pages identified in an index. Obviously, this has a negative effect on query time, so one of the steps in query optimization is to convert such conditions to be SARGable.

To make a condition such as the one above into a SARGable one, we need to avoid the use of functions on the indexed columns. To do that, we must express the request with this logically equivalent (and SARGable) query, using the Like operator:

Notice the greatly improved execution time.

Conclusion

In this second installment on Top SQL Query Mistakes, we learned how non-SARGable query conditions can degrade query performance by forcing the database engine to evaluate every row of a table. The fix is to express the request with a logically equivalent (and SARGable) condition that does not rely on a function call.

If you'd like to give Navicat 16 for MySQL a test drive, you can download a 14 day trial here.

NOT IN Versus NOT EXISTS

There's a term the is commonly thrown around in programming circles called "anti-patterns". It refers to a response to a recurring problem that is not only ineffective, but also risks being highly counterproductive. The term was originally coined in 1995 by computer programmer Andrew Koenig, in his book Design Patterns, as the antithesis of design patterns that are considered to be both reliable and effective.

Although SQL is not really a programmatic language, it turns out that it's equally susceptible to anti-patterns, especially when the query in question is fairly complex. Sometimes mistakes are hard to spot, and do not reveal themselves until the query is thrust into the pressure cooker of the production environment.

With the goal of catching SQL mistakes earlier, the next several blogs will be devoted to highlighting some of the most common culprits. We'll be using MySQL to execute today's examples, but the concepts are equally valid in any flavor of SQL.

NOT IN Versus NOT EXISTS

One common type of SELECT query retrieves data that is not included in a list of values. To illustrate, here are two very simple tables created in Navicat for MySQL 16. The first table contains colors:

The second table contains products:

What we would like to do is select all of the colors that have not yet been associated to any products. In other words, we need to construct a query that returns only those colors for which there is no product with that color. One might be tempted to employ the NOT IN predicate to fetch the records in question.

We would expect the following query to return two rows (for "black" and "green") when, in fact, an empty result set is returned:

The problem? The present of a NULL value in the color column on the products table, which are translated by the NOT IN predicate to:

color NOT IN (Red, Blue, NULL)

OR

NOT(color=Red OR color=Blue OR color=NULL)

The expression "color=NULL" evaluates to UNKNOWN and, what many database developers overlook, is that NOT UNKNOWN also evaluates to UNKNOWN! As a result, all rows are filtered out and the query returns zero rows.

This issue can also surface if requirements change, and a non-nullable column is updated to allow NULLs. Hence, even if a column disallows NULLs in the initial design, you should make sure your queries will continue to work correctly with NULLs, should things ever change.

The simplest solution is to use the EXISTS predicate instead of IN:

Problem solved!

So why does this work? Whereas the IN keyword will compare all values in the corresponding subquery column, EXISTS evaluates true or false. Consequently, using the IN operator, the SQL engine will scan all records fetched from the inner query. On the other hand, if we are using EXISTS, the SQL engine will stop the scanning process as soon as it found a match.

Conclusion

In this first installment on Top SQL Query Mistakes, we learned about how anti-patterns can occur in SELECT queries, starting with the erroneous use of the NOT IN predicate.

If you'd like to give Navicat 16 for MySQL a test drive, you can download a 14 day trial here.

Querying by Date

In this final installment in this series on Dates and Times in MySQL, we'll be putting everything we've learned thus far into practice by writing SELECT queries to obtain date-related insights into our data.

Selecting a Date from a Datetime Column

One of the first challenges database practitioners encounter when trying to query with dates is that a good deal of temporal data is stored as DateTime and Timestamp data types. For example, the Sakila Sample Database stores the customer table's create_date column as a Datetime:

Hence, if we try to select customer records that were created on a specific date, we can't simply supply a date value:

One simple workaround is to convert the Datetime values to Dates by using the DATE() function:

Now any record whose date matches ours will be returned.

Obtaining the Difference Between Two Dates

It is extremely common to perform queries that determine how long ago something happened. In MySQL, the way to do that is to employ the DATEDIFF() function. It accepts two date values and returns the number of days between them. Here's a simple example using Navicat for MySQL 16:

Notice that, in the above example, DATEDIFF() is telling us that the first date is 10 days later than the second one. We can also use an earlier date for the first argument and it will return a negative value:

Calculating Periods Other than Days

For periods other than days, we need to do a little conversion. For example, we can divide by 7 to obtain the number of weeks between two dates. Rounding is also employed to show whole weeks in the results:

ROUND(DATEDIFF(end_date, start_date)/7, 0) AS weeksout

For other time periods, the TIMESTAMPDIFF() function may be of help. It accepts two TIMESTAMP or DATETIME values (DATE values will auto-convert in MySQL) as well as the unit of time we want to base the difference on. For instance, we can specify MONTH as the unit in the first parameter:

SELECT TIMESTAMPDIFF(MONTH, '2012-05-05', '2012-06-04')-- Outputs: 0SELECT TIMESTAMPDIFF(MONTH, '2012-05-05', '2012-06-05')-- Outputs: 1SELECT TIMESTAMPDIFF(MONTH, '2012-05-05', '2012-06-15')-- Outputs: 1SELECT TIMESTAMPDIFF(MONTH, '2012-05-05', '2012-12-16')-- Outputs: 7

A More Complex Example

Once you've got the hang of the DATEDIFF() function, you can using it in more advanced ways. Case in point, here's a query that uses the DATEDIFF() function to calculate the average number of days that customers keep their film rentals before returning them:

To do that, the results of the DATEDIFF() function is passed to the AVG() function and then rounded to 1 decimal place.

Series Conclusion

We've covered a lot of ground in this series on Dates and Times, including:

• MySQL's five temporal data types
• some important date/time-oriented functions
• how to create dates and times in MySQL
• querying by date

While there certainly is a lot more to working with temporal data in MySQL, hopefully this series gave you a good head start on your road to MySQL proficiency.

Date/Time Creation

In this this series on Dates and Times, we've explored MySQL's five temporal data types, as well as some of its many date/time-oriented functions. In this installment, we'll be covering a few ways to create dates and times in MySQL.

Using the MAKEDATE() Function

In Part 3, we took a brief look at the MAKEDATE() function. It takes a year and dayofyear and returns the resulting Date value. For instance, MAKEDATE(2021, 200) would return a Date of "2021-07-19". The downside to this function should be readily obvious; it takes some calculation to determine the dayofyear if you have a year, month, and day. In that case, you can make a DATE by combining MAKEDATE() with DATE_ADD(). MAKEDATE() with a day of 1 will give you a DATE for the first day of the given year, and then you can add to it the month and day with DATE_ADD(). Here's an example that sets the year and month only:

This SELECT statement includes the day as well:

The MAKETIME() Function

If you're looking to create a TIME only, MAKETIME() returns a time value calculated from the hour, minute, and second arguments:

The second argument can have a fractional part for milliseconds:

The STR_TO_DATE() Function

Another option for creating a DATE, TIME, or DATETIME is to use the STR_TO_DATE() function. It takes a date string and a format string and returns:

• a DATE value if the string contains only date
• a TIME value if the string contains only time
• a DATETIME value if the format string contains both date and time parts

Moreover, if the date, time, or datetime value extracted from str is invalid, STR_TO_DATE() returns NULL and produces a warning.

Some Examples

Here are a couple of Dates in Navicat for MySQL 16:

Scanning starts at the beginning of str and fails if format is found not to match. Meanwhile, extra characters at the end of str are ignored:

Unspecified date or time parts have a value of 0, so incompletely specified values in the date/time string produce a result with some or all parts set to 0:

For the full list of specifiers that can be used in format, see the DATE_FORMAT() function description in the official MySQL docs.

Combining the MAKEDATE(), MAKETIME(), AND STR_TO_DATE() Functions

If we had two separate DATE and TIME values, we could get a DATETIME value by concatenating the the results of MAKEDATE() and MAKETIME() and then passing the combined string to STR_TO_DATE(). While that might sound like a lot of work, it's really quite simple in practice:

Conclusion

In this installment of the Working with Dates and Times in MySQL series, we covered a few ways to create dates and times in MySQL using some of its specialized date and time functions. In the next installment, we'll look at how to use temporal data in your SELECT queries.

Important Functions

In the first two installments of this series on Dates and Times, we covered MySQL's five temporal data types. Now it's time to turn our attention to some of MySQL's many date/time-oriented functions.

Getting the Current Date and Time

Back in May of 2021, we covered some of SQL Server's notable Date & Time functions, starting with how to obtain the current date and time. It offers the GETDATE() function for that purpose. MySQL's equivalent function is simply called NOW(). In Navicat for MySQL 16, we can invoke this function without connecting to a database, since we aren't selecting any table columns:

As mentioned in Part2, the TIMESTAMP type is similar to DATETIME, but are generally used to track changes to records. To obtain the current date and time as a TIMESTAMP, we can use the current_timestamp() function. Here's its output:

Getting the Current Date Without the Time

If you only want to get current date in MySQL, you can use either the curdate() or current_date() functions. The system variable current_date also works. In any event, all three give latest date in YYYY-MM-DD format:

Getting the Current Time Only

Likewise, we can get the current time in MySQL using the curtime() or current_time() functions, as well as the current_time system variable. These all give the latest time in HH:MM:SS format:

Parsing Out Individual Date Parts

SQL Server offers the versatile DATEPART() function to extract part of a datetime. MySQL provides the equivalent EXTRACT() function for this purpose. Similar to the SQL Server function, EXTRACT() accepts a part unit and the date:

EXTRACT(part FROM date)

Here are all of the valid part values:

• MICROSECOND
• SECOND
• MINUTE
• HOUR
• DAY
• WEEK
• MONTH
• QUARTER
• YEAR
• SECOND_MICROSECOND
• MINUTE_MICROSECOND
• MINUTE_SECOND
• HOUR_MICROSECOND
• HOUR_SECOND
• HOUR_MINUTE
• DAY_MICROSECOND
• DAY_SECOND
• DAY_MINUTE
• DAY_HOUR
• YEAR_MONTH

Being February at the time of this writing, the following call to EXTRACT() yields a value of "2":

As the following query shows, it is currently 43 minutes past the hour:

Additional Date Parsing Functions

Having trouble remembering all of the part units? That's OK, because MySQL provides separate functions for date and time parsing a well.

For parsing either the date or time from a datetime value, there are the DATE() and TIME() functions, respectively:

To split a date into its constituent parts, we can use the YEAR(), MONTH(), and DAYOFMONTH() (or DAY()) functions:

Time portions also get their own function: HOUR(), MINUTE(), and SECOND() respectively:

Constructing a Datetime From Separate Parts

In MySQL, there are many ways to create a datetime from separate date and time parts, enough to garner their own article. For now, let's look at one way to create a Date. The MAKEDATE() function returns a date given a year and dayofyear. Here's an example:

Going Forward

In this blog, we explored some of MySQL's many date/time-oriented functions. In the next installment, we'll cover some other ways to create dates and times in MySQL.

TIMESTAMP and YEAR Types

Welcome back to this series on working with dates and times in MySQL. In the first two installments, we're looking at MySQL's temporal data types. Part 1 covered the DATE, TIME, and DATETIME data types, while this installment will cover the remaining TIMESTAMP and YEAR types.

The TIMESTAMP Type

The TIMESTAMP type is similar to DATETIME in MySQL in that both are temporal data types that hold a combination of date and time. This begs the question why have two types for the same information? For starters, timestamps in MySQL are generally used to track changes to records, and are often updated every time the record is changed, whereas datetimes are used to store a specific temporal values. Another way to think about it is that DATETIME represents a date (as found in a calendar) and a time (as seen on a wall clock), while TIMESTAMP represents a well defined point in time. This distinction could be very important if your application handles timezones, as how long ago was '2009-11-01 14:35:00' depends on what timezone you're in. Meanwhile, 1248761460 seconds since '1970-01-01 00:00:00 UTC' always refers to the same point in time.

In terms of storage, a TIMESTAMP requires 4 bytes while DATETIME requires 5. TIMESTAMP columns store 14 characters, but you can display it in different ways, depending on how you define it. For example, if you define the column as TIMESTAMP(2), only the two-digit year will be displayed (even though the full value is stored). The advantage to this approach is that, if you later decide to display the full value, you can change the table definition, and the full value will appear.

Below is a list of various ways to define a TIMESTAMP, and the resultant display format:

• TIMESTAMP(14): YYYY-MM-DD HH:MM:SS
• TIMESTAMP(12): YY-MM-DD HH:MM:SS
• TIMESTAMP(10): YY-MM-DD HH:MM
• TIMESTAMP(8): YYYY-MM-DD
• TIMESTAMP(6): YY-MM-DD
• TIMESTAMP(4): YY-MM
• TIMESTAMP(2): YY

In the Navicat 16 Table Designer, a timestamp's precision may be defined in the Length column:

If no Length is supplied, as in the above example, Navicat displays the full field, as if it was declared as TIMESTAMP(14):

The YEAR Type

Many DBAs opt to store years as integers. While that can certainly work, it is more efficient to use MySQL's dedicate YEAR type for that purpose, as the YEAR type uses a mere 1 byte. It can be declared as YEAR(2) or YEAR(4) to specify a display width of two or four characters. If no width is given the default is four characters. YEAR(4) and YEAR(2) have different display formats but have the same range of values:

• For 4-digit format, MySQL displays YEAR values in YYYY format, with a range of 1901 to 2155, or 0000.
• For 2-digit format, MySQL displays only the last two (least significant) digits; for example, 70 (1970 or 2070) or 69 (2069).

Here's an example of a year column in the Navicat Table Designer with a four digit format:

As a result, we see the full year in the table:

Conclusion

That concludes our exploration of the five MySQL temporal data types. The next installment will cover some useful date and time functions.

DATE, TIME, and DATETIME Types

The vast majority of databases store a great deal of "temporal" data. Temporal data is simply data that represents a state in time. An organization may collect temporal data for a variety of reasons, such as to analyze weather patterns and other environmental variables, monitor traffic conditions, study demographic trends, etc. Businesses also routinely need to store temporal data about when orders were placed, stock refilled, staff hired, and a whole host of other information about their day-to-day business.

You may be surprised to learn that relational databases do not store dates and times in the same way. MySQL is especially prescriptive. For instance, It stores date values using the universal yyyy-mm-dd format. This format is fixed and may not be changed. You may prefer to use a mm-dd-yyyy format, but it is not possible to do so. However, you can use the DATE_FORMAT function to format the date the way you want in the presentation layer, usually an application. In the first two installments on working with Dates and Times in MySQL, we'll be looking at MySQL's temporal data types, starting with DATE, TIME, and DATETIME.

Types At a Glance

MySQL provides five types for storing dates and times, some just for dates, others for time, and some that include both. Here's a table that summarizes each type:

The rest of this article will cover the first three types in more detail, while the next one will focus on the other two.

TheDATE Type

MySQL uses 3 bytes to store a DATE value. The DATE values range from 1000-01-01 to 9999-12-31. Moreover, when strict mode is disabled, MySQL converts any invalid date e.g., 2015-02-30 to the zero date value 0000-00-00. In Navicat 16, you can select the DATE type in the Table Designer from the Types drop-down:

To set a DATE value, you can simply choose it using the calendar control:

Of course, you can also insert a DATE using the INSERT statement:

TheTIME Type

MySQL uses the 'HH:MM:SS' format for querying and displaying a time value that represents a time of day, which is within 24 hours. To represent a time interval between two events, MySQL uses the 'HHH:MM:SS' format, which is larger than 24 hours.

Here is the TIME type in the Navicat 16 Types drop-down:

To set a TIME value, Navicat provides the TIME INPUT control:

Here's an INSERT statement that sets a start and end time:

TheDATETIME Type

Quite often, you'll need to store both a date and time. To do that, you can use the MySQL DATETIME type. By default, DATETIME values range from 1000-01-01 00:00:00 to 9999-12-31 23:59:59. When you query data from a DATETIME column, MySQL displays the DATETIME value in the same YYYY-MM-DD HH:MM:SS format.

A DATETIME value uses 5 bytes for storage. In addition, a DATETIME value can include a trailing fractional second up to microseconds with the format YYYY-MM-DD HH:MM:SS[.fraction], for example, 2015-12-20 10:01:00.999999.

For inputting DATETIME values, Navicat provides the DATETIME INPUT control, which combines the DATE and TIME controls:

DATETIME values may be set using a string literal that contains the "T" time portion delineator or by casting to a DATETIME:

Going Forward

Having explored the DATE, TIME, and DATETIME Types, the next installment will cover the remaining two temporal types: TIMESTAMP and YEAR.

Back in May of 2021, we examined a few of SQL Server's Important SQL Server Functions. Now, it's time to turn our attention to MySQL to see what it offers us in terms of math and numeric functions. To see how they work in practice, we'll use them in queries that we'll run in Navicat 16 for MySQL.

Comparing SQL Server and MySQL Functions

In the Important SQL Server Functions article, we reviewed the Abs, Round, Ceiling, and Floor functions. As it turns out, not only does MySQL also implement these functions, but with exactly the same names. Perhaps this is not surprising, as these are fundamental numeric functions across database products and programming languages.

Since the functions are the same in both DBMSes, there's no point in rehashing their use. Instead, we'll forge onwards and explore other useful numeric functions in MySQL.

AVG

You probably already know what the Average, or Arithmetic Mean, is. You may even know that it is calculated by adding all of the value within a data set, and then dividing that result by the number of data points in the set. Hence, if we had five numbers such as 4, 5, 6, 5, 3, we would calculate their average as follows:

(4 + 5 + 6 + 5 + 3) / 5 = 4.6

Simple enough to do with a small sample, but what happens when you have 10,000 rows? The answer, or course, is to use MySQL's built-in AVG function (identically named in SQL Server, by the way). All we need to do is provide it with a numeric expression and it returns its average value. Here's its simple syntax:

AVG(expression)

Most of the time, you'll find yourself passing in a column name whose average you'd like to calculate. For example, here's a query that gives us the average running time of all the films in the Sakila Sample Database:

The GROUP BY breaks up values by the category_id so that averages are based on each Film Type, i.e., "Action", "Drama", etc.

By passing the results of the AVG function to ROUND, we can omit some of the extra decimal places.

A More Complex Example

The interesting thing about numeric functions is that they can be used as part of larger calculations. Case in point, here's a query that shows how many film categories in which the average difference between the film replacement cost and the rental rate larger than 17 dollars:

To calculate the replacement cost, the average rental rate is subtracted from the average replacement cost. No need for temporary variables; just subtract the results of one function from the other:

( AVG( replacement_cost ) - AVG( rental_rate ) ) AS replace_sub_rental

MIN/MAX

Have you noticed that a lot of numeric functions have three letter names? Not sure why that is, but here are two related functions for calculating the minimum and maximum values of a set. Again, the most typical usage is to pass a column name to the function. The following query selects film details for the first and last rentals, according to the rental_date column. As such, it is passed to both the MIN and MAX functions:

Mixing aggregate functions and scalar data can be problematic, so the MIN and MAX rental_dates are fetched within a subquery for comparison to those of each Film table row.

Conclusion

This blog presented a few useful numeric functions in MySQL, including AVG, MIN, and MAX using Navicat 16 for MySQL as our database client. Speaking of which, if you'd like to give Navicat 16 for MySQL a test drive, you can download a 14 day trial here.

One of the key ingredients to writing effective SQL queries is the ability to articulate a wide variety of conditions using SQL syntax. One condition that gives both newbies and experienced database developers pause for thought is the Exclusive OR. Software programmers tend to be more familiar with the syntax for the Exclusive OR condition, probably because most programming languages support the XOR logical operator, whereas many databases do not.

In simple terms, the Exclusive OR condition is similar to the regular OR, except that, in the case of the Exclusive OR, only one of the compared operands may be true, and not both. In this blog, we'll learn how to express an Exclusive OR condition for a variety of databases, whether they support the XOR operator, or not.

Using the XOR Operator

Some popular relational databases, such as MySQL, support the XOR operator, making writing Exclusive OR conditions fairly trivial. To illustrate, let's consider a scenario where we need to find customers that reside within a specific city, or whose account was created after a specific date, but not both. More specifically, say that we want to see customers who reside in Lethbridge, Alberta, OR, if they do not reside in Lethbridge, whose account was created after Jan 1st, 2020.

Here's just such a query, executed against the Sakila Sample Database using Navicat Premium 16:

Looking at the results, we can see that the first customer, whose account was created on 2020-07-07, has a store_id of 2, while the rest of the customers, all have a store_id of 1 (the Lethbridge store).

Meanwhile, if we replace the XOR to a regular OR, we now see customers who shop at store #1 whose accounts were also created after 2020-01-01:

Allowing both operands to evaluate to TRUE is what differentiates OR from XOR.

Writing an Exclusive OR Condition Where XOR Is Not Supported

Luckily, it's not that hard to formulate an Exclusive OR condition without the XOR operator; you just need to think about it a bit more. Mathematically speaking, x XOR y is equal to:

(x AND (NOT y)) OR ((NOT x) AND y)

We can simplify the above formula to the following for the purposes of SQL writing:

(A OR B) AND NOT (A AND B)

We'll try out this formula by rewriting the first query for SQL Server. If we try to execute it against that database, we get the following error, which states that SQL Server did not recognize the XOR operator:

Using the above formula, we can rewrite the XOR condition as:

WHERE   (ci.city = 'Lethbridge' OR c.create_date  > '2020-01-01')AND NOT (ci.city = 'Lethbridge' AND c.create_date > '2020-01-01')

Here are the results in SQL Server (note that the data in both databases is not identical):

Conclusion

In today's blog, we learned how to articulate an Exclusive OR condition in a variety of databases, both using the XOR operator, and without.

If you'd like to give Navicat 16 a test drive, you can download a 14 day trial here.

Recently, we learned how to generate test data using Navicat 16's new Data Generation tool. It can help produce a large volume of complex testing data over multiple related tables, all guided by a multi-step wizard. In today's follow-up, we'll go through the process of creating a MySQL test database - all using Navicat 16.

Duplicating a Production Database

For optimal testing, the best approach is to duplicate the structure of your production databases (DBs), while replacing the "real" data with sanitized test values. For the purposes of this tutorial, we'll use an instance of the MySQL classicmodels Sample Database as the source that is the basis for our test database. Here it is in Navicat Premium 16:

There is usually more than one way to accomplish a task in Navicat. Duplicating a database is no exception. There are several ways to do it; here are a couple:

Create a New Database

Rather than copy the database, we can create a brand new one and then generate the test data for it. To do that:

• In the Navigation pane, right-click your connection and select New Database:

  

• Enter the database properties in the pop-up window:

  

  Hint: if you aren't sure what Character Set and Collation to use, you can open the Edit Database dialog on the source DB to see their values.
• To copy over the table structures without data, simply select all of the tables in the Objects pane and drag them over to the new DB. A popup menu will appear asking you whether to copy over the Structure and Data or the Structure only:

  

  Choose the latter option.

Generate Tables From Model

Most organizations maintain model diagrams of their databases. Navicat's Modeling tool can generate database objects from a model (forward engineering) as well as generate a model from an existing DB (reverse engineering). Let's use it now to generate our test tables.

• Follow the two first steps from the last exercise to create the classicmodels_test database.
• Click the Model button on the main toolbar to see available models:

  

  Hint: if you don't have a model for your database, you can generate one by right-clicking the database in the Navigation Pane and choosing Reverse Schema to Model from the popup menu.
• Open the model in the Modeling tool by selecting the model in the Objects pane and clicking the Design Model button in the Objects pane toolbar:

  

• In the Modeling tool, select File -> Synchronize to Database... from the main menu.
• In the Synchronize to Database dialog, designate classicmodels_test as the target database and click the Compare button:

  

• Navicat will then determine which objects to create, update, or drop to synchronize both databases. In our case, it will generate all of the necessary tables:

  

• On the next screen, we can review the SQL statements that will be executed:

  

  Click the Start button to run the script.
• We'll get a detailed progress report as the script runs:

  

From there, we only need to follow the steps outlined in the Create a Model from a Database in Navicat blog.

Conclusion

Navicat 16 provides numerous options for duplicating an existing database, either on the same server or in a completely different environment.

Speaking of Navicat 16, if you'd like to give it a test drive, you can download a 14 day trial here.

As mentioned in the recent Present Your Data More Effectively with Navicat 16 blog, Navicat 16 added some new charting features, such as support for more data sources and chart types as well as an increased focus on usability and accessibility. These improvements coincide with the release of the new standalone Navicat Charts Creator. This blog will provide a tour of the Navicat Charts Creator and demonstrate how the Navicat Charts Creator can help you to gain deeper insights from your data.

About Workspaces

As the name suggests, a workspace is the place that gathers various resources together to work with as a cohesive unit. In the case of the Navicat Charts Creator, the workspace contains dashboards, charts and data sources. You can create multiple dashboards, charts and data sources within a single workspace. If you are new to the Navicat Charts Creator you can open the sample workspace and explore its contents:

Creating a Workspace

In the main window, click New Workspace. You're now ready to create data sources, charts and dashboards:

When it's time to save your workspace, you have the option of saving it locally or in the Cloud. To save locally, you would choose File -> Save from the main menu, while to save to the Cloud, you would select File -> Save to Cloud. From there, you only need to enter the workspace name and choose the project.

Data Sources

When building a chart, you will need to specify a data source that will supply the chart data via a dataset. The fields in the dataset can be used to construct a chart. Data sources can reference tables in your connections or data in files/ODBC sources, as well as select data from tables on different server types.

Creating a Data Source

To create a data source:

• In the Workspace window, click the New Data Source button, located directly under the main toolbar.
• Enter the name of the data source and select the desired connections, files or existing data sources. On the New Data Source dialog, you can choose from three options: Database, File/ODBC, or Connections in Existing Data Sources:

  

Upon clicking the OK button, a tab will open where you can edit the data source. Moreover, if you want to add more connections, you can click the plus (+) symbol at the top of the Connections pane and repeat the above steps.

To add tables to the Design pane, simply drag and drop them from the Connections pane! Likewise, nodes may also be connected to one another to create joins via drag and drop. Join types may be configured if necessary by clicking on them. To view the table data at any time, click the Preview button.

Charts

A chart is what provides a visual representation of the data in your data source. Mapping to a single data source, a chart can display correlations between several fields in the data. Navicat Charts Creator supports no less than 17 chart types! You can even make the chart interactive by adding a control chart. Here's one that allows the user to select order date months:

Conclusion

I hope that you enjoyed the tour of the Navicat Charts Creator. This blog really only scratched the surface of what it can do, but we'll be looking at the Navicat Charts Creator's capabilities in more detail over the next several weeks. In the meantime, you can download the Standalone Charts Creator and try it for free for 14 days!

There are many times where you'll want to see the relative contribution of a row (or group of rows) to the total row count. In other words, what percentage of the total count a row represents. To illustrate, let's take the following table, shown in Navicat Premium 16:

We can easily find out how many orders were received for each type of fruit by combining the count() function with the Group By clause:

So now, how would we view what percentage each fruit's orders contributed to the total number of orders? In fact, there are three standard ways to calculate row percentages in SQL. They are:

• Using OVER() clause
• Using subquery
• Using a Common Table Expression, or CTE

The rest of this blog will explore each of these in turn.

The OVER() Clause

Used predominantly with Window Functions, the OVER clause is used to determine which rows from the query are applied to the function, what order they are evaluated in by that function, and when the function's calculations should restart.

The OVER clause is the most efficient way to calculate row percentages in SQL, so it should be your first choice if efficiency is a priority for you. Here's the formula to obtain a percentage:

count(*) * 100.0 / sum(count(*)) over()

Adding the above SQL to our original query produces the following results:

Looks good, but some rounding wouldn't hurt. Unfortunately, that's not easily done using the over() clause. Perhaps the next option will be more to your liking.

Using a Subquery

Not all databases support the OVER() clause, so the subquery approach can be a very valuable fallback solution. It's sometimes referred to as the "universal solution" since it works in all databases. Another benefit of this approach is that it is also the easiest to incorporate with functions such as Round(). Here is what we'll need to add to our query:

count(*) * 100.0 / (select count(*) from <YourTable>)

And here is the universal solution in action:

Using a Common Table Expression (CTE)

The With common_table_expression clause specifies a temporary named result set, known as a common table expression (CTE). We can then select from the temporary result set to apply more functions to retrieved fields. IN our case, we can apply the sum() function to the counts to obtain the percentages:

Keep in mind that this approach is the least efficient as the CTE basically runs a second query against the results of the inner (initial) one. That being said, there may be times that you'll need to use a CTE to perform additional processing that you couldn't easily do in one go.

Conclusion

In this blog, we learned three ways to express the relative contribution of a row (or group of rows) to the total row count as a percentage. Each approach has its own strengths and weaknesses, so you'll have to choose one based on your specific requirements.

If you'd like to give Navicat 16 a try, you can download a 14-day fully functional FREE trial of Navicat here.

Throughout the past several weeks, we've been looking at Navicat 16's new features. While exciting to see, one should not discount the many improvements to Navicat that enhance its already great User Interface (UI) and workflow. Hence, today's blog will focus on improvements whose aim is to maximize performance and productivity.

Connection Profile

Keeping with the new work from home paradigm, Navicat 16 lets you configure multiple profiles for out-of-office users who may need to switch to a more secure connection depending on the location of the device they are using. To create a Connection Profile:

• In the connection window, click .

  

• Click + New Connection Profile -> New Profile.
• Enter the name of the profile.
• Enter the connection settings.
• Click OK.

Then, to switch profiles:

• In the main window, right-click a connection and select Switch Connection Profile.
• Select the profile name.

You can also set the default active profile in the connection window.

Value Picker

In the Table Viewer, you can filter records according to numerous criteria using the Filter tool. In Navicat 16, it has been fully rewritten to embed an intuitive panel for selecting values from a list, or inputting possible values to further limit the data in the exact way you want:

It even includes a Search field for looking up specific values:

Field Information

Also in the Table Viewer, the right-hand panel now includes field information that gives you a quick view of column characteristics. It can help you to get column information and compare columns with ease.

Details include:

• Data type
• Not null
• Default value
• Comments

Here's an example using the orders table of the classicmodels sample MySQL database:

Query Summary

Located at the bottom of the SQL Editor, the new query Summary tab shows a detailed summary of each SQL statement. It's an easy-to-read, one page summary of the health and performance of your queries that includes a link to jump over to the potential errors:

You can also view the full query by clicking on the ellipsis [...] button:

The Query Summary tab is especially useful in cases where you have many statements in the editor - especially Data Manipulation Language (DML) statements such as UPDATE, INSERT INTO, DELETE, and CREATE:

Conclusion

In this blog we explored four improvements in Navicat 16 whose aim is to maximize performance and productivity. These updates are all part of Navicat's commitment to maintaining your databases and ensuring that Navicat products perform at the highest possible level, so that your database administration and development activities are both efficient and fulfilling.

If you'd like to give Navicat 16 a try, you can download a 14-day fully functional FREE trial of Navicat here.

With all of the excitement surrounding the release of Navicat 16, other noteworthy developments have been overshadowed somewhat. Perhaps none more so that the new standalone Navicat Charts Creator. Charting has been a part of Navicat products for some time now. Navicat 15 went even further to include data visualization in order to help identify trends, patterns and outliers. Navicat 16 adds even more features, by supporting more data sources and chart types as well as a focus on usability and accessibility. Suffice to say, Navicat can deliver information and present your findings in dashboard for sharing to a wider audience than ever before. In today's blog, we'll take a quick tour of Navicat 16's new charting tools.

Data Connectors

In order to create visualizations of your data, you first need to connect to a data source. Navicat Charts Creator allows you to quickly and securely connect to any data source of your choice via four built-in data connectors. These include:

• Relational DB:For working with relational databases such as MySQL, MariaDB, PostgreSQL, Oracle, SQLite and SQL Server.

• File Types: Allows you to import data from external files such as Excel, Access, or CSV, from data stored on your computer, network, or accessible via an URL.

• ODBC:For importing data from any ODBC data source including Sybase, Snowflake and DB2.

• Linked File:Lets you link your chart to data in data sources to update the chart according to changes in the underlying data.

Chart Types

It's vitally important to choose the right type of chart so that your presentations convey the message you want to communicate. To that end, Navicat 16 includes a wide array of chart types, ranging from standard to exotic:

• Bar Chart
• Line / Area Chart
• Bar / Line Chart
• Pie Chart
• Heatmap / Treemap
• Pivot Table
• Scatter Chart
• Value
• Control
• KPI

Navicat 16 also introduces the Waterfall, Tornado, and Gauge chart types.

Dashboards

Dashboards combine a collection of widgets to give you an overview of the reports and metrics you care about most. As such, they provide a way to monitor many metrics at once, so you can quickly see correlations between different reports. In Navicat 16, a dashboard shows various topics that you would like to track in one place by displaying a collection of charts in an interactive way. These may be synchronized in order to better demonstrate how the charts are related. For example, hovering over one chart shows the effect in the other charts:

Each dashboard receives a thumbnail that gives you a visual hint of the types of charts you have, so you can more easily navigate between dashboards:

Conclusion

This blog provided a quick tour of Navicat 16's new charting tools. These are available in all Navicat database development tools as well as in the brand new standalone Charts Creator.

If you'd like to give Navicat 16 a try, you can download a 14-day fully functional FREE trial of Navicat here.

When the Navicat team added the Navicat Cloud collaboration tool a few years ago, little did anyone know that a global pandemic would make collaboration a vital part of most organizations - especially those who provide any kind of Information technology (IT) related services. Being where we are in the last days of 2021, it should come as no surprise that Navicat has expanded its cloud solutions for Navicat 16. Now, Navicat Cloud supports more objects, and Navicat has just introduced an On-Prem Server for businesses working with sensitive data. Today's blog will provide an overview of Navicat 16's improved collaboration features.

Navicat Cloud: More Objects Added

In case you weren't already familiar with Navicat Cloud, it provides a cloud service for synchronizing Navicat connections, queries, models and virtual groups from different machines and platforms. Whenever you add a connection to Navicat Cloud, it stores connection settings and queries. You can synchronize model files to Navicat Cloud and create virtual groups as well. All the Navicat Cloud objects are located under different projects for easy access. Besides helping you keep your objects synchronized across devices, Navicat Cloud facilitates project sharing with other Navicat Cloud accounts for collaboration.

In addition to connections, queries, models and virtual groups, Navicat 16 adds Code Snippets and Charts workspaces to Navicat Cloud and the new On-Prem Server. Now you'll be able to save your Code Snippet and Charts workspace files to the cloud and share them across your Navicat products and with team members.

Introducing the Navicat On-Prem Server

Navicat On-Prem Server is an on-premise solution that provides you with the option to host a cloud environment for storing Navicat objects internally at your location. In the Navicat On-Prem environment, you can enjoy complete control over your system and maintain 100% privacy. It is intended for organizations who wish or need to maintain a level of control that the cloud often cannot provide.

In terms of functionality, Navicat On-Prem Server behaves very much like its Navicat Cloud counterpart. Hence, it can synchronize your connection settings, queries, models, snippets, chart workspaces and virtual group information across all your Windows, macOS and/or Linux devices. Files stored in Navicat On-Prem Server will automatically show up on Navicat Family and Navicat On-Prem Server Portal so that you can get real-time access from anywhere, anytime.

Navicat On-Prem Server is installed locally on your own servers and behind your firewall. Everything is run on the server with no third-party access. That way, you retain full control of security and data ownership within your environments. Moreover, all changes, configurations and upgrades may be performed at your sole discretion.

Conclusion

Today's blog provided an overview of Navicat 16's improved collaboration features.

Navicat Cloud and Navicat On-Prem Server are available on all Navicat products and all platforms including Windows, macOS and Linux.

The recent Navicat 16 listed some of its most note-worthy features and improvements, including:

• Data Generation
• Charts
• On-Prem Server
• Collaboration
• UI/UX Improvements

As promised, we'll be exploring these in much more detail throughout the coming weeks. In today's blog, we'll start with the entirely new Data Generation tool. We'll familiarize ourselves with it by going through the process of creating testing data for multiple related tables in Navicat Premium 16 for Windows.

Setting Up the Test Database

The database that we'll be working with is the chinook sample database for SQLite. You can download it using the following link:

Download SQLite sample database

Chinook represents a fictional digital media store, and hence includes tables for artists, albums, media tracks, invoices, and customers. Here they are in Navicat Premium 16:

Launching the Wizard

The Data Generation tool is located under the Tools item in the main toolbar:

The ellipsis at the end of the label (...) tells us that the command will open a dialog or wizard. In this case, the latter is true.

Selecting a Database

The first wizard screen lets us set the database for which to generate the test data. The wizard is smart enough to know that, since we already have an active database connection open, we probably want to generate data for it:

At any stage, you can Save or Load a profile so that you don't have to start over when working with the same database(s). There is also an Options button that opens a dialog where you can configure a few general preferences:

Tables Population and Ordering

The next screen is where we set which tables and fields to generate data for. (It goes without saying that you'll want to select empty tables that are based on the real tables that you're testing.) By default, Navicat generates 1000 rows of data, but we can change that value via the Number of Rows to Generate text field:

Navicat will automatically determine which order to follow when generating data, but we can change it on the Table Generation Order dialog:

Data Previews

The next screen will show us a preview of what the generated data will look like for each table that we selected back on the second screen. This will give us the opportunity to manually change values or Regenerate all data for a table:

Once we're satisfied with the data we can generate it by clicking the Start button.

Progress Report

Navicat provides a complete report of its progress. We can see here that a UNIQUE constraint failed on the artists.ArtistId field. That happened because that table already contained data!

Using the Back button, we can return to a previous screen to fix reported errors and try again. (This time I selected the test tables)

Conclusion

In this blog we familiarized ourselves with Navicat 16's new Data Generation tool by going through the process of creating testing data for the Chinook Sample Database for SQLite.

Interested in trying Navicat 16 for yourself? You can download a 14 day free trial here.

In software development, there is a Boolean data type for working with binary states. Hence, it only has two possible states: true and false. However, there exists a third state that must often be accounted for, and that is one for "none of the above" or simply "other". In relational databases, NULL might seem to be a good candidate for this state, but is not, due to its historical context. Recall from previous blogs that NULL has a very specific meaning in Structured Query Language (SQL) to indicate that a data value does not exist in the database. The NULL value was actually introduced by none other than the creator of the relational database model himself, E. F. Codd. In SQL, NULL has come to indicate "missing and/or inapplicable information". Seen in this light, NULL can hardly represent a "none of the above" or "other" condition. So then, what is the best way to represent ternary - or three-state - data in relational databases? We will answer that question here today for MySQL and PostgreSQL. Next week we'll cover SQL Server and Oracle.

Introducing Enumerated Types

Enumerated Types - also known as Enums - are data types that contain a static, ordered set of values. Enums are ideal for storing things such as the days of the week, user preferences, and any other collection of related data that seldom change. Having enjoyed support in a number of programming languages for decades, some of the biggest relational database players, including MySQL and PostgreSQL, have also introduced the Enum type. Unfortunately, there are a few holdouts, including SQL Server and Oracle, which we'll talk about next week.

Creating and Using Enums in MySQL

To get an idea how one would use Enums, let's start with the number one relational database in the world. Yes, I speak of MySQL. As you can see in the following CREATE TABLE statement, designating a column as an Enum type is quite trivial:

CREATE TABLE shirts (  name VARCHAR(40),  size ENUM('x-small', 'small', 'medium', 'large', 'x-large'));

From there, you can refer to an Enum using one of its string values:

INSERT INTO shirts (name, size) VALUES ('dress shirt','large'),        ('t-shirt','medium'),       ('polo shirt','small');  SELECT name, size FROM shirts WHERE size = 'medium';UPDATE shirts SET size = 'small' WHERE size = 'large';

With regards to the tri-state issue, we can implement one as follows:

CREATE TABLE employee (  name VARCHAR(50),  security_clearance ENUM('enhanced', 'secret', 'none'));

Now, trying to insert an invalid value into an Enum column will result in an error and will fail:

Creating and Using Enums in PostgreSQL

In PostgreSQL, Enum types are created using the CREATE TYPE command:

CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');

Once created, the Enum type can be used in table much like any other type:

CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');CREATE TABLE person (    name text,    current_mood mood);INSERT INTO person VALUES ('Moe', 'happy');SELECT * FROM person WHERE current_mood = 'happy'; name | current_mood ------+-------------- Moe  | happy(1 row)

Conclusion

In today's blog, we saw how to represent tri-state data, and other discreet values, in MySQL and PostgreSQL, using Enumerated Types. But what about other database types? Do they not support tri-state data? They do, but using different data types. We'll explore these next week.

How many times have you found a query to be sufficiently performant when testing against sanitized data, only to see it stall once in production? It happens all the time, due to differences between the environments such as workload and volume of data. This may lead you to try out your query in production. After all, the fastest way to tune a query for production is on the production server, is it not? While correct, there are many dangers awaiting those foolish enough to tempt fate with such a cavalier disregard for safeguards and protocols. In this blog, we'll explore some of the risks associated with testing queries in production.

Some Risks to Consider

Those who do would be wise to remember that the point of a "test" is that you are testing something that runs the risk of "Bad Things" happening. You might not be able to think of any risks, but that's because you don't - and can't - know what those are...until you run your tests. Some of the Bad Things that can happen include:

Logical Data Corruption

In the case of INSERT and UPDATE statements, these are by their very nature likely to create records with spurious or malformed data, or that doesn't implement referential integrity correctly. Moreover, the bad data can break logical assumptions that previously tested and well-behaved applications hold about the data. Should an application encounter a faulty record, it may result in anything from "wrong answers" to crashing your entire site until the corruption is identified and manually fixed.

Performance Degradation

A common scenario is that your test app is doing a table scan that you didn't identify in your dev instance because it only contains about 10,000 records, compared to the 100 million in production. Once your application commences a table scan on one or more core tables, your prod database can become deadlocked until you kill the queries one-by-one. Worse still, once your app has used up all the available database connections, you won't even be able to open a command shell on the database to kill the queries.

Locking Issues

If you were to forget to COMMIT a transaction, you could wind up locking half the database because you have uncommitted multi-statement transactions sitting in your database connection pool. As a result, the customer operations could time out and randomly deadlock.

End Users See Wrong Data

Bad data is a problem at many levels. The badness can range from selling inventory that you don't have to having user accounts without passwords that open you to hacks. A related concern is forgetting to delete test users. Since you're probably the only one who knows about them, they're likely to stick around until a hacker manages to discover your "test123" user with the "password123" password.

Unknown and Undefined Risks

The are countless other scenarios of disaster and destruction that are limited only by your imagination or those of higher powers who enjoy watching us suffer and struggle...

Conclusion

Never think that because you "understand" your code that bad things can't happen. Down that path lies madness. You may get away with your shenanigans for a while, maybe even a good while. Nonetheless, the day will come when things go badly, and when they do, they will go very badly.

SQL Server provides a number of data types that support all types of data that you may want to store. As you may have guessed, data type is an attribute that specifies the type of data that a column can store. It can be an integer, character string, monetary, date and time, and so on. One data type that causes some confusion among database designers and developers are those for storing character strings. A character string is a series of characters manipulated as a group. In the context of relational databases, character string data types are those which allow you to store either fixed-length (char) or variable-length data (varchar). Moreover, SQL Server splits its string types into two broad categories: Unicode and non-Unicode. These equate to nchar, nvarchar, and ntext for Unicode types and char, varchar/varchar (max) and text for non-Unicode. In today's blog, we'll compare the two categories to decide when to use one over the other.

Tracing the Roots of Unicode and Non-Unicode Data Types

Nchar is short for "NATIONAL CHARACTER", nvarchar stands for "NATIONAL CHARACTER VARYING", and ntext is the ISO synonym for "NATIONAL TEXT". Originally intended for pre-Unicode multibyte encodings like JIS encoding for Asian characters. The idea was that VARCHAR would continue to be utilized for ASCII, with NVARCHAR being employed for non-ASCII characters.

This use-case was designed when the Internet was still in its infancy and before the Unicode project had taken off. In those days, Asian languages in particular, employed their own specific - and mutually incompatible - encodings, with GB for mainland Chinese, JIS/SJIS for Japanese, BIG5 in Hong Kong and Taiwan, CNS in Taiwan, etc. However, all of that changed with the emergence of the Unicode project encodings, as database vendors realized that it was easier to just allow VARCHAR itself to support multibyte character encodings, and use Character Sets and Collations to deal with specific encodings. For instance, you can use UTF-8 to encode any character you need in any language your applications need to support. Thus, the need for a whole group of character data types that were specific to "NATIONAL CHARACTER" soon faded away.

Today, in many modern DB engines, "NVARCHAR" and "NATIONAL CHARACTER VARYING" are really just aliases for VARCHAR, with the actual implementation being virtually (if not exactly) identical. Having said that, SQL Server does treat the two differently. As stated in the docs:

The key difference between varchar and nvarchar is the way they are stored, varchar is stored as regular 8-bit data(1 byte per character) and nvarchar stores data at 2 bytes per character. Due to this reason, nvarchar can hold up to 4000 characters and it takes double the space as SQL varchar.

Which to Use?

As specified above, the biggest concern when deciding between types is the amount of storage used. For example, nvarchar uses 2 bytes per character, whereas varchar uses 1. Thus, nvarchar(4000) uses the same amount of storage space as varchar(8000). Hence, if you have requirements to store UNICODE or multilingual data, nvarchar is the best choice. Varchar stores ASCII data and should be your data type of choice for normal use. Another consideration is that joining a VARCHAR column to a NVARCHAR (and vice-versa) in queries can lead to a considerable performance hit.

Conclusion

In today's blog, we compared SQL Server's Unicode and Non-Unicode String Data Types to decide when to use one over the other.

Have you ever seen a WHERE 1=1 condition in a SELECT query. I have, within many different queries and across many SQL engines. The condition obviously means WHERE TRUE, so it's just returning the same query result as it would without the WHERE clause. Also, since the query optimizer would almost certainly remove it, there's no impact on query execution time. So, what is the purpose of the WHERE 1=1? That is the question that we're going to answer here today!

Does WHERE 1=1 Improve Query Execution?

As stated in the introduction, we would expect the query optimizer to remove the hard-coded WHERE 1=1 clause, so we should not see a reduced query execution time. To confirm this assumption, let's run a SELECT query in Navicat both with and without the WHERE 1=1 clause.

First, here's a query against the Sakila Sample Database that fetches customers who rented movies from the Lethbridge store:

The execution time of 0.004 seconds (highlighted with a red outline) can bee seen at the bottom of the Messages tab.

Now, let's run the same query, except with the addition of the WHERE 1=1 clause:

Again, the execution time was 0.004 seconds. Although a query's run time can fluctuate slightly, depending on many factors, it is safe to say that the WHERE 1=1 clause had no effect.

So, why use it then? Simply put, it's...

A Matter of Convenience

The truth of the matter is that the WHERE 1=1 clause is merely a convention adopted by some developers to make working with their SQL statements a little easier, both in static and dynamic form.

In Static SQL

When adding in conditions to a query that already has WHERE 1=1, all conditions thereafter will contain AND, so it's easier when commenting out conditions on experimental queries.

This is similar to another technique where you'd have commas before column names rather than after. Again, it's easier for commenting:

In Dynamic SQL

It's also a common practice when building an SQL query programmatically. It's easier to start with 'WHERE 1=1 ' and then append other criteria such as ' and customer.id=:custId', depending on whether or not a customer ID is provided. This allows the developer to append the next criterion in the query starting with 'and ...'. Here's a hypothetical example:

stmt  = "SELECT * "stmt += "FROM TABLE "stmt += "WHERE 1=1 "if user chooses option a then stmt += "and A is not null "if user chooses option b then stmt += "and B is not null "if user chooses option b then stmt += "and C is not null "if user chooses option b then stmt += "and D is not null "

Conclusion

In this blog, we learned the answer to the age-old question of "what is the purpose of the WHERE 1=1?" It's not an advanced optimization technique, but a style convention espoused by some developers.

SQLite and MySQL are equally popular open source Relational Database Management Systems (RDBMS). Both are fast, cross-platform, robust, and feature-rich. Yet, beyond these similarities, the two databases are dissimilar in several important respects. Since you are probably more familiar with MySQL, this tutorial will list SQLite's most important features, as well as dissimilitudes to MySQL, all with the goal of steering you towards the product that will best suit your needs.

Storage and Portability

SQLite was designed and built with storage and portability in mind. This is apparent when viewing it's main design features:

• Built with the C language.
• Implements an embedded, server-less, zero-configuration, transactional SQL database engine.
• Does not have a separate server process (unlike most other SQL databases).
• SQLite reads and writes directly to ordinary disk files.
• All tables, indices, triggers, and views, are contained within a single disk file.
• The database file format is cross-platform and may be copied between 32-bit and 64-bit systems.

The SQLite library is about 250 KB in size, while the MySQL server is about 600 MB. Moreover, no configurations are required, and the process can be done using minimal support.

Before copying or exporting a MySQL database, you need to condense it into a single file. For larger databases, this can be a time-consuming process.

Security and Ease of Setup

As alluded to in the previous section, SQLite requires little to no configuration, making it extremely easy to set up. On the other hand, MySQL requires significantly more configuration as compared to SQLite. As the same time, MySQL also has more setup guides available to help with this.

SQLite does not have an inbuilt authentication mechanism. Hence, the database files can be accessed by anyone. Meanwhile, MySQL comes with many inbuilt security features. This includes authentication with a username, password, and connection over SSH.

Multiple Access and Scalability

SQLite does not include user management functionality and so, it is not suitable for multiple user access. MySQL has a fine-grained user management system which can handle multiple users and grant various levels of access.

In terms of scalability, SQLite is well suited to for smaller databases. As the database grows the memory requirement also increases, SQLite's performance will degrade. Adding to this issue is that performance optimization is more difficult to achieve when using SQLite. On the other hand, MySQL is easily scalable and can handle very large databases, including tables with billions of rows!

Database Administration

The are a number of free and commercial grade graphical database administration tools for both SQLite and MySQL. For SQLite, there's SQLite Administrator. It helps you to create, design and administrate SQLite database files. The SQL code editor helps you to quickly write SQL queries and includes features such as code completion and highlighting.

MySQL's free graphical administration tool is MySQL Workbench. It's a unified visual tool for database architects, developers, and DBAs. MySQL Workbench provides data modeling, SQL development, and comprehensive administration tools for server configuration, user administration, backup, and more. MySQL Workbench is available on Windows, Linux and Mac OS X.

For more professional applications, there's Navicat for SQLite, Navicat for MySQL, or Navicat Premium. Navicat's powerful and comprehensive GUI provides a complete set of functions for database management and development. Helping you optimize your workflow and productivity, you can quickly and securely create, organize, access, and share information.

Conclusion

SQLite is an effective solution for developing small standalone apps and for smaller projects which do not require much scalability. Meanwhile, MySQL is the superior option when you require access for multiple users with strong security and authentication, as well as for larger datasets.

Back in March of 2020, the The NULL Value and its Purpose in Relational Database Systems article presented the NULL value and its special meaning in relational databases. That article also described how to allow NULLs in your database tables and how to reference them in queries. In today's blog, we'll learn how to combine NULLs with the SQL Count() function to achieve a variety of objectives.

Counting Null and Non-null Values

The Count() function comes in two flavors: COUNT(*) returns all rows in the table, whereas COUNT(Expression) ignores Null expressions. Hence, if you provide a column name that allows NULL values, then Count() will return all rows that have a non-null value. These two separate uses of Count() provide an important clue as to how we can obtain a count of NULL values for a specific column. And that is by subtracting the non-NULL fields from the Total fields, like so:

SELECT COUNT(*) - COUNT(<Column Name>)

Now that we know how to count null, non-null, and all rows in a table, let's see an example. We'll run this query against the customers table of the MySQL classicmodels Sample Database. Here is that table in Navicat Premium's Table Designer:

The addressline2 field contains additional address details that are not part of the street name and number. Hence, it's not required for all addresses, as we can see in this sample of table data:

This query uses the Count() function in three ways to show all table rows, the number of populated addressLine2 rows and Nulls:

SELECT COUNT(*) AS All_Rows,       COUNT(addressLine2) AS addressLine2_Count,        COUNT(*) - COUNT(addressLine2) AS Null_addressLine2_RowsFROM customers;

Here is the above SELECT statement in Navicat Premium's Query Designer, along with the results:

As expected, the addressLine2_Count and Null_addressLine2_Rows results add up to the All_Rows count.

Using NULL in Content Analytics

That fact that the COUNT(Expression) version of the Count() function ignores Null expressions can be extremely helpful in compiling statistics about table data, especially when combined with other functions such as the SQL IF() function, which is basically the SQL equivalent of the Ternary Operator:

IF(predicate, true-value, false-value)

If the predicate is true, IF evaluates to the true-value, or 1 in the query below. If the predicate is false, it evaluates to the false-value, or NULL, as seen in the statement below. The COUNTs then tabulate each row where the IFs evaluate to 1, i.e., where the predicate is true:

SELECT count(IF(country = 'Australia', 1, NULL)) as Australia_Count,        count(IF(country = 'Germany', 1, NULL)) as Germany_Count,        count(IF(country = 'Canada' OR country = 'USA', 1, NULL)) as North_America_Count,        count(IF(country like 'F%', 1, NULL)) as F_Countries_Count,        count(IF(creditLimit between 20000 and 1000000, 1, NULL)) as CreditLimit_Range_Count,        count(*) as Total_CountFROM customersWHERE dob >= '1960-01-01';

Here is the query and results in Navicat:

Conclusion

In today's blog, we learned how to combine NULLs with the SQL Count() function to achieve a variety of objectives. More than a way to count NULL and non-NULL values, when combined with other SQL function such as IF() and SUM(), these can be utilized to compile all sorts of statistics on your data!

CROSS APPLY and OUTER APPLY Examples

Last blog introduced the APPLY operator and covered how it differs from regular JOINs. In today's follow-up, we'll compare the performance of APPLY to that of an INNER JOIN as well as learn how to use APPLY with table valued functions.

APPLY and INNER JOIN Comparison

Recall that, at the end of Part 1, we ran a query made up of two parts: the first query selected data from the Department table and used a CROSS APPLY to evaluate the Employee table for each record of the Department table; the second query joined the Department table with the Employee table to produce the same results:

In Navicat, we can click on the EXPLAIN button to obtain valuable information about the database execution plan. Here's what it reveals about the above queries:

Although the execution plans for both queries are similar and carry an equal cost, their execution plans do differ somewhat from each other:

• The APPLY query uses a Compute Scalar. It's an operator that is used to calculate a new value from the existing row value by performing a scalar computation operation that results a computed value. These Scalar computations includes conversion or concatenation of the scalar value. Note that the Compute Scalar operator is not an expensive operator, and adds very little cost to the overall weight of our query, causing a minimal overhead.
• The JOIN query contains an additional Clustered index scan. This occurs when SQL server reads through for the Row(s) from top to bottom in the clustered index, such as when searching for data in non key column. This is a slightly more costly operation than Compute Scalar.

Using the APPLY Operator To Join Table Valued Functions and Tables

A table-valued function is a user-defined function that returns data of a table type. The return type of a table-valued function is a table, therefore, you can use the table-valued function just like you would use a table. Joining table valued functions with other tables is what the APPLY operator was designed for.

Let's create a table-valued function that accepts a DepartmentID as its parameter and returns all the employees who belong to that department. In Navicat, we can create a function by clicking the big function button on the main toolbar and then clicking on New Function on the Function toolbar:

Here is the GetAllEmployeesForDepartment function, after clicking the Save button:

Watch what happens when we join our new function to each department using both CROSS OUTER and OUTER APPLY:

In each case, the query passes the DepartmentID for each row from the outer table expression and evaluates the function for each row, similar to a correlated subquery. Whereas the CROSS APPLY returned only correlated data, the OUTER APPLY returned non-correlated data as well, which resulted in NULLs for the missing columns.

We could not replace the CROSS/OUTER APPLY in the above queries with an INNER JOIN/LEFT OUTER JOIN. Doing so would produce the error "The multi-part identifier "D.DepartmentID" could not be bound.". This is because the execution context of the outer (JOINed) query differs from that of the function (or a derived table). Thus, you cannot bind a value/variable from the outer query to the function as a parameter. Hence the APPLY operator is required for such queries.

Conclusion

That concludes our look at the CROSS APPLY and OUTER APPLY statements. So, in summary, while the APPLY operator is required when you have to use a table-valued function in the query, it may be utilized with inline SELECT statements as well.

Navicat 15 was released with much fanfare back in November of 2019. It came packed with many new features and improvements, most notably in data transfers, the SQL Builder, and modeling. It also added Data Visualization, Dark Mode and native Linux support. Almost two years later to the day, its time to announce the upcoming release of Navicat 16! It's currently downloadable in Beta mode, with the official release to be announced shortly. While we're waiting for that, this blog will outline some of the most note-worthy features and improvements.

Data Generation

Most organizations won't permit the copying of production data into test environments, and rightly so! Navicat 16's Data Generation tool assists you in creating a large volume of testing data. It can create complex data over multiple related tables. The entire process is guided by a multi-step wizard. On the table selection screen, you can choose the exact order in which to populate the tables, so that no Foreign Key constraints are violated:

Navicat then shows a detailed preview of the data that will be generated. There, you can choose to regenerate the data for each table, and even edit it in place manually.

Charts

Although charts are not new to Navicat 16, this latest version supports more data sources and chart types than any previous version. With a focus on usability and accessibility, Navicat can deliver information and present your findings in dashboard for sharing to a wider audience than ever before. Moreover, the process of creating a chart and/or dashboard has been streamlined into clear steps:

• Create Data Source
• Design Chart
• Present Dashboard

All of your work may be associated with a workspace to keep all of your data visualizations and presentations well organized.

On-Prem Server

The On-Prem Server is a brand new Navicat product. It's an add-on cloud service which works closely with Navicat 16 that gives you with the option to host a cloud environment for storing Navicat objects internally at your location. Using the On-Prem Server, you can sync and share your connection strings, queries and models from multiple devices as well as share them with your team members from anywhere, anytime.

Collaboration

Those of you who are familiar with Navicat 15 are probably well aware of Navicat Cloud. Now, in version 16, the popular cloud service adds Charts and Code Snippets. Navicat Cloud helps keep your team be productive as well as to collaborate more effectively and efficiently.

Navicat Cloud Portal provides a comprehensive tool set to manage your files and projects. It simplifies user management activities and allows you to monitor your cloud services through one interface, improving operational efficiency while reducing management costs.

UI/UX Improvements

Not only has the UX been completely updated, but, many existing features, such as Connection Profile, Query Summary, and Value Picker have been updated to increase the overall efficiency of your database development.

Conclusion

Today's blog provided an overview of Navicat 16's many new features and improvements. In the coming weeks, we'll be exploring these in much more detail.

If you'd like to give Navicat 16 Beta a try, you can download it here.

Part 1: APPLY vs JOIN

As you are probably aware, JOIN operations in SQL Server are used to join two or more tables. However, in SQL Server, JOIN operations cannot be used to join a table with the output of a table valued function. In case you have not heard of table valued functions, these are functions that return data in the form of tables. In order to allow the joining of two table expressions SQL Server 2005 introduced the APPLY operator. In this blog, we'll learn how the APPLY operator differs from regular JOINs.

About CROSS APPLY and OUTER APPLY

SQL Server APPLY operator comes in two variations: CROSS APPLY and OUTER APPLY:

• The CROSS APPLY operator returns only those rows from the left table expression (in its final output) if it matches with the right table expression.
  Thus, the CROSS APPLY is similar to an INNER JOIN, or, more precisely, like a CROSS JOIN with a correlated sub-query with an implicit join condition of 1=1.
• The OUTER APPLY operator returns all the rows from the left table expression regardless of its match with the right table expression. For those rows for which there are no corresponding matches in the right table expression, it returns NULL values in columns of the right table expression.
  Hence, the OUTER APPLY is equivalent to a LEFT OUTER JOIN.

Although the same query can be written using a normal JOIN, the need of APPLY arises when you have a table-valued expression on the right side and you want this table-valued expression to be evaluated for each row from the left table expression. Moreover, there are cases where the use of the APPLY operator can boost query performance.

Let's explore the APPLY operator further with some examples.

The Sample Data

We'll execute our queries against two new tables that we'll create in Navicat for SQL Server. Here is the design for the Department table:

Here is the design for the Employee table:

Execute the following SQL in the Navicat Query Editor to populate the tables:

INSERT [Department] ([DepartmentID], [Name])  VALUES (1, N'Engineering') INSERT [Department] ([DepartmentID], [Name])  VALUES (2, N'Administration') INSERT [Department] ([DepartmentID], [Name])  VALUES (3, N'Sales') INSERT [Department] ([DepartmentID], [Name])  VALUES (4, N'Marketing') INSERT [Department] ([DepartmentID], [Name])  VALUES (5, N'Finance') GO  INSERT [Employee] ([EmployeeID], [FirstName], [LastName], [DepartmentID]) VALUES (1, N'Orlando', N'Gee', 1 ) INSERT [Employee] ([EmployeeID], [FirstName], [LastName], [DepartmentID]) VALUES (2, N'Keith', N'Harris', 2 ) INSERT [Employee] ([EmployeeID], [FirstName], [LastName], [DepartmentID]) VALUES (3, N'Donna', N'Carreras', 3 ) INSERT [Employee] ([EmployeeID], [FirstName], [LastName], [DepartmentID]) VALUES (4, N'Janet', N'Gates', 3 ) 

CROSS APPLY vs INNER JOIN

Here is a query that is made up of two parts: the first query selects data from the Department table and uses a CROSS APPLY to evaluate the Employee table for each record of the Department table; the second query simply joins the Department table with the Employee table to produce the same results:

Coming up in Part 2

Having introduced the APPLY operator in this blog, Part 2 will outline the differences between using APPLY and JOIN and provide additional uses for APPLY.

Recently, the subject of database indexes has come up a couple of times, specifically, in the The Downside of Database Indexing and The Impact of Database Indexes On Write Operations articles. Both pieces alluded to the fact that relational databases support a number of index types. Today's blog will provide an overview of the most common ones.

The Role of Database Indexes

In RDBMS (Relational Database Management Systems), indexes are a special object that allow the user to quickly retrieve records from the database. Typically, an index is implemented as a lookup table that has only two columns: the first column contains a copy of the primary or candidate key of a table; the second column contains a set of pointers for holding the address of the disk block where that specific key value is stored.

Two Types of Indexing Methods

Index types may be classified based on their indexing attributes. These fall into the two main categories of Primary andSecondary Indexing.

A Primary Index is an ordered file whose records are of fixed length with two fields. The first field of the index replicates the primary key of the data file in an ordered manner, and the second field contains a pointer that points to the data-block where a record containing the key is available.

Secondary indexes are indexes that store the primary key value rather than store a pointer to the data. The advantage is that, by accessing data through a primary key, there's no need for any additional data lookup, as all of the data you need can be found in the primary key's leaf pages.

The secondary Index in DBMS can be generated by a field which has a unique value for each record, and it should be a candidate key. It is also known as a non-clustering index. This two-level database indexing technique is used to reduce the mapping size of the first level.

Dense vs. sparse Indexes

In a dense index, a record is created for every search key valued in the database. This helps you to search faster but needs more space to store index records. In this Indexing, method records contain search key value and points to the real record on the disk.

A Sparse Index is an index record that appears for only some of the values in the file. Sparse Indexes helps you to resolve the issues of dense Indexing in DBMS. In this method of indexing, a range of index columns stores the same data block address, and when data needs to be retrieved, the block address will be fetched. Since sparse indexes only store index records for some search-key values, it needs less space, less maintenance overhead for insertion, and deletions. The drawback is that they are slower compared to dense indexes for locating records.

An Example of Primary and Secondary Indexing

In Navicat, fields that are part of the Primary Key are identified on the Fields tab of the Table Designer:

Secondary indexes are often required on tables so that users can search on fields that are not part of the Primary Key. In Navicat, all secondary indexes are listed on the Indexes tab:

By clicking the EXPLAIN button, we can see what indexes the database is using to fetch records for a given query:

Conclusion

This blog provided an overview of the most common RDBMS index types and provided an example using Navicat Premium. If you're interested in learning more about Navicat Premium, you can try it for free for 14 days!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Over time, system requirements change. These may necessitate the creation of new databases, tables, and columns as well as the altering of existing table structures. Changing a column's data type may be a trivial operation or a difficult one, depending on the source and target data types, as well as the data contained within the column. This blog will address some of the common challenges in changing a column's data type, along with strategies which you can employ to facilitate the process.

Alter Table Statement

The structure (schema) of existing tables can be altered using the ALTER TABLE statement. It's a Data Definition Language (DDL) statement, just like CREATE TABLE, DROP FUNCTION, and GRANT. It's basic syntax is:

ALTER TABLE table_to_change    what_to_change    (additional_arguments)

The ALTER TABLE statement may be utilized to change all sorts of table properties, from changing the table name to adding, dropping, and modifying columns.

One Statement, Varying Syntax

You may have noticed that, after the first line, the ALTER TABLE statement's syntax becomes quite vague. That's because it varies from vendor to vendor. For example:

In SQL Server

ALTER TABLE table_nameALTER COLUMN column_name column_type;

In PostgreSQL

ALTER TABLE table_nameALTER COLUMN column_name TYPE column_definition;

In Oracle, MySQL, and MariaDB

ALTER TABLE table_nameMODIFY column_name column_type;

A Simple Example

Some databases, such as Oracle, don't allow you to run an ALTER query on tables that contain data. If you do, you'll get an error such as this:

Error report:SQL Error: ORA-01439: column to be modified must be empty to change datatype01439. 00000   column to be modified must be empty to change datatype

However, most database types do allow you to make changes to populated tables.

Here's a MySQL table in Navicat Premium's Table Designer that shows the column definitions:

We can execute an ALTER TABLE statement to increases the name (VARCHAR) column's capacity to 255 characters:

Converting a Column from VARCHAR to INT

It's not uncommon to see VARCHAR columns that contain numeric data. In some cases, it may be advantageous to change its type to a numeric type. In Navicat, we can set a column's type by choosing it from a drop-down:

Changes are made once the Save button is clicked. If you forget, Navicat will prompt you to save your changes when you close the Table Designer.

Data Truncated Error

You should avoid diminishing the size of a column's data type whenever possible; otherwise, you'll get a Data Truncated error, such as:

#1265 - Data truncated for column 'name' at row 2

There are no hard and fast rules for dealing with this error, but generally, you can update the value(s) in question yourself and then re-run the ALTER TABLE statement. For instance, here's a statement that truncates all name values to ten characters:

Conclusion

This blog outlined some of the common challenges in changing a column's data type, along with strategies which you can employ to facilitate the process.

Interested in Navicat Premium? You can try it for free for 14 days!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Although MySQL DECIMAL and NUMERIC data types are both fixed-point values, they are still susceptible to rounding errors. The reason is that, no matter how many digits a type can accommodate (the maximum number of digits for DECIMAL is 65!) that number is still fixed. Moreover, DECIMAL columns can be assigned a precision or scale that could have the potential affect of truncation to the allowed number of digits.

I became aware of potential rounding errors in MySQL when a reader asked me why a couple of similar queries were returning slightly different DECIMAL values in calculations. This prompted me to go on a journey of discovery. In today's blog, I would like to share some of what I learned about floating point rounding in MySQL.

A Tale of Two Queries

Here are the queries I used to show the discrepancy:

Note: some rows were removed from the Group By With Rollup query to reduce the height of the image.

The reader was calculating employee salaries, but I did not have an identical table to query, so I used the most similar table that I could find, and that was the payments table of the classicmodels sample database:

In this context, perhaps calculating hourly payments does not make much sense, but the queries did highlight the rounding differences between the two SELECT statements (4256.653475 vs. 4256.653476).

So, why does SUM using a subquery and the GROUP BY WITH ROLLUP produce different results?

Floating-point Approximate Values Versus Fixed-point Exact Values

The floating-point (approximate value) types are FLOAT, REAL, and DOUBLE, while the fixed-point (exact value) types are INTEGER, SMALLINT, DECIMAL, and NUMERIC. Floating-point means the decimal point can be placed anywhere relative to the significant digits of the number with the actual position being indicated separately. Meanwhile, a fixed-point value is an integer that is scaled by a specific factor.

Back in version 5.5, MySQL added support for precision math, which included a library for fixed-point arithmetic that replaced the underlying C library and allowed operations to be handled in the same manner across different platforms. Since this update, if no approximate values or strings are being used in a calculation, expressions are evaluated using DECIMAL exact value arithmetic with precision of 65 digits. For GROUP BY functions, STDDEV() and VARIANCE() return DOUBLE, an approximate floating-point type, while SUM() and AVG() return a DECIMAL for exact-value arguments and a DOUBLE for approximate value.

Another ramification of the new MySQL library for fixed-point arithmetic is that type conversion is now handled using floating-point values. Thus, the results of type conversion may vary and can be affected by factors such as computer architecture, the compiler version or even the optimization level. One way to avoid these problems is to use an explicit CAST() rather than the implicit conversion.

Which Result is Correct?

Getting back to the initial queries, which value is more accurate and which one is the correct way to obtain the SUM? The truth is, neither is exactly accurate, but, by using a little algebra, the query can be simplified to yield an accurate result:

The key to accurate rounding is to work with whole numbers in as many initial steps as possible.

MySQL provides several different log files that can help you find out what's going on inside your MySQL server instance. These include:

• error log
• isam log
• general query log
• binary log
• slow log

Of these, the slow query log is especially useful for finding inefficient or time-consuming queries, which can adversely affect database and overall server performance. This blog will describe how to read and interpret slow query log output to better debug query performance.

Enabling the Slow Query Log

The slow query log consists of SQL statements that take more than long_query_time seconds to execute and require at least min_examined_row_limit rows to be examined. Hence, queries that appear in the slow query log are those that take a substantial time to execute and are thusly candidates for optimization.

The slow query log is disabled by default so as to save disk space. You can turn it on by setting the --slow_query_log variable to 1 (ON in Navicat). Likewise, providing no argument also turns on the slow query log. Likewise, an argument of 0 (OFF in Navicat) disables the log.

In Navicat, you can access system variables using the Server Monitor tool. It's accessible via the Tools main menu command. In the Server Monitor, click on the middle Variables tab and scroll down to see the slow_query_log and slow_query_log_file server variables in the list:

Reading the Slow Query Log

Examining a long slow query log can be a time-consuming task due to the huge amount of content to sift through. Here is what a typical entry in the slow log file might look like:

# Time: 140905  6:33:11# User@Host: dbuser[dbname] @ hostname [1.2.3.4]# Query_time: 0.116250  Lock_time: 0.000035 Rows_sent: 0  Rows_examined: 20878use dbname;SET timestamp=1409898791;...SLOW QUERY HERE...

To make reading the log contents easier, you can use the mysqldumpslow command-line utility to process a slow query log file and summarize its contents:

~ $ mysqldumpslow -a /var/lib/mysql/slowquery.log Reading mysql slow query log from /var/lib/mysql/slowquery.log Count: 2  Time=316.67s (633s)  Lock=0.00s (0s)  Rows_sent=0.5 (1), Rows_examined=0.0 (0), Rows_affected=0.0 (0), root[root]@localhost...SLOW QUERY HERE...

Navicat Query Analyzer

Navicat Monitor's Query Analyzer tool provides a graphical representation of the query logs that makes interpreting their contents much easier. In addition, the Query Analyzer tool enables you to monitor and optimize query performance, visualize query activity statistics, analyze SQL statements, as well as quickly identify and resolve long running queries.

In addition to the Slow Query Log, the Query Analyzer collects information about query statements using one of the following methods:

1. Retrieve the General Query Log from the server and analyze its information.
2. Query the performance_schema database and analyze it for specific performance information.

You'll find the Query Analyzer section below the Latest Deadlock Query and Process List sections:

Conclusion

This blog presented a few ways to read slow query log output to better debug the performance of your queries.

Click here for more details about all of Navicat Monitor's features, or, download the 14-day fully functional free trial!

Usually, when your database runs slower for an extended period, the culprit is more often than not a "bad" query. That is to say, a query that is not fully optimized, poorly written, or gives users the ability to fetch an unlimited number of rows from the database. We can alleviate some pain by throwing more resources at the server, but this is really a short term fix and doe not address the underlying issue. The best course of action is to identify and fix the problem query or queries, which shouldn't be too difficult, given some time and effort. Of course, the first step is to identify which query or queries is/are the stalwarts. There are a few ways to do that, depending on your specific database type. Today's blog will highlight a few strategies for MySQL.

Using the MySQL PROCESSLIST Table

The PROCESSLIST Table is one of many metadata tables within the INFORMATION_SCHEMA database. As the name suggests, it maintains information for all processes running within a database instance. There are several ways to access it, as shown in the next several sections.

Using the mysqladmin Command Line Tool

The mysqladmin command line tool ships with MySQL. Run it with the flag "processlist" (or "proc" for short) to see currently running processes. Moreover, adding the "statistics" flag (or "stat" for short) will show running statistics for queries since MySQL's last restart:

Here is some sample output:

+-------+------+-----------+-----------+---------+------+-------+--------------------+----------+| Id    | User | Host      | db        | Command | Time | State | Info               | Progress |+-------+------+-----------+-----------+---------+------+-------+--------------------+----------+| 77255 | root | localhost | employees | Query   | 150  |       | call While_Loop2() | 0.000    || 77285 | root | localhost |           | Query   | 0    | init  | show processlist   | 0.000    |+-------+------+-----------+-----------+---------+------+-------+--------------------+----------+Uptime: 781398  Threads: 2  Questions: 18761833  Slow queries: 0  Opens: 2976  Flush tables: 1  Open tables: 101  Queries per second avg: 26.543

Since this command runs on the shell interface, you can pipe output to other scripts and tools. The downside is that the PROCESSLIST table's info column is always truncated so it does not provide the full query on longer queries.

Querying the MySQL PROCESSLIST Table

One way to query the PROCESSLIST table is to run the "show processlist;" query from within MySQL's interactive mode prompt. Navicat users can execute the show processlist query directly within the SQL Editor just like any query:

Note that adding the "full" modifier to the command is sometimes required in order to disable truncation of the Info column. (This is necessary when viewing long queries.)

Using a Monitoring Tool

For more in-depth analysis of query performance, many professional database administrators (DBAs) employ a database monitor such as Navicat Monitor. It has a query analyzer that monitors queries in real time to quickly improve the performance and efficiency of your server. It shows the summary information of all executing queries and lets you easily uncover problematic queries. As you can see in the image below, Navicat Monitor can sort queries by execution time, so that the slowest can be found at a glance:

Conclusion

In this blog we learned a few easy ways to identify slow queries using the MySQL PROCESSLIST Table as well as Navicat Monitor.

Click here for more details about all of Navicat Monitor's features, or, download the 14-day fully functional free trial!

In last week's blog, we learned about the ramifications of poor indexing, as well as how to choose which columns to include as part of a clustered index. In this article, we'll cover how the same indexes that provide better performance for some operations, can add overhead for others.

How Clustered Indexes Affect INSERTs, UPDATEs and DELETEs

In general, having indexes on tables comes with the additional cost that more data pages and memory is used. On Clustered tables, the effects of indexes are even more pronounced. A Clustered table is one where a clustered index is used to store the data rows sorted based on the clustered index key values. SELECT statements tend to execute noticeably faster on a clustered table, whereas INSERTs, UPDATEs, and DELETEs require more time, as not only data is updated, but the indexes are updated also. For clustered indexes, the time increase is more significant than on single indexes, as the records have to maintain the correct order in data pages. Whether a new record is inserted, or an existing deleted or updated, this usually requires the records to be reordered.

INSERTs tend to perform fastest on a table without any indexes. This is because neither re-ordering nor index updating are required. On the same table, executing UPDATEs and DELETEs is the most expensive. The reason is that the database requires most time to find the specific records within the table.

Conversely, the costs can be higher for a table with a non-optimal clustered index, followed by on tables with a non-clustered index or no indexes at all.

With regards to SELECT statements, you can reduce the cost of execution by:

• specifying the list of returned columns, and
• executing the statement on the table where the clustered index is created on the primary key column

Examples of DML Impact

We can see the impact of indexes on DML (Data Manipulation Language) statements on the following album table, whose definition shows a large number of indexes:

In Navicat, we can view index details in the Table Designer' Indexes tab:

Both the Index Type and method may be chosen via drop-downs, which are tailored specifically to the database type. Here are the available selections for MySQL 7:

By running a simple benchmark we can test the insert rate of the current album table with the original definition that includes a Primary Index only:

Here are the timed results:

Inserting data into the table with additional indexes was four times slower in my informal simple bulk tests. There are other factors that can contribute to the slower speed; however, my results provide a representative indication that adding indexes to a table has a direct effect on write performance.

Conclusion

As shown, indexes can speed up some queries and slow down others. In this article, we provided some basic guidelines for clustered and non-clustered indexes, as well as which columns are preferred to build indexes on, and which should be avoided. Finding the right balance between the benefits and overhead indexes bring provides optimal performance to your queries and stored procedures.

Interested in Navicat Premium? You can try it for free for 14 days!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

It is common knowledge that judicious use of indexes can help SELECT queries execute significantly faster. This can tempt some database admins (DBAs) to try to milk as much performance gains as possible by adding indexes to every column that might possibly be included in a query. The downside to adding indexes to a table is that they affect the performance of writes. Moreover, improperly created indexes can even adversely affect SELECT queries! Any table configuration where performance suffers due to excessive, improper, or missing indexes is considered to be poor indexing. In today's blog, we'll learn about the ramifications of poor indexing, as well as cover how to choose which columns to include as part of a clustered index.

The Effects of Poor Indexing

A poor index can be an index created on a column that doesn't provide easier data manipulation or an index created on multiple columns which, rather than speed up queries, slows them down.

If indexes are not created properly, the database has to go through more records in order to retrieve the data requested by a query. Therefore, it uses more hardware resources (processor, memory, disk, and network) and makes fetching the data take longer.

A table without a clustered index can also be considered as a poor indexing practice in some cases. Execution of a SELECT statement, inserting, updating, and deleting records is in most cases slower on a heap table (i.e. a table without a clustered index) than on a clustered one.

Choosing Columns For Clustered Indexes

When you create a table with a primary key (PK) in a relational database, a unique clustered index is automatically created on the primary key column. Although this default action is perfectly acceptable in most cases, this might not be the optimal index for your data.

The columns that make up a clustered index should form a unique, identity, primary key, or any combination where values are increased for each new entry. As clustered indexes sort the records based on the value, using a column already ordered ascending, such as an identity column, is a good choice.

A column whose value changes frequently should not be used for a clustered index. The reason is that each change of the column used for the clustered index requires the records to be reordered. This re-ordering can easily be avoided by using a column that is updated less frequently, or ideally, not updated at all.

Likewise, columns that store large data, such as BLOB columns (text, nvarchar(max), image, etc.), and GUID columns are not ideal for clustered indexes. This is because sorting large values is highly inefficient, and in case of GUID and image columns, doesn't make much sense.

Finally, a clustered index should not be built on a column already used in a unique index.

Conclusion

In today's blog, we learned about the ramifications of poor indexing, as well as how to choose which columns to include as part of a clustered index. In an up-coming article, we'll cover how the same indexes that provide better performance for some operations, can add overhead for others.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Databases play a central role in of most business processes and applications. As IT infrastructures become more diverse and sophisticated, it becomes increasingly important to be able to nip database issues in the bud. In simpler times, one or more database administrators (DBAs) could resolve issues manually as they came up in true fire fighter fashion. Today, that approach is almost certainly doomed to fail.

Smart DBAs rely on database monitoring to not only pinpoint trouble quickly, but even to predict future issues before they cause real problems. In this article, we'll examine what database monitors do. In up-coming installments we'll learn more about how they work and explore some best practices for using monitoring software.

Database Monitoring Explained

Simply put, database monitoring is the tracking of database performance and resources using key metrics with a goal of enabling high performance and availability to more fully support an organization's application infrastructure. Categories of common metrics for for database monitoring include:

• Query details (top CPU, slow running, and most frequent)
• Session details (current user connections and locks)
• Scheduled jobs
• Replication details
• Database performance (buffer, cache, connection, lock)

Data from each of these categories is analyzed in order to minimize, or ideally, prevent database outages or slowdowns. The selection of the data points and how they are analyzed will vary based on the type of database. Moreover, the above metrics (and many others) are typically monitored in real time, thus allowing you to identify or predict issues. When done properly, effective database monitoring gives you the opportunity to enhance or optimize your database, in order to augment overall performance.

To maximize the efficacy of your database monitoring strategy, you should analyze data across a range of categories, with the intention of minimizing or preventing lags or unavailability. In that regards, note that different types of databases will require different metrics (and/or data points) to be analyzed.

Ideally, database monitoring tracks the performance of both hardware and software by taking frequent snapshots of performance indicators. This allows you to identify any changes, identify bottlenecks, and pinpoint the exact moment problems started to occur. With this information in hand, you can then rule out potential causes, and can address the real root cause of the issue.

Monitors Are Not All Created Equally

There are several competing products on the market that all provide similar functionality. If you use MySQL, MariaDB, SQL Server, or Cloud databases like Amazon RDS, Amazon Aurora, Oracle Cloud, Google Cloud or Microsoft Azure, you should consider Navicat Monitor. Although it comes packed with powerful features to make your monitoring effective as possible, that in itself is not the sole reason for choosing it. The most important feature is that Navicat Monitor provides agentless remote server monitoring. As such, it does not require any software to be installed on monitored servers, thus leaving their full resources available to process requests. Another benefit to utilizing agentless architecture is that Navicat Monitor can be accessed from anywhere via a web browser. With web access, you can easily and seamlessly keep track of your servers around the world, around the clock.

Conclusion

In this article, we examined the main functions of relational database monitors. In up-coming installments we'll learn more about how they work and explore some best practices for using monitoring software.

Interested in Navicat Monitor? You can try it for free for 14 days!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

In last week's blog we learned about the potential uses and advantages to utilizing Database Partitioning when working with large data sets. In today's follow-up, we'll create a MySQL partition in Navicat for MySQL using the HASH partitioning criteria.

Launching the Partitioning Dialog in Navicat

In Navicat, you'll find the Partition button on the Options tab of the Table Designer, at the bottom of the page:

Click this button to open the Partition dialog.

Creating a HASH Partition on a Table

The very first control on the Partition dialog is the Partition By drop-down:

The types of partitioning supported depend on the database type and version. Here are the options that you'll find in Navicat for MySQL 7:

• Range partitioning: Range (or Interval) partitioning is useful when organizing similar data - especially date and time data. Hence, Range partitioning is ideal for partitioning historical data.
• List partitioning: Explicitly maps rows to partitions based on discrete values. For example, all the customers for southern states could be stored in one partition while customers from northern states would be stored in different partitions.
• Composite partitioning: Partitions on multiple dimensions, based on identification by a partitioning key. For example, you may decide to store data for a specific product type in a read-only, compressed format, and keep other product type data uncompressed. Composite partitioning also increases the number of partitions significantly, which may be beneficial for efficient parallel execution.
• Round-robin partitioning: Assigns rows in a round-robin manner to each partition so that each partition contains a more or less equal number of rows and load balancing is achieved. In this case there is no partition key, so rows are distributed randomly across all partitions.
• Hash partitioning: Randomly distributes data across partitions based on a hashing algorithm, rather than grouping similar data. Useful for times when it is not obvious in which partition data should reside, although the partitioning key can be identified. Hence, data is distributed such that it does not correspond to a business or a logical view of the data, as it does in Range partitioning.

Some Caveats

In order to benefit from Partitioning, you'll want to make sure that:

• If you do supply the column on which to partition the table, that it is a part of every unique key in that table.
• You are partitioning the table on the column(s) which is/are most commonly utilized in your queries. Otherwise, there will be no benefit from creating partitions.

Defining the Partition Details

The Partition dialog supports many options, including subpartitions as well as the ability to manually create partition definitions. However, for a simple HASH partition, we only need to provide the partition criteria, (table column) and number of partitions:

Click the OK button to create the partition in one easy step!

On the SQL Preview tab, you can view the SQL statement that was generated by Navicat:

ALTER TABLE `sakila2`.`film` PARTITION BY HASH (actor)PARTITIONS 10(PARTITION `p0` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p1` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p2` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p3` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p4` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p5` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p6` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p7` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p8` MAX_ROWS = 0 MIN_ROWS = 0 ,PARTITION `p9` MAX_ROWS = 0 MIN_ROWS = 0 );

Conclusion

In today's blog, we created a MySQL partition in Navicat for MySQL using HASH partitioning criteria.

Interested in Navicat for MySQL? You can try it for free for 14 days!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Any time that you categorize data into different types, the need to convert from one data type to another is inevitable. Off the top of my head, a common use case is to process variables that were passed in from a web form via a query parameter or POST request body. Serializing data in order to send it across the network tends to coerce all variables into strings. As such, they often need to be converted into a more appropriate data type, such as a number, date, or what-have-you.

In relational databases, reasons for converting one data type to another include porting data from one database type to another, changing the data type of a column, or temporarily switching between data types for evaluation. In MySQL, we can convert between data types using the CAST() and CONVERT() functions. In today's blog, we'll learn how to employ both functions using examples to illustrate their usage.

What's the Difference?

Both CAST() and CONVERT() can change data types in MySQL. Since the two are so similar, many SQL newbies (and some more experienced users!) wonder what the difference is. The main difference is that CONVERT() can also convert the character set of data into another character set. CAST() cannot be used to change character sets. Hence, CAST() should be your goto conversion function, unless you need to convert a character set.

The CAST() Function

MySQL CAST() accepts two inputs:

• the data to be typecasted
• the data type (decimal, char, etc.) to which you want to convert this data. You can cast data into BINARY, CHAR, DATE, DATETIME, TIME, DECIMAL, SIGNED, UNSIGNED data types.

Here's the syntax:

CAST(data as data_type)

An Almost Real-life Example

One useful application of the CAST() function is to make a very large data type less unwieldy (more wieldy?). The following query returns information about a particular film in the MySQL Sakila Sample Database. One of the columns - description - is a text field. That means that it can store a huge amount of text! We can use CAST() to truncate the description to 100 characters, so that we don't get a whole book about the movie:

Speaking of the Sakila Sample Database, did you know that it's named after MySQL's dolphin mascot? It was chosen from a huge list of names suggested by users in a "Name the Dolphin" contest. The winning name was submitted by Ambrose Twebaze, an Open Source software developer from Eswatini (formerly Swaziland), Africa.

The CONVERT() Function

The CONVERT() function's syntax is similar to CAST(), but the expression and result type are supplied in a slightly different format. One way is to supply two separate arguments:

CONVERT(expr, data_type)

Other than that, the data_type parameter can be any of the same types that are supported by the CAST() function.

An Not Quite Real-life Example

Since the major difference between CAST() and CONVERT() is that the latter can the character set of a column into a different one, let's show that in action.

The first thing to be aware of is that the syntax is a little different for converting character sets. In that case, we need to add the USING keyword between the expression and character set:

CONVERT(expr USING charset);

In Navicat for MySQL (or Navicat Premium), we can see a table's character set and collation on the Info Pane:

With that in mind, we could apply the CONVERT() function to the previous query to convert the description field from UTF-8 to Latin1. In case you're curious, the difference between the two is that, in latin1, each character is exactly one byte long, while, in utf8, a character can consist of more than one byte. Consequently utf8 has more characters than latin1. Moreover, the characters they do have in common aren't necessarily represented by the same byte/bytesequence.

Conclusion

In today's blog, we saw how to use CAST() to convert data into a different type and how to convert between character sets using CONVERT(). To reiterate, CAST() should be your goto conversion function. CONVERT() is better suited for switching between character sets.

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

The MAX() function is often used to return the largest value of a given column. It's not picky about types, so the column may contain salaries, best before dates, or last names. The question is, can the MAX() function also find the highest value across multiple columns? The short answer is Yes. The longer explanation is that it depends on the database you're using. In today's blog, we'll explore a few ways to obtain the maximum value among two or more columns, either using the MAX() function, or an even better alternative.

The MySQL Solution

If you're working with MySQL, you can combine MAX() with the GREATEST() function to get the biggest value from two or more fields. Here's the syntax for GREATEST:

GREATEST(value1,value2,...)

Given two or more arguments, it returns the largest (maximum-valued) argument. If any argument is NULL, GREATEST returns NULL.

An Example

If you're going to look for the maximum value across fields, it helps to compare columns that contain similar data - apples against apples, so to speak. The classicmodels database's products table contains two similar columns: "buyPrice" and "MSRP". Both store dollar figures as decimal data:

Ideally, the GREATEST() input parameters should be scalar values. As it happens, the MAX() function returns the largest value in a column! Here's the Query and result in Navicat for SQL Server:

Not surprisingly, the MSRP contained the highest value. Otherwise, the company might want to consider a different vendor.

Some Other Solutions

For other database that don't support the GREATEST() function, there are ways to compare multiple columns using MAX(). It just takes a bit of creativity! Here are a few solutions, using SQL Server:

UNION ALL

The UNION ALL command combines the result set of two or more SELECT statements. Unlike the UNION command, UNION ALL includes duplicates. In any event, either command may be utilized to combine different columns into one long result set. Its results may then be treated as a subquery from which the maximum value is derived:

SELECT MAX(T.field) AS MaxOfColumnsFROM (    SELECT column1 AS field    FROM YourTable     UNION ALL    SELECT column2 AS field    FROM YourTable    UNION ALL    SELECT column3 As field    FROM YourTable) AS T

Here's an example query against the Sakila Sample Database in Navicat for SQL Server that includes both the rental and return date from the rentals table:

Select MAX from VALUES

The SQL VALUES keyword is not just for INSERTs. You can also SELECT from a list of values using the following syntax:

select (values (1), (2), (3)) as temp(c)

This statement can be expanded to serve our purpose as follows:

SELECT (  SELECT MAX(myval)   FROM (VALUES (column1),(column2),(column3)) AS temp(myval)) AS MaxOfColumnsFROMYourTable

We can use this template to serve as a basis for our query against the rentals table:

Conclusion

AS we saw here today, there are several ways to obtain the maximum value across multiple columns. These include using the GREATEST() function, or by getting a bit creative with the MAX() function.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Like most database developers, you've probably written your fair share of queries that search for that proverbial needle in a haystack of text or binary data. I know I have! Perhaps even more important than the SELECT statements that you write against the database are the indexes that it contains. To that end, an inverted index can go a long way towards making mounds of data accessible in an expeditious manner. In today's blog, we'll learn what inverted indexes are, and how to use them in your databases, using MySQL as an example.

Forward Index versus Inverted Index

Inverted indexes were actually invented decades ago, around the same time that much of the first AI and machine learning algorithms were born. However, it wasn't until recent increases in computing power that it became possible to make use of inverted indexes in traditional relational databases. Inverted indexes allows information in relational databases to be found much faster as well as allow queries to be far more complex and specific.

Unlike a regular (forward) index, that maps table rows to a list of keywords, an inverted index maps the keywords to their respective rows. Here's a side-by-side comparison:

Searching using a forward index is a slower process because the database engine has to look at the entire contents of the index to retrieve all pages related to a word. Meanwhile, searching via an Inverted Index is very fast because there are no duplicate keywords in the index and each word points directly to the relevant row(s).

Inverted Indexes in MySQL

MySQL's InnoDB engine implements Full-text indexes on text-based columns (CHAR, VARCHAR, or TEXT columns) to speed up queries and DML operations on data contained within those columns. Full-text indexes employ an inverted index design so that each keyword in the index points to a list of documents that the word appears in. It also supports proximity searches, whereby two or more words that occur within a certain number of words from each other may also be located, by storing position information for each word.

In Navicat database administration development tools, such as Navicat for MySQL and Navicat Premium, you can view a table's engine in the General Information panel:

Assuming that your table uses the InnoDB engine, you can assign a FULLTEXT index via the Index Type drop-down on the Indexes tab of the Table Designer. Here's an example of the perfect column on which to add a FULLTEXT index - the Description column on the Sakila Sample Database's Film table:

Text fields such as this are good candidates for an Inverted Index because there are so many words and phrases to search on:

Conclusion

Inverted indexes are a great way to speed up your queries while allowing them to be far more complex and specific. Just be aware that the indexing process takes longer than it does for forward indexes.

Interested in Navicat for MySQL or Navicat Premium? You can try both for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Avoiding and/or Minimizing Deadlocks

In relational database systems (RDBMS), a deadlock is a situation where two concurrent transactions cannot make progress because each one is waiting for the other to release the lock. In Part 1 of this series, we we established what Object Locking is in Relational Databases, the different types of locks, and deadlocking. Then, in Part 2, we compared the pros and cons of Pessimistic and Optimistic locking. In this installment, we'll be exploring a few causes of deadlocks, as well as strategies for avoiding, or at least, minimizing them.

Inefficient Queries

Deadlocks are unavoidable to some degree, but their infrequent occurrence does not automatically spell disaster, as long as one of the two transactions ends in a timely fashion. As it turns out, one of the most common sources of blocking issues are long and inefficient SQL statements that cause the database to "hang" while they run their course. These may be remedied in two steps:

• Optimize poorly performing SQL statements so locks are released in the shortest time possible.
• Identify whether the locks can be released before any long-running SQL statements are executed within the same session.

For example, if locks are acquired due to a DELETE statement being executed and is immediately followed by a SELECT statement that performs a complete table scan, you should ascertain whether it may possible to execute a COMMIT statement between them. This should help the locks release earlier.

Nested Transactions

Another frequent cause of blocking issues are sleeping sessions that have lost track of the nesting level of the transaction. For example, if an application cancels an SQL statement or is timed out but doesn't issue a COMMIT or ROLLBACK statement, then resources could remain locked indefinitely. Some ways to deal with this issue include:

• Following any application error, submit an IF@@TRANCOUNT > 0 ROLLBACK TRAN statement within the application's error handler.
• Include the SET XACT_ABORT ON statement in any stored procedures that start transactions - especially if they aren't cleaning up after an error. By doing so, should a run-time error occur, any open transactions will be aborted and control returned to the client.
• If connection pooling is employed by an application that opens the connection and runs a few queries before returning the connection to the pool, then you may want to consider temporarily disabling connection pooling. By doing so, the DB Server connection is physically logged out, resulting in the server rolling back any open transactions.

Fetching Partial Results

A lesser known source of deadlocks are applications that don't fetch all result rows in one go. This is a problem because, when a query has been sent to the server, applications must be able to fetch all result rows to completion. If this doesn't happen, locks can be kept on tables, which results in blocking for other users. Therefore, try to code your applications so that they fetch all of the rows they need rather than spread it out over several iterations.

Conclusion

Today's blog listed a few causes of deadlocks, as well as strategies for avoiding them and dealing with deadlocks when they do occur. Next week, we'll be moving on to an entirely new subject.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Pessimistic versus Optimistic Locking

Relational database systems (RDBMS) employ various locking strategies to enforce transaction ACID properties when modifying (e.g., UPDATING or DELETING) table records. On occasion, deadlock may occur when two concurrent transactions cannot make progress because each one is waiting for the other to release the lock. In Part 1 of this series, we we established what Object Locking is in Relational Databases, the different types of locks, and deadlocking. In today's follow-up, we'll be comparing the pros and cons of Pessimistic and Optimistic locking.

Pessimistic Locking

With Pessimistic Locking, a resource is locked from the time it is first accessed in a transaction until the transaction is finished, making it inaccessible to other transactions during that time. In situations where most transactions simply read the resource and never update it, an exclusive lock may be overkill as it leads to more lock contention (deadlocks). Thinking back to the banking example from Part 1, the account would be locked as soon as it was accessed in a transaction. Any attempt to use the account in other transactions while it was locked would either result in the other process being delayed until the account lock was released, or that the process transaction would be cancelled and rolled back to the previous state.

Optimistic Locking

Using optimistic locking, a resource is not actually locked when it is first is accessed by a transaction. Instead, the pristine state of the resource is persisted. Other transactions are still able to access to the resource, making the possibility of conflicting changes a known risk. At commit time, when the resource is about to be updated in persistent storage, the state of the resource is read from storage again and compared to the state that was saved when the resource was first accessed in the transaction. If the two states differ, that means that a conflicting update was made, so the transaction is rolled back. In our banking example, the amount of the account would be saved when it's first accessed. If the transaction changed the account amount, the amount would be read from the store again just before the amount was about to be updated. If the amount had changed since the transaction began, the transaction would fail itself, otherwise the new amount would stand and be saved.

Deciding Between Pessimistic and Optimistic Locking

Now that we've covered what both types of locking are, the question becomes which to use. In most cases, Optimistic Locking is more efficient and offers higher performance. Meanwhile, Pessimistic Locking provides better integrity on the data, BUT, management of the lock is harder as there is a greater chance of encountering deadlocks. When choosing between pessimistic and optimistic locking, consider the following three guidelines:

• Pessimistic locking is useful if there are a lot of updates and relatively high chances of users trying to update data concurrently.
• Pessimistic locking is also more appropriate in applications that contain small tables that are frequently updated. In the case of these such "hotspots", conflicts are so likely that optimistic locking wastes effort in rolling back conflicting transactions.
• Optimistic locking is useful when the possibility for conflicts is very low, i.e., there are many records but relatively few users, or very few updates and mostly read operations.

Conclusion

In this blog, we compared Pessimistic versus Optimistic locking. In the next installment, we'll be exploring strategies for recovering from a deadlock situation.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Part 1: Overview, Lock Granularity, and Deadlocks

Recently, we've had a few blogs about database transactions and they enforce the the four ACID (Atomicity Consistency Isolation Durability) properties. In today's blog, we'll be taking a look at another mechanism employed by relational databases (RDBMS) to enforce ACID properties, namely, Object Locking. Specifically, we'll learn what it is, what role(s) it plays in RDBMS transactions, and some of the side effects locking may cause. While Database Object Locking can be a fairly technical and complicated subject, we're going to break it down into layman's terms here and keep things as simple as possible.

What Is Object Locking?

Simply put, Object Locking is a way to prevent simultaneous access to data in a database, in order to avoid data inconsistencies. To illustrate how Object Locking works, imagine two bank tellers attempting to update the same bank account for two different transactions. Both tellers retrieve (i.e. copy) the account's record. Teller A applies and saves a transaction. Teller B applies a different transaction to his/her saved copy, and saves the result, overwriting the transaction entered by teller A. Now the record no longer reflects the first transaction, as if it had never even happened!

The fix is to lock the record whenever it is being modified by any user, so that no other user can alter it at the same time. This prevents records from being overwritten incorrectly, but allows only one record to be processed at a time, locking out other users who need to edit records at the same instance. Hence, anyone attempting to retrieve the same record for editing is denied write access because of the lock (depending on the exact implementation, they may be still be able to view the record in a read-only state). Once the record is saved (or edits are canceled), the lock is released. By preventing records from being saved so as to overwrite other changes, data integrity (the I in ACID) is preserved.

Lock Granularity

The above example demonstrated an instance of record-level locking. Now imagine if the two bank tellers above were serving two different customers, but both their accounts were contained in one ledger. In that situation, then the entire ledger - or, one or more database tables - would need to be locked for editing. As you can imagine, locking entire tables can lead to a lot of unnecessary waiting. If the tellers could remove one page from the ledger, containing the account of the current customer (plus a few other accounts, perhaps), then multiple customers can be serviced concurrently, provided that each customer's account is found on a different page than the others. If both customers have accounts on the same page, then only one may be serviced at a time. This is an analogy of page-level locking in a database.

There are four types of locks. Here they are in increasing granularity:

• database locks
• table locks
• page locks
• row locks

Lock Granularity and Deadlocks

The utilization of granular locks creates the possibility for a situation called "deadlock". A deadlock may occur when incremental locking (locking one entity, then locking one or more additional entities) is utilized. To illustrate, my wife and I often transfer money between our personal accounts. If we were to each ask a teller to obtain our individual account information so we could transfer some money into the other spouse's account, the two accounts would essentially be locked. Then, when our tellers attempted to transfer money into each other's accounts, they would each find the other account to be "in use", forcing them to wait for the accounts to be freed up. Unknowingly, the two tellers are waiting for each other, and neither of them would be able complete their transaction until the other gives up and returns the account! Thankfully, various techniques have been devised to circumvent such problems. These will be addressed in the next installment.

Going Forward

In today's blog, we established what Object Locking is in Relational Databases, the different types of locks, and deadlocking. In the next installment, we'll review some collision resolution strategies, as well as pessimistic versus optimistic locking.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Both DBeaver and Navicat are Universal Database Tools, which means that they support all popular databases, including MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite. Moreover, both are compatible with cloud databases as well, such as Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas. But, as the saying goes, "the devil is in the details", so, while the two products may seem similar at first glance, a closer examination of each tool's Visual Appeal, Secure Connectivity, and NoSQL Support will reveal that the differences between the two far outnumber any apparent similarities.

Visual Appeal

Perhaps visual appearance is not the first thing one thinks of when considering application features, but how an application's GUI looks can tell us a lot about what kind of user experience (UX) it provides. Here are side-by-side screen captures of DBeaver and Navicat main screens in Windows:

There can be little doubt that both products have well designed GUIs. Having said that, Navicat's interface is cleaner and more intuitive, IMHO. Here are a few reasons why:

• All of the main actions are accessible via menu items at the top of the screen.
• There is a large button toolbar for accessing other application screens and utilities.
• Different Object types are identified by distinct icons, as seen in the left pane.

Secure Connectivity

For business professionals, it is imperative to be able to connect securely to database instances.

DBeaver support configuration of standard (host, port, user credentials) as well as advanced connection properties. These include SSH tunnel, SOCKS proxy, and Shell commands to be executed before/after actual database connection.

Navicat establishes secure connections through SSH Tunneling and SSL to ensure that every connection is secure, stable, and reliable. Supported authentication methods include PAM authentication for MySQL and MariaDB, Kerberos and X.509 authentication for MongoDB, and GSSAPI authentication for PostgreSQL. Navicat provides more authentication mechanisms than DBeaver, and most of its competitors for that matter!

NoSQL/BigData Database Support

Due to their many and significant differences to traditional relational databases, NoSQL databases such as MongoDB present their own unique requirements.

DBeaver has special extensions for MongoDB, as well as other document databases. NoSQL databases have an SQL interface so that you can work with them in the same way as you would relational databases.

Navicat is fully compatible with MongoDB out of the box. Navicat also takes a different approach to working with NoSQL databases. Rather than try to use MongoDB as an SQL database, it uses MongoDB's proper syntax for managing data, so that developers may utilize its full capabilities:

Moreover, Navicat can present NoSQL data in one of three ways, for working with documents in various capacities. They are:

• Grid view
• Tree view
• JSON view

Grid View (pictured above) is the traditional tabular display that DBAs are most familiar with. It can handle any documents size, and supports advanced features like highlighting cells based on data types, column hiding and more.

Tree View shows your documents in a hierarchical view. All embedded documents and arrays are represented as nodes, which can be expanded or collapsed as needed:

You can also show your data as JSON documents, while documents can be added with the built-in validation mechanism which ensures your edits are correct.

Conclusion

In part 2 of this series on DBeaver vs. Navicat Premium, we compared the Visual Appeal, Secure Connectivity, and NoSQL Support of both products. As we saw, while both look similar to some degree, if one delves beneath the surface, there are some enormous differences between the two.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

In my early days as an IT consultant, I relied on a variety of open source tools to accomplish my tasks. My rational was that I was saving money by costs associated with commercial products. It would be a few years later that I would come to realize that commercial products can actually save time and money by streamlining and automating many of the common tasks that we tend to perform on a regular basis.

Database clients are a category of software that many developers shy away from spending money. The assumption here is that you don't need a lot of features to view database tables and perform queries against them. That may be true, to a point, but if you find yourself doing a lot of database work, it may be high time to upgrade your DB client.

I was recently introduced to a free universal database tool called DBeaver. Not knowing much about it, I thought that it might be informative to compare it to Navicat Premium. Let the showdown begin and may the best product win!

About the Competitors

In the left corner, we have the challenger: DBeaver. It's a free and open source universal database tool for developers, database administrators, or anyone who needs to work with data in a professional capacity. Written in Java and based on Eclipse platform, DBeaver uses the JDBC application programming interface (API) to interact with databases via a JDBC driver. For other databases such as NoSQL it relies on its own proprietary database drivers.

Like many open source tools, DBeaver was started in 2010 as a hobby project. It was meant to be free and open-source with an appealing UI. From its early days, the focus was to include the most frequently utilized features of database developers. The first official release was in 2011 on Freecode. It quickly became a popular tool in the open-source community.

In the right corner, we have the reigning champion: Navicat Premium. It is a commercial database development and design tool that allows you to simultaneously connect to multiple local and/or cloud databases from a single application. It was designed to meet the needs of a variety of audiences, from database administrators and programmers to various businesses/companies that serve clients and share information with partners.

The main goal of the initial version of Navicat was to simplify the management of MySQL instances. In 2008, Navicat for MySQL was the winner of the Hong Kong ICT 2008 Award of the Year, Best Business Grand Award and Best Business (Product) Gold Award. Navicat Premium was launched in 2009. It combined all previous Navicat versions into a single product and could connect to all popular database types simultaneously, giving users the ability to perform data migration between different (heterogeneous) database types.

Conclusion

Now that we've introduced our participants, the next installment(s) will delve into each tool's feature set, and compare them for usability, performance, user ratings, and more!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Back in August of 2020, The Many Flavors of the SQL Count() Function provided an overview of COUNT's many input parameter variations. Another way to use the COUNT() function is to combine it with the GROUP BY clause. Using the COUNT() function in conjunction with GROUP BY is useful for breaking down counts according to various groupings. In today's blog, we'll learn how to group counts by different criteria by querying the Sakila Sample Database using Navicat Premium as our database client.

Case 1: Actors Who Have Appeared In Most PG Movies

By itself, the COUNT() function could tell us how many actors have appeared in PG Movies. However, if we wanted to know how many PG movies each actor has appeared in, we would need to add the actor_id to the GROUP BY clause. Recall that the GROUP BY clause groups records into summary rows and returns one record for each group. GROUP BY queries often include aggregate functions such as COUNT, MAX, SUM, AVG, etc.

Here is the SELECT statement, along with the query results, as shown in Navicat Premium:

Notice that, when using GROUP BY, we can also order records by counts in descending order so that actors with the highest number of PG films in their filmography appear at the top of the results.

Case 2: Number of Films Rented Per Day

Applying the COUNT() function to the rental table can tell us how many movies have been rented in total. For more detailed counts, we need to turn to the GROUP BY clause. For example, we can break down counts by individual days by grouping by the rental_date. We also have to specify that the return_date must not be NULL, so that we don't count movies that have been rented but not yet returned.

In this case, ORDER BY is not required because results are automatically ordered by the grouped column, i.e., the rental date.

Case 3: Number of Films Rented by Customer Per Month

After a couple of relatively simple examples, it's time to ratchet up the level of difficulty a bit. A GROUP BY clause can group on multiple fields to obtain even more fine-grained tabulations. Case in point, this query counts movie rentals for each customer, month, and year:

In this case, we included the ORDER BY to sort results by customers' last names (last_name) rather than by customer_id. A similar thing is being done with the months in that month names are displayed, but grouped and ordered according to the month number.

Conclusion

In today's blog, we learned how to group counts by different criteria by querying the Sakila Sample Database using Navicat Premium. As we saw here today, using the COUNT() function in conjunction with GROUP BY is useful for breaking down counts according to various groupings. In fact, you'd be hard-pressed to obtain the same data without combining COUNT() with GROUP BY!

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Part 4: Miscellaneous Functions

This last category of important SQL Server functions includes those that deal with nulls, conversion, and control flow. Far from leftovers, these functions are among some of the most useful you'll ever come across!

COALESCE

Anytime you select a column whose value is not mandatory, you're bound to encounter null values. That only makes sense, because null values represent absent or missing information. Trouble is, nulls can reek havoc when included in calculations as well as other operations that one might perform on column data.

The COALESCE function accepts a list of arguments and returns the first one that does not contain a null value. Hence, SQL Server proceeds through each input parameter you provided until it either encounters one that isn't null or simply runs out of arguments. Here's its syntax:

COALESCE(val1, val2, ...., val_n)

It is commonplace to substitute a value of zero in the place of null. In some instances a different value may make more sense. for example, the film table of the Sakila Sample Database contains a column named original_language_id for films that are not originally in English. We can employ COALESCE to set its value to 1 (the language_id for English) whenever a null is found:

CONVERT

Converting an output value into a specified data type is par for the course in database work. In SQL Server, you can change the data type of a value to another using the CONVERT function. Its syntax is simple:

CONVERT(type, value)

One good reason to use CONVERT is for removing the time portion from a datetime field. Here's a query that shows the same field in its original datetime format and without the time portion:

IIF

If/else statements are the most commonly used control flow structures in programming. SQL Server provides the power of the if/else statement to our queries in the form of the IIF function. Its syntax is:

IIF(expression, value_if_true, value_if_false)

We can utilize the IIF function to separate film lengths into three groups - Short, Medium, and Long - depending on their lengths. We'll categorize film lengths as follows:

• Short: under 80 minutes
• Medium between 80 and 120 minutes
• Long: over 120 minutes

We use the IIF() function to compare the length of the film to a given expression, and depending on the result, it returns a 1 or a NULL. If it returns a 1, it will be counted under that column heading (Short, Medium, or Long):

Conclusion

That brings us to the end of our series on the most important SQL Server Functions. As mentioned at the beginning of the series, it's helpful to know the exact function names and signatures for your specific database type because they can vary from provider to provider. Case in point, the IIF function is only found in Microsoft products.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Part 3: Date Functions

After twenty years in IT, I can confirm that dates and times can be notoriously difficult to work with. Thankfully, modern relational databases like SQL Server provide a wealth of highly useful functions for this purpose. In today's blog, we'll explore some of the most popular ones.

Getting the Current Date and Time

Every programmatic language requires a way to get the current date and/or time. In SQL Server, there are a couple of ways to get the current date and time, via the CURRENT_TIMESTAMP and GETDATE() functions. Both return the current date and time, in a 'YYYY-MM-DD hh:mm:ss.mmm' format:

So, why the two functions? As you can see in the above screenshot, GETDATE() requires parentheses, while CURRENT_TIMESTAMP does not. That makes it ideal for setting the default value of auditing fields such as create and last modified columns:

DATEPART

Being able to get the current date and time is one thing, but sometimes you need to parse out individual date parts. That's where the DATEPART() function comes in. It returns a specified part of a date as an integer value. Here's its syntax:

DATEPART(interval, date)

The interval parameter has to be a specific date part or abbreviation. For example, the year can be expressed as either year, yyyy, or yy. Here's the full list:

• year, yyyy, yy = Year
• quarter, qq, q = Quarter
• month, mm, m = month
• dayofyear, dy, y = Day of the year
• day, dd, d = Day of the month
• week, ww, wk = Week
• weekday, dw, w = Weekday
• hour, hh = hour
• minute, mi, n = Minute
• second, ss, s = Second
• millisecond, ms = Millisecond

The following query that breaks the current date into its day, month, and year constituents:

DATEFROMPARTS

Date/time functions can also help us construct a date from disparate pieces of data. It accepts a year, month, and day as input parameters and combines them to form a complete date:

DATEFROMPARTS(year, month, day)

Here's an example:

DATEADD

Adding and subtracting date/time intervals to and from a date is among the most common operations on dates. In SQL Server, the function to do that is DATEADD. It accepts three input parameters: the interval to add, how many, and the date to apply the intervals to:

DATEADD(interval, number, date)

The intervals accepted by DATEADD are identical to those of DATEPART, which we say earlier, so I won't repeat them here. Instead, let's take a look at a couple of examples of this important function.

Our first example adds three months to today's date:

To subtract an interval, just provide a negative number parameter:

DATEDIFF

Our last function return the difference between two date values, as expressed by the provided interval (see above for the full list of accepted values):

DATEDIFF(interval, date1, date2)

The following query returns the difference between two dates in months:

The first date would normally be considered to be the earlier one, so if the second date parameter precedes the first, then the DATEDIFF result is expressed as a negative value:

Conclusion

In today's blog, we covered some of the most important SQL Server date and time functions. In the next and final installment, we'll be looking at miscellaneous functions.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Part 2: Numeric Functions

Like most modern relational database offerings, SQL Server comes loaded with an impressive collection of built-in functions. While some functions are amazingly similar across the board, exact names and signatures may vary. Therefore, it's a good idea to brush up on the SQL Server specific implementations of common SQL function. In part 1 of this series, we explored string functions. In today's installment, we'll be moving on to numerical functions, a category that is highly useful in the generation of statistics and calculated values!

Abs

These are not the Abs that people train to get ready for the beach. Rather, Abs is short for "Absolute". Hence, the Abs function accepts a numeric value as its argument and returns its absolute equivalent. In simpler terms, Abs returns the positive version of a given number, whether its positive or negative to begin with. Here's the function signature:

ABS(inputNumber)

In mathematics and statistics, deviation is a measure of difference between the value of a variable and some other value, often that variable's mean, or average. The deviation can either be signed or unsigned. The latter is where the Abs function comes in. Here's a query against the ClassicModels Sample Database that shows the signed and unsigned (absolute) deviation of customers' credit limits, grouped by city:

Round

Another extremely popular numeric function is Round. Rounding functions can vary quite a bit in their implementation; some only round to an integer, while others let you specify the number of decimal places to round to. SQL Server's Round function goes one step further, by accepting up to three arguments:

ROUND(number, decimals, operation)

• number: a floating-point (decimal) number to be rounded
• decimals: the number of decimal places to round number to
• operation: an optional parameter that affects rounding operations. If 0 (or omitted), the function performs regular rounding, whereby a number of 5 or greater increases the next digit. Any value other than 0, causes the function to truncate the result to the number of decimals.

It's extremely common to round currency values to 2 decimal places. Here's our previous query with rounded figures:

Ceiling

The Ceiling function is similar to Round, except that it always rounds up to the next integer value. Hence, both 25.01 and 25.75 would be rounded up to 26. Here's its syntax:

CEILING(number)

Let's apply the Ceiling function to our previous query by comparing the credit limits rounded to the nearest integer with those filtered through Ceiling:

Floor

Floor is the reverse of the Ceiling function; it always rounds a number down to the first integer that is less than or equal to that number. With positive numbers, Floor simply truncates decimals without altering the next highest integer. However, with negative numbers, it does increment the integer - downwards. For example, the floor of -0.5 is -1, as it is the first integer that is less than -0.5.

FLOOR(number)

Applying the Floor function to our example query without the use of Abs shows its effect on both positive and negative numbers:

Conclusion

In today's blog, we covered some of the most important numerical functions of SQL Server. In the next installment, we'll be looking at Date functions.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial businesses. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Part 1: String Utilities

There are certain functions that seem to come up in every programming language. Although SQL differs from your typical procedural programming language like C# or Java in many ways, it too comes equipped with an impressive assortment of built-in functions. These may be applied to Char, Varchar, and Text data types. Each database vendor does not implement functions in exactly the same way, so it pays to familiarize yourself with functions that are specific to the database you work with. In this series, we'll be taking a look at a few important SQL functions, as implemented by SQL Server. Today's blog will tackle string functions, while subsequent installments will explore numerical, date functions, and more!

Len

One of the most useful string functions is one that returns its length in characters (including spaces and punctuation). In Microsoft products, there is a long tradition of calling this function "Len". Here's the function signature:

LEN(inputString)

As an example, we'll execute a real query against the Sakila Sample Database using Navicat for SQL Server as our database client. The query selects the top 10 longest titles from the film table in descending order:

Trim

Looking to trim some fat off of a string? Then the trim function is for you! It eliminates excess spaces and tabs from the beginning and end of a string that we pass in as its argument. Here's the signature for trim:

TRIM(inputString)

We can use trim to find out if any of our film titles contain any leading or trailing spaces by comparing the length of the trimmed title to what's there currently:

Concat

In programming, the combining of strings is known as concatenation. Hence, the concat function combines two or more strings that we pass in as its arguments. Here's its signature:

CONCAT(string1, string2, ...., string_n)

The concat function is really useful to format multiple columns together in a way that works for you and your users. The following query combines the ID, title, and release year for each film and separates them using commas:

Upper & Lower

These two counterpart functions take a string argument and return the same string but with all its characters cast to uppercase and lowercase, respectively.

UPPER(inputString)

To show the effects of the Upper & Lower functions, we can show film titles in their original case and altered through each function:

Working with Functions in Navicat

One of the features of Navicat's SQL Editor is auto-completion. As soon as you begin to type a word, a list of suggestions comes up that includes all database objects, including schema, table/view, column, procedure, and, of course, function names:

Once a function (or stored procedure) is selected, input parameters are highlighted for entry. If there are more than one, each parameter is tabbable for quick access:

Conclusion

In this first instalment of this series on Important SQL Server Functions, we looked at several useful string utility functions, including Len, Trim, Concat, Upper, and Lower. Next time, we'll be moving on to numeric functions.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes!

Being a transactional programming language, SQL is designed to execute its work in an all-or-nothing capacity. Meanwhile, procedural programming languages such as C# and Java are often iterative in nature. As such, they tend to loop over the same code until the stack is diminished and fully processed. Cursors are a notable exception to SQL's transactional approach. Like WHILE loops, cursors allow programmers to process each row of a SELECT result set individually by iterating over them. While many SQL purists shun cursors out of disdain or fear, they have their place in database development and are well worth learning. To that end, today's blog will describe when and how to use cursors within your stored procedures.

Cursors Defined

As mentioned above, a database cursor is a special control structure that enables traversal over the records in a database in order to process individual rows of a query result set for sequential processing. In Stored Procedures, a cursor makes it possible to perform complex logic on a row by row basis.

Cursors have three important properties:

• Asensitive: The server may or may not make a copy of its result table.
• Read-only: The data may not be updated.
• Nonscrollable: Can be traversed only in one direction and cannot skip rows.

How to Use a Cursor

Using a cursor within a stored procedure is a four step process:

• Declare a cursor.
• Open a cursor.
• Fetch the data into variables.
• Close the cursor when done.

Declare a Cursor

The following statement declares a cursor and associates it with a SELECT statement that retrieves the rows to be traversed by the cursor:

DECLARE cursor_name CURSOR FOR select_statement

Open a Cursor

The following statement opens a previously declared cursor.

OPEN cursor_name

Fetch the Data into Variables

This statement fetches the next row for the SELECT statement associated with the specified cursor (which must be open) and advances the cursor pointer. If a row exists, the fetched columns are stored in the named variable(s). The number of columns retrieved by the SELECT statement must match the number of output variables specified in the FETCH statement.

FETCH [[NEXT] FROM] cursor_name INTO var_name [, var_name] ...

Close the Cursor When Done

This statement closes the cursor. An error occurs if the cursor is not open.

CLOSE cursor_name

A Practical Example

Here's the definition of a stored procedure (shown in Navicat for MySQL) that employs a cursor to generate a list of emails for all staff members in the Sakila sample database:

Within the getEmail LOOP, the cursor iterates over the email list, and concatenates all emails separated by a semicolon (;). The finished variable informs the cursor to terminate the loop when there was no email fetched. Here is the value of the emailList after execution of the stored procedure:

Conclusion

In today's blog we learned when and how to use cursors within your stored procedures.

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

There are many times where one needs to copy data from an existing table to a new one, for example, to back up data or to replicate data in one environment in another, as one might do for testing purposes. In SQL, one would typically use CREATE TABLE and SELECT statements as follows:

CREATE TABLE new_table; SELECT SELECT col, col2, col3 INTO new_table FROM    existing_table;

In the first statement, the database creates a new table with the name indicated in the CREATE TABLE statement. The structure of the new table is defined by the result set of the SELECT statement. Then, the database populates data with the results of the SELECT statement to the new table.

While the above procedure works perfectly well, there's an easier way to copy a table into a new one using a variation of the CREATE TABLE statement! We'll learn how to use it here today.

Introducing the CREATE TABLE AS SELECT Statement

The CREATE TABLE statement provides a way to create one table from another by adding a SELECT statement at the end of the CREATE TABLE statement. The full syntax for the statement is:

CREATE TABLE new_tbl [AS] SELECT * FROM orig_tbl;

It's a way to do, in one line of code, the exact same thing as we did using two separate statements above.

Copying Partial Data

Since the SELECT statement support all clauses that you'd usually employ in your SQL statements, including the WHERE and ORDER BY clauses, we can limit what we copy over by supplying a condition in our statement. Here's the syntax for that:

CREATE TABLE new_table SELECT col1, col2, col3 FROM    existing_tableWHERE    conditions;

Some Examples

Here are a couple of examples using Navicat Premium as our database client:

In its most basic form, the CREATE TABLE AS SELECT statement can copy a table "as-is" using a SELECT All (*). Here's an example:

Here's a more complex example that only copies three columns from an orders table and limits rows to those with a recent requiredDate:

We can see that the new table only has three columns a selected:

Conclusion

There's no question that the CREATE TABLE AS SELECT statement offers a quick and easy way to copy data from a table into a new one. Having said that, it does have its limitations. For starters, not all relational databases support it. I know that MySQL and SQL Server do, but other databases may or may not.

It is also worth noting that the CREATE TABLE AS SELECT statement just copies the table and its data. It does not copy other database objects such as indexes, primary key constraint, foreign key constraints, triggers, etc., associated with the table. To copy not only the data but also all database objects associated with a table, we should use two separate statements as follows:

CREATE TABLE orders_copy LIKE orders;INSERT orders_copySELECT * FROM orders;

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

In the Understanding Database Transactions blog, we leaned how transactions are a fantastic way to guard against data loss and inconsistencies by guaranteeing that all operations performed with a transaction succeed or fail together. In today's follow-up, we'll learn how to employ a transaction within a stored procedure in order to ensure that all tables involved remain in a consistent state.

About the sp_delete_from_table Stored Procedure

If you've read any of my previous blog articles, you probably know that I often illustrate new concepts using the Sakila Sample Database. And why not? It was developed specifically as a learning database for MySQL. If you aren't already aware, the Sakila Sample Database contains data pertaining to a fictitious movie rental store chain. Besides tables and views, you'll also find user functions, triggers, queries, and stored procedures that illustrate most commonly used database objects and tasks.

One of the stored procedures that is especially relevant to this blog is sp_delete_from_table. It accepts three input parameters as follows:

• @table: the name of the table from which to delete rows.
• @whereclause: the criteria for identifying which rows to delete.
• @delcnt: how many rows we expect to be deleted.

The procedure returns the @actcnt (bigint) output parameter that contains the number of rows that were actually deleted.

Here's the full definition as shown in Navicat Premium:

Important Transaction Statements

Relational databases provide us with several important statements to control transactions:

• To start a transaction, use the BEGIN TRANSACTION statement. Both START or BEGIN WORK are aliases of the BEGIN TRANSACTION. You'll find it on line 17 of the sp_delete_from_table procedure.
• To commit the current transaction and make its changes permanent, use the COMMIT statement. That happens on line 32 of the procedure.
• To roll back the current transaction and cancel its changes, use the ROLLBACK statement. There are a couple of situations where that comes up in the code:
  1. If the statement would have deleted all rows in the table, a message is displayed and the transaction is rolled back on line 26.
  2. Should the number of rows deleted not match the number that you expected, again, a message is displayed and the transaction is rolled back. That happens on line 38.
• To disable or enable the auto-commit mode for the current transaction, use the SET autocommit statement. By default, some databases, such as MySQL, run with autocommit mode enabled by default. This means that, when not otherwise inside a transaction, each statement is atomic, as if it were surrounded by START TRANSACTION and COMMIT. You cannot use ROLLBACK to undo the effect. However, if an error occurs during statement execution, the statement is rolled back. Since most of the work takes place within a transaction in the sp_delete_from_table procedure, the SET autocommit statement is not needed.

Testing a Transaction Rollback

Since we know that the sp_delete_from_table procedure will abort if the expected count does not match the actual number of rows deleted, we can test for rollbacks by either making sure that our @whereclause criteria would delete every row in the table or by simply providing a @delcnt value that we know won't match. Let's try the latter.

In Navicat, we can run a stored procedure from the editor via the Execute button. Clicking it causes a dialog to come up which accepts input parameters (output params may be ignored):

After the procedure terminates, we can see output messages in the Message tab. We can see that it has been rolled back as expected:

Conclusion

In today's blog, we learned how to employ a transaction within a stored procedure in order to ensure that all tables involved remain in a consistent state, no matter the outcome.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Atomicity Consistency Isolation Durability, or "ACID", was coined by Andreas Reuter in 1983. It's a concept in database management systems (DBMS) that identifies a set of standard properties used to guarantee the reliability of a database. ACID properties ensure that all database transactions remain accurate and consistent, and support the recovery from failures that might occur during processing operations. As such, it is implemented by nearly all Relational Databases.

As it turns out, DBMS that offer support for transactions enforce the four ACID properties automatically. In today's blog, we'll learn how transactions do that. In up-coming articles, we'll look at how to use transactions in our stored procedures to guard against data inconsistencies.

Transactions Explained

Before we can employ transactions within our own stored procedures, it might help to understand what a transaction is. Simply put, a transaction is a set of operations performed so all operations are guaranteed to succeed or fail as one unit.

As an example, consider the process of transferring money from a checking account to a savings account. This action is actually made up of two parts:

• Withdraw funds from the checking account.
• Deposit it into the savings account.

Now, imagine what might happen if the power went out after the first step. I think that we can agree that there would be a problem if funds were deducted from the checking account but not added to the savings account! Just as you would not want this to happen with your financial transactions, updating one database table without updating tables that refer to it is just as undesirable. By employing a transaction, both the operations are guaranteed to succeed or fail together. That way, all entities involved remain in a consistent state.

ACID and Transactions

Transactions play a key role in enforcing all four ACID properties: Atomicity, Consistency, Isolation, and Durability. Let's see how they do that.

Atomicity

A database operation is considered atomic if it cannot be further broken down into separate operations. A transaction is also atomic because all of the operations that occur within a transaction either succeed or fail together. Should any single operation fail during a transaction, then everything is considered to have failed and must be undone (i.e. rolled back).

Consistency

One of the main perks of using a transaction is that it should leave the database in a consistent state, whether or not it completes successfully. This ensures that data modified by the transaction complies with all the constraints placed on the columns so that data integrity is maintained.

Isolation

Every transaction is isolated from other transactions. Therefore, a transaction shouldn't affect other transactions running at the same time. Stated another way, data modifications made by one transaction should be isolated from the data modifications made by other transactions. So, while a transaction can see data in the state it was in before another concurrent transaction modified it, as well as after the second transaction has completed, it cannot see any intermediate states.

Durability

Transactions help with Durability in a few ways: data modifications that take place within a successful transaction may be safely considered to be stored in the database regardless of whatever else may occur. As each transaction is completed, a row is entered in the database transaction log. Thus, in the event of a system failure that requires the database to be restored from a backup you can use this transaction log to get the database back to the state it was in after a successful transaction.

Conclusion

Transactions are a fantastic way to enforce the four ACID properties within your database(s). In today's blog, we learned how transactions do that. In the next article, we'll look at how to use transactions in our stored procedures to guard against data inconsistencies.

Many organizations make some effort to protect their data by implementing input validation within their applications. As valuable as that is, it should be noted that many cyber attacks are aimed squarely at the database servers themselves, where application security does not come into play at all! As a Database administrator (DBA) or Database Developer, you have tremendous power to reduce the risk of cyber attacks, and/or damage that may occur as a result, including from the most common form of cyber attack: SQL Injection. In today's blog, we'll explore a few practices that can greatly reduce exposure to SQL Injection attacks.

Place All Database Logic within Stored Procedures

The more easily a malicious entity can pass in unfiltered SQL to the database server(s), the more susceptible your data will be to loss or theft. By placing all of your queries and data manipulation statements (DML) inside of stored procedures, you can make it much more difficult for hackers to issue DML statements.

The following code example uses a CallableStatement, Java's implementation of the stored procedure interface, to execute the same database query:

String custname = request.getParameter("customerName");try {  CallableStatement cs = connection.prepareCall("{call sp_getCustomerAccount(?)}");  cs.setString(1, custname);  ResultSet results = cs.executeQuery();  // ...result set handling} catch (SQLException se) {  // ...logging and error handling}

Whitelist Input Validation

User supplied values are not the place to bind database entities such table, column names, or even the sort order indicator (ASC or DESC). Those values should come from your own SQL code, and not from user parameters. To target specific table and column names, parameter values should be mapped to the legal - i.e. expected - table and/or column names to prevent unvalidated user input ending up in the query.

Here is an example of table name validation:

String tableName;switch(PARAM):  case "Value1": tableName = "clientTable";                 break;  case "Value2": tableName = "employeeTable";                 break;  ...  default      : throw new InputValidationException("unexpected value provided"                                                  + " for table name");

For something simple like a sort order, one solution is to accept the user supplied input as a boolean, which is then utilized to select the safe value to append to the query. In fact, this is a common practice in dynamic query construction.

public String myMethod(boolean sortOrder) {  String SQLquery = "some SQL ... order by Salary " + (sortOrder ? "ASC" : "DESC");  ...

Escape/Sanitize All User-Supplied Input

Only when none of the above are feasible should user input escaping be employed. The reason that this defense is considered to be frail compared to other defenses is because there is no guarantee that it will prevent all SQL Injection across every possible situation.

It's vitally important that you match the user input escaping to your particular database type as every DBMS supports one or more character escaping schemes specific to certain kinds of queries. By escaping all user supplied input using the proper escaping scheme for the specific database you are using, the DBMS will not confuse that input with SQL code written by the developer, thus avoiding virtually any possible SQL injection vulnerabilities.

The OWASP Enterprise Security API (ESAPI) is a free, open source, web application security control library that makes it easier for programmers to harden their applications against cyber attacks. The ESAPI libraries are designed to make it easy for programmers to retrofit security into existing applications as well.

To use an ESAPI database codec is pretty simple. An Oracle example looks something like:

ESAPI.encoder().encodeForSQL( new OracleCodec(), queryparam );

Conclusion

As a Database administrator (DBA) or Database Developer, you have tremendous power to reduce the risk of cyber attacks, and/or damage that may occur as a result thereof, including from the most common form of cyber attack: SQL Injection. By following the practices outlines here today, you can greatly reduce exposure to SQL Injection attacks.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. You can hire Rob by emailing him at rgconsulting(AT)robgravelle(DOT)com. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Not so long ago, the word "atom" referred to a thing that could not be split any further. Despite having discovered that atoms themselves are made up of even smaller particles, the term continues to retain its original meaning. With respect to relational databases, Atomicity means that operations (DMLs/DDLs, etc.) executed by the database will be atomic. The unit of atomicity usually provided by relational databases is a transaction. Why is this important? A guarantee of atomicity prevents updates to the database occurring only partially, which can cause greater problems than rejecting the whole series of operations outright. In today's blog, we'll learn what Atomicity is and how to enforce it within your database instances.

You Can't Spell ACID without Atomicity

You've probably heard the term "ACID" thrown about with respect to relational databases. It stands for "Atomicity Consistency Isolation Durability". It's a concept in database management systems (DBMS) that identifies a set of standard properties used to guarantee the reliability of a database. ACID properties ensure that all database transactions remain accurate and consistent, and support the recovery from failures that might occur during processing operations. As such, it is implemented by nearly all Relational Databases.

Here's where Atomicity comes in:

Say that you were performing a database UPDATE that will take 10 seconds to process all the rows in the table. As the updates proceed, the power suddenly goes out! Once power is restored, you go to read the data, and discover that some of the rows were updated according to your SQL statement, and the rest of the rows were not. You've now got yourself a bit of a mess!

Luckily, this can't happen with today's modern databases, right? Wrong.

Know Your Storage Engine

In many cases, the type of database you use is not as important as the storage engine that's being employed. The storage engine is the underlying software component that DBMS use to create, read, update and delete (CRUD) data. Most databases support several different types of storage engines. For example, MySQL currently offers the following out of the box:

• InnoDB
• MyISAM
• Memory
• CSV
• Archive
• Blackhole
• NDB
• Merge
• Federated
• Example

You are not restricted to using the same storage engine for an entire server or schema. You can specify the storage engine at the table level.

There exists a wide variety of storage engines because certain storage engines are effective in certain operations and environments yet very ineffective in others. This is important to take into consideration and pick which storage engines will work best for your usage patterns.

Back to our example, if you were using the MyISAM engine, you could be in trouble, because MyISAM does not enforce atomicity. Hence, a single change can be partially applied, whereby some rows in the intended set are affected, but the rest of the set are not. On the other hand, the InnoDB storage engine DOES ensure that any UPDATE will be applied to the complete set of rows you intended, or else it will apply to none of the rows if an error occurs or if the transaction is interrupted for some reason.

Selecting a Storage Engine in Navicat

Navicat makes selecting a storage engine for each table in your database easy via a drop-down on the Options tab within the Table Designer. Here's what you'll find for MySQL in Navicat Premium:

Conclusion

In today's blog, we learned what database Atomicity is and how to enforce it within your database instances.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. You can hire Rob by emailing him at rgconsulting(AT)robgravelle(DOT)com. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

Both GROUP BY and ORDER BY are clauses (or statements) that serve similar functions; that is to sort query results. However, each of these serve very different purposes; so different in fact, that they can be employed separately or together. And that is where things can get a little dicey if you are unsure of what you're doing. In today's blog, we'll learn what each clause does and how to use them together for the ultimate control over your query output. To do that we'll be using Navicat Premium against the Sakila Sample Database.

GROUP BY and ORDER BY Explained

The purpose of the ORDER BY clause is to sort the query result by one or more columns. Meanwhile, the GROUP BY clause is used to arrange data into groups with the help of aggregate functions such as COUNT(), AVG, MIN() and MAX(). The way that it works is, if a particular column has the same values in different rows then it will amalgamate these rows into a group.

Let's look at an example of each.

Here's a query that displays the first and last names of all actors from the table actor, sorted by last name, followed by first name:

Now, here's another query that groups actors by the number of films that they have appeared in:

Using Group By and Order By Together

Notice that, in the preceding query, records are ordered by the actor_id field, which is what results are grouped on. If we wanted to order results using different - i.e. non-grouped - fields, we would have to add an ORDER BY clause. Here's the same query, but ordered by the number of films which each actor has appeared in, from most to least:

Notice that, once you include the Order By clause, the default group ordering is lost. If you'd like to keep it, you can add grouped columns to the Order By field list:

Points to Keep in Mind

When combining the Group By and Order By clauses, it is important to bear in mind that, in terms of placement within a SELECT statement:

• The GROUP BY clause is placed after the WHERE clause.
• The GROUP BY clause is placed before the ORDER BY clause.

GROUP BY goes before the ORDER BY statement because the latter operates on the final result of the query.

Bonus Section: the Having Clause

You can filter the grouped data further by using the HAVING clause. The HAVING clause is similar to the WHERE clause, but operates on groups of rows rather than on individual rows. To illustrate how the HAVING clause works, we can use it to limit results to those actors who've appeared in more than ten films:

Navicat's SQL Editor greatly facilitates query writing thanks to features like syntax highlighting, reusable code snippets for control flow/DDL/syntax statements, as well as auto-complete. It can suggest everything from schema, tables, and columns to stored procedure and functions. Here is the HAVING keyword:

The Having Clause should be placed after the Group By clause, but before the Order By clause.

Conclusion

In today's blog, we learned what each clause does and how to use them together for the ultimate control over your query output using Navicat Premium.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. You can hire Rob by emailing him at rgconsulting(AT)robgravelle(DOT)com. In his spare time, Rob has become an accomplished music artist with several CDs and digital releases to his credit.

In previous blog, we tabulated the average daily counts for a given column in SQL Server using Navicat for SQL Server. In today's follow-up, we're going to raise the difficulty factor slightly by calculating the daily average date/time interval that is based on start and end date columns. For demonstration purposes, I'll be working with MySQL using Navicat Premium.

Calculating Movie Rental Durations in Days

In the Sakila Sample Database's rental table there are two date fields that represent a time interval: they are the rental and return dates. These, of course, store the date and time that a film was rented, and when it was returned.

With that in mind, suppose that we needed to write a query that shows the average length of movie rentals for each day. The first step would be to calculate the length of all movie rentals. Here's what that query would look like in Navicat:

To convert the rental_date from a datetime to a pure date we can use the DATE() function. It accepts any valid date or datetime expression.

The number of days is calculated using the MySQL DATEDIFF() function. It returns the number of days between two dates or datetimes. Navicat can help us use the DATEDIFF() function by providing auto-complete. When you start to type a word, a popup list appears with suggestions for everything from schemas, tables/views, columns, as well as stored procedures and functions. Here is the DATEDIFF() function in the suggestion list:

After you select a function, it gets inserted into your code at the cursor position with tabbable, color-coded, input parameters for quick entry:

Grouping Results by Day

The next step is to group results by day. This is done via the GROUP BY clause. It allows us to apply aggregate functions such as COUNT() and AVG() to the number of days that rentals were out for each rental_date. Here is the updated query with the GROUP BY clause:

You'll notice that I rounded the avg_days_rented to one decimal place. Otherwise, we'd get 4 points of precision, which may be a bit much for our purposes!

Conclusion

Thanks to MySQL's many date/time functions, calculating the daily average date/time interval based on start and end date columns is made a lot easier than it otherwise might be. Moreover, Navicat's feature-rich SQL Editor further simplifies query writing by providing auto-complete for just about any database entity, including schemas, tables, columns, as well as functions and stored procedures.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. In his spare time, Rob has become an accomplished guitar player and has released several CDs and digital singles. You can hire Rob by emailing him at rgconsulting@robgravelle.com .

Normally, querying a normalized database necessitates joining tables together on one or more common fields. Otherwise, you risk generating a cartesian product. That is a result set whose number of rows equals those in the first table multiplied by the number of rows in the second table. So, if the input contains 1000 persons and 1000 phone numbers, the result consists of 1,000,000 pairs! Not good. Having said that, if you wanted to aggregate data from similar tables that are not directly related, you can do that using the UNION operator. In today's blog, we'll learn some of the finer points on using UNION, along with its close cousin, UNION ALL.

UNION versus UNION ALL

Some people think that Union and Union All are interchangeable. They are not. They differ in that Union removes duplicate rows or records, whereas Union All does not; instead, it just selects all the rows from the tables which meet the conditions of your queries' WHERE criteria and combines them into the results.

Combining Results with UNION

Here's a query that combines the results of two SELECT queries using the Sakila Sample Database:

Let's say that you wanted to find the names of all the actors and customers whose first name is the same as the first name of the actor with ID 8, but without returning the actor 8's details. Although there is more than one way to achieve this, one solution is to employ UNION ALL (UNION would work as well, since there are no duplicates between each result set). Here is the query that does the job, along with the results, in the Navicat Premium database development and admin client:

In the above query, the top SELECT fetches customers whose first name matches that of the actor with the actor_id of 8, while the bottom query fetches actors with the same first name as their fellow actor with ID 8.

If you are a Navicat user, you are already aware that it's editor is one of the best. It provides syntax highlighting, reusable code snippets, as well as auto-complete. When you start to type a word, a popup list appears with suggestions for everything from schemas, tables/views, columns, as well as stored procedures and functions. Here is the UNION operator:

A few final words about UNION and UNION ALL. It is crucial that all queries return the same number of columns or you'll get an error similar to the following:

That being said, the column types do not have to match across all SELECT statements.

And finally, keep in mind when employing UNION or UNION ALL that the column names for the result set are determined by the first SELECT.

Conclusion

The UNION and UNION ALL operators are the perfect tool for aggregating data from similar tables that are not directly related.

If you'd like to give Navicat Premium a try, you can test drive it for 14 days completely free of charge for evaluation purposes!

I recently wrote a node.js script to iterate over millions of files per day and insert their contents into a MySQL database. Rather than process one record at a time, the script stored file contents in memory and then ran an INSERT statement every 1000 files. To do that, I used the bulk insert form of the INSERT statement. Depending on your particular requirements, you may opt to go with a different solution. In today's blog, we'll go over a few alternatives.

INSERT Statement Variation for Bulk Inserts

The INSERT statement supports several syntax variations, one of which is for inserting multiple rows at the same time. To do that, we simply need to enclose each value list in parentheses and separate them using a comma:

INSERT INTO table_name (column_list) VALUES     (value_list_1),     (value_list_2),     ...     (value_list_n); 

Simple enough. Here's a sample statement shown in Navicat for MySQL:

While the above statement is formatted for readability, you don't have to concern yourself with that when generating the SQL dynamically. As long as the syntax is semantically correct, it will work just fine. Finally, note that 1000 is the maximum number of rows that can be inserted at one time using an INSERT statement.

LOAD DATA INFILE

Another option, for those of you who aren't thrilled about writing scripting code, is to use something like LOAD DATA INFILE. That's a MySQL-specific command, but most other database systems (DBMS) support something similar. It can import a variety of delimited file formats, including commas (CSV), Tabs (TDV), and others.

Here's the statement for importing data from the "c:\tmp\discounts.csv" file into the discounts table:

LOAD DATA INFILE 'c:/tmp/discounts.csv'  INTO TABLE discounts  FIELDS TERMINATED BY ','  ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS; 

In the above statement, the IGNORE 1 ROWS option is employed to ignore headers.

I would have liked to have used this method for importing data, but the files that we were importing from utilized a highly specialized and complex format that required a lot of front-end logic.

Using an Import Utility

Still another approach would be to use an import utility such as Navicat's Import Wizard. It supports just about any format that you can imagine, including CSV, Excel, HTML, XML, JSON, and many other formats:

There is a screen for choosing the record delimiter, field delimiter, and text qualifier:

Navicat shows you the progress in real time:

Once your done, you can save all your settings for later use, which is not only useful for running the same on a regular basis, but it also allows you to automate it, so that imports happen without any additional intervention required on your part!

Conclusion

In today's blog, we covered a few alternatives for performing bulk inserts into MySQL and other DBMS.

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

Joins and subqueries are both used to combine data from different tables into a single result set. As such, they share many similarities as well as differences. One key difference is performance. If execution speed is paramount in your business, then you should favor one over the other. Which one? Read on to find out!

The Verdict

I won't leave you in suspense, between Joins and Subqueries, joins tend to execute faster. In fact, query retrieval time using joins will almost always outperform one that employs a subquery. The reason is that joins mitigate the processing burden on the database by replacing multiple queries with one join query. This in turn makes better use of the database's ability to search through, filter, and sort records. Having said that, as you add more joins to a query, the database server has to do more work, which translates to slower data retrieval times.

While joins are a necessary part of data retrieval from a normalized database, it is important that joins be written correctly, as improper joins can result in serious performance degradation and inaccurate query results. There are also some cases where a subquery can replace complex joins and unions with only minimal performance degradation, if any.

Examples of Subqueries

Sometimes you can't easily get at the data you want using a subquery. Here are a couple of examples using the Sakila Sample Database for MySQL and the Navicat Premium database development and admin client.

Example #1: Using an Aggregate Function as Part of a Join Clause

Most of the time, tables are joined on a common field. In fact, the common fields often share the same name in order to show that they refer to the same piece of information. But that is not always the case. In the following query, the customer table is joined to the latest (MAX) create_date so that query results pertain to the customer with the most recent sign up date:

In the above SELECT statement, a subquery is employed because you cannot use aggregate functions as part of a WHERE clause. This ingenious workaround circumnavigates that limitation!

Example #2: Double Aggregation

In this example, a subquery is employed to fetch an intermediary result set so that we can apply the AVG() function to the COUNT of movies rented. This is what I call a double aggregation because we are applying an aggregation (AVG) to the result of another (COUNT).

This particular query is quite fast - taking only 0.044 seconds - because the inner query returns a single (scalar) value. Usually, the slowest queries are those that require full table scans, which is not the case here.

Conclusion

While both joins and subqueries have their place in SQL statements, I would personally recommend that you always try to write your queries using joins exclusively. Only when you cannot fetch the data you're interested in without a subquery, should you introduce one.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Database optimization is a rather large and sprawling topic that encompasses a multitude of strategies for reducing database system response times. These are often tailored to the specific usage patterns of a database instance or cluster. For instance, in some cases, lightning fast queries might be a goal, whereas for some organizations, faster write times may be what's desired most.

Improving query response times may include activities such as:

• careful construction of queries
• use of indexes
• using analysis tools such as EXPLAIN

In today's blog, we'll learn more about this vital topic in database administration.

Optimization Activities Described

As mentioned in the introduction, database optimization involves a number of strategies whose aim is to reduce database system response times. To that end, administrators (DBAs), developers and analysts may seek to decrease write times by working to improve the servers' data access methods and retrieval times through design techniques, statistical analysis and monitoring of system traffic. In this role, DBAs/analysts need to possess a strong knowledge of the structure of the data, the applications installed on the server and the impact varied tasks have on the database's overall performance.

Typically, database tuning and optimization can require a high degree of expertise, an understanding of execution plans, as well as the ability to write high-performing SQL. It also tends to be a highly time-consuming endeavor, because there can be a huge number of SQL statements to fine tune. Once you've determined which statements need tuning, you then need to refine your tuning approach to suit each and every query, as there is no one-size-fits-all solution.

Tools of the Trade

Query optimization is usually the best place to focus your efforts for two reasons: it's the easiest part of the optimization equation and tends to result in the most bang for your buck in terms of reward versus effort. Part of the reason that query optimization is the lowest hanging fruit is that there are a number of tools that you can use to aid you in your quest for improved database performance. Here are a few:

Using EXPLAIN

If you've got one query that runs consistently slow, then it probably needs to be optimized further. A good way to see what it needs is to use the EXPLAIN command. It returns a formatted description of the query optimizer's execution plan for the specified statement. You can use this information to analyze and troubleshoot the query.

By default, EXPLAIN output represents the query plan as a hierarchy whereby each level represents a single database operation that the optimizer defines to execute the query. In Navicat database clients, there's a button in the SQL Editor that runs EXPLAIN. Results are displayed in an easy-to-read grid format:

Analyzing Query Performance using a Monitoring Tool

You can also analyze your query performance using a tool like Navicat Monitor. It has a Query Analyzer that shows information of all executing queries. Moreover, it can help identify slow queries and detect deadlocks, i.e. when two or more queries permanently block each other.

Conclusion

Finally, if your DBMS supports query profiling, you can use it to measure query execution time. While perhaps not quite as powerful as the tools we saw here today, it might be worth a try.

While you've almost certainly heard of relational and NoSQL databases, there is a better than even chance that you're completely unfamiliar with flat file databases. Flat file databases are indeed a real thing, but they don't get much love these days. As we'll learn in today's blog, there is a better way to work with flat file databases than in years gone by. In fact, if you use any of Navicat's database development and admin clients, you're in the ideal position to do so!

History and Limitations

Flat file databases have been around ever since the very first computers. They are a type of database that stores data in a plain text file, whereby each line of the file holds one record, and fields are separated by delimiters - typically commas or tabs. As such, flat file databases share more commonality with a spreadsheet than a relational database. Due to their simple structure, the "tables" represented within a flat file database support limited functionality, such as record and column sorting.

Flat file databases flourished as a back-end to applications. Their simple structure takes up less space than structured database files and work well for configuration data. If you have some programming savvy, you can find ODBC drivers for most languages for interfacing with flat file databases. Unfortunately, most relational database clients cannot connect directly to a flat file database. However, relational databases provide commands to import flat file databases and use them in a larger relational database.

Importing a Flat file

If the structure of a flat file database sounds familiar to you, it's because it is very similar to a CSV (Comma Separated Values), TSV (Tab Separated Values), or any DSV (Delimiter Separated Values) file.

Every relational database provides its own command(s) for importing data from a flat file. For example, MySQL provides the LOAD DATA INFILE statement. You have to create the database and table(s) first, but you only need to do that once for each data set. Once you've done that, LOAD DATA INFILE is very fast! Here's an example statement for importing a CSV (Comma Separated Values) file:

LOAD DATA INFILE 'c:/path/to/file.csv'  INTO TABLE discounts  FIELDS TERMINATED BY ','  ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS; 

Using Navicat Import

Navicat's powerful Import utility is a wizard-driven process that helps you import a wide variety of formats from DSV, JSON, XML, and more.

It lets you choose your record delimiter, field delimiter, and text qualifier:

You get a full progress report as the import proceeds, including the number of tables and rows processed, along with errors encountered and time taken:

Conclusion

In today's blog, we learned about flat file databases and how to import them into your relational database using native database commands as well as Navicat.

For a more in-depth look at Navicat's Import utility, I wrote the Importing XML, CSV, Text, and MS Excel Files into MySQL article a couple of years ago showing how to import data in a variety of formats using Navicat for MySQL.

Rob Gravelle resides in Ottawa, Canada, and has been an IT Guru for over 20 years. In that time, Rob has built systems for intelligence-related organizations such as Canada Border Services and various commercial organizations. In his spare time, Rob has become an accomplished guitar player and has released several CDs. You can hire Rob by emailing him at rgconsulting@robgravell.com .

Sometimes a database administrator (DBA) needs to furnish a report on the number of missing values in a table or tables. Whether the goal is to show counts or row content with missing values, there are a couple of ways to go about it, depending on how flexible you want to be about it. The first would be to construct a query against the table(s) in question, using information that you have about field names, data types, and constraints. The second, more elaborate, approach would be to write a stored procedure that fetches column info from the INFORMATION_SCHEMA.COLUMNS table. In today's blog, we'll take a look at the non-generic approach, while next week's blog will address the stored procedure solution.

Showing Nullable Columns

Since not every field in a table can accept null values, it is helpful to inspect the table design and see which fields may contain nulls. In Navicat database development and admin clients, the Table Design identifies all mandatory columns using a checkbox under the Not null header. Hence, all columns whose checkbox is not checked may contain nulls. Those would be the fields that our query will focus:

One way to find fields with nulls is to build the query using the Query Builder tool. It lets us select many conditions from a menu, including "is null", "is not null", "is empty", "is not empty", etc...

Once built, we can insert the SQL directly into the Editor:

Here are all rows of a customers table which contain at least one column with a null value:

Obtaining Stats on Filled and Empty Fields

In cases where we only want statistics on filled versus empty fields, we can use the count() function to tally fields which are either filled or null. In the following query, percentages are expressed as a the number of rows which contain a null value for that particular field:

Likewise, we can count and shows null columns for a specific row, identified here by the customerNumber:

In the above query, a CASE statement is employed to only include null values in the counts. This time, the percentage is showing how many of the total fourteen table columns contain nulls, rounded to 2 decimal places.

Conclusion

In today's blog, we learned how to query for missing values in a table or tables. Next week's blog will introduce a more generic approach using a stored procedure. In the meantime, here's a query for SQL Server to whet your appetite. It fetches column metadata from the INFORMATION_SCHEMA.COLUMNS table in order to generate queries for every table:

The above query will return a list of select queries. We can then copy & paste these to the Navicat Query Editor with a 'union' between the selects to find missing values in every table in a database!

Calculating average daily counts seems like something that would be done fairly often, and yet, I have never done it. I asked my wife, who is also a programmer of database-backed applications, and she never had the occasion to do so either! So, it is with great enthusiasm that I take on this challenge today.

To qualify what is meant by an "average daily count", for the purposes of this blog, it describes a monthly count of patients at a doctor's office, widgets manufactured, or products sold within a month. The daily average is then calculated by dividing the monthly count by the number of days in a month so that we know how much the daily count contributed towards the monthly total. For example, if a car dealership sold 10 Honda Civics in the month of May, then those 10 sales represent an average of 0.32 of one vehicle per day. Meanwhile, if the dealership were to sell a whopping 50 Honda Civics in one month, then the daily average would soar to 1.61 Honda Civics per day.

The Query

Our SELECT statement will query the Sakila Sample Database to tabulate movie rentals for each month in the following format:

ID| MONTH | MONTHLY_COUNT | AVG_DAILY_COUNT -------------------------------------------  1| Jan   | 152           | 10.3  2| Jan   | 15000         | 1611  3| Jan   | 14255         | 2177  1| Feb   | 4300          | 113  2| Feb   | 9700          | 782  3| Feb   | 1900          | 97 etc... 

The AVG_DAILY_COUNT column above increases the complexity of the query substantially because we need the obtain the monthly counts first. Therefore, the query consists of both inner and outer SELECT statements. Here is the inner query and results, sorted by year, month, and inventory_id:

The Outer Query

From that data, we can tabulate the average daily counts as follows:

I included the number of days in each month for reference, since it plays a key role in calculating the daily averages. It's also edifying to see how it's obtained, since the number of days in each month is required to calculate the average daily count. Here is that code isolated from the rest of the query:

datediff(day,          datefromparts(rental_year, rental_month, 1),          dateadd(month, 1, datefromparts(rental_year, rental_month, 1))) days_in_month

The datediff() function returns the number of days between the first day of the month to the first day of the following month. The datefromparts() function creates a date from the rental_year and rental_month columns of the inner query.

We can see the same code in the calculation of the daily_avg:

round(    cast(cnt as FLOAT) / cast(datediff(day,                               datefromparts(rental_year, rental_month, 1),                               dateadd(month, 1, datefromparts(rental_year, rental_month, 1))) as FLOAT                         ), 4) daily_avg

Notice that both the Dividend (cnt) and divisor (days in month) have to be cast to a FLOAT. Otherwise, decimals are discarded in the calculation. We want to keep as much precision as possible until the end, where we round to four decimal places.

Conclusion

In today's blog, we calculated the average daily counts for a given column in SQL Server using Navicat for SQL Server. Interested in giving Navicat for SQL Server a try? You can download it for 14 days completely free of charge for evaluation purposes!

Choosing between commercial and open source database offerings is not an easy one as many popular commercial databases are made available to developers and/or students at a greatly reduced cost or even for free. In other cases, the parent companies offer similar open source versions of their enterprise level products.

Judging by user polls, when looking at a snapshot of all database types - whether paid or free - an interesting trend emerges. It seems that, overall, users of database systems (DBMS) have gone with an open source solution the vast majority of the time. And not just any open source solution, but MySQL specifically. MySQL Community Edition has held the number one spot on the list of Most Popular Database Platforms for years now. Here are the top 5 databases - both commercial and FREE - for 2019 and their percentage of market share:

• MySQL: 52%
• PostgreSQL: 36%
• MS SQL Server: 34%
• SQLite: 30%
• MongoDB: 26%

So why is MySQL so darned popular?

It's Open Source

As stated above, it's free, but MySQL has a lot more going for it than price. Another attractive feature is that MySQL is open source software. This allows it to be customized or modified according to users' needs. Moreover, many third-party tools and interfaces have been developed for MySQL due to the fact that there are no licensing fees required. One of which is of course Navicat for MySQL.

Since its inception in the mid 90s by a Swedish company, MySQL has been acquired by larger companies on a couple of occasions: Sun Microsystems acquired MySQL in 2008, then Oracle purchased it in 2010. Each time, there was much speculation that MySQL's free status would soon change. However, this has not been the case this far. At this time all indications are that Oracle will continue to offer MySQL Community Edition completely free of charge.

Versatility

MySQL is an extremely versatile database. It's portable enough for development use, and yet robust enough for the most mission critical applications. In fact, many of the world's largest and fastest-growing organizations, including Facebook, Google, Adobe, Alcatel Lucent and Zappos, rely on MySQL to save time and money powering their high-volume web sites, business-critical systems and packaged software.

The Emergence of Web Applications

Some people have proposed that the rise of web applications developed by small startups or individuals has contributed to the popularity of MySQL in numerous ways. Again, besides the price, the fact that MySQL uses basic SQL and is easily set up and configure have made it the choice to power many web apps.

Community Support

Another great feature of MySQL is the tremendous online community that support it. Freelance developers from all over the world are continually adding to the functionality and utility of the database platform. Paid support is also available from Oracle, at extra cost.

Third Party Software

MySQL's tremendous popularity has prompted many 3rd party vendors to create software for working with it. These include development and administration clients such as Navicat for MySQL and monitoring tools like Navicat Monitor. Although neither of these tools are free, they earn their keep by providing an all-inclusive front-end that features an intuitive and powerful graphical interface for database management, development, and maintenance. Designed to be powerful and yet easy-to-use, tools such as these save developers and administrators time and effort in everything that they do.

Conclusion

With so many great features, as well as the proliferation of high end user tools like Navicat for MySQL and Navicat Monitor, MySQL Community Edition continues to enjoy the top spot of the relational database heap.

Many database administrators (DBAs) spend at least some of their time trying to identify and remove duplicate records from database tables. Much of this time could be diverted to other pursuits if more attention was paid to preventing duplicates from being inserted in the first place. In principle, this is not difficult to do. However, in practice, it is all-too-possible to have duplicate rows and not even know it! Today's blog will present a few strategies for minimizing the occurrence of duplicate records in your database tables by preventing them from being inserted into a table.

Employ PRIMARY KEY and UNIQUE Indexes

To ensure that rows in a table are unique, one or more columns must be constrained to reject non-unique values. By satisfying this requirement, any row in the table may be quickly retrieved via its unique identifier. We can enforce column uniqueness by including a PRIMARY KEY or UNIQUE index on the applicable fields.

To illustrate, let's take a look at a table that contains product details such as the name, line, vendor, description, etc. Here it is in the Navicat Table Designer:

Navicat indicates fields that are part of a KEY using a key icon under the Key heading, along with a number that denotes its position within a composite key. A single key with a number 1 tells us that the productCode is the sole Primary Key (PK) column for the table. By definition, the PK must be unique and may not contain NULL values.

Meanwhile, if we then take a look at the Indexes tab, it shows that the productLine column is indexed as well:

In many instances, a single column is not sufficient to make a row unique, so we must add additional fields to the PK. Here's a payments table that requires both the customerNumber and checkNumber to make a unique PK because the same customer can make several payments:

The Downside of Auto-incrementing Primary Keys

Many database designers/developers (myself included!) love using numeric auto-incrementing PKs because:

• Ease of use. The database takes care of them for you!
• Collisions are impossible because each new row receives a unique integer.

Here is just such a table:

In Navicat, all you need to do to create an auto-incrementing PK is to choose a numeric data type (such as an integer) and check the Auto Increment box. That will cause all values for that column to be generated by the database.

And now for the bad news; auto-incrementing PKs do little to prevent duplicate rows - especially if you don't include any other table indexes. For example, imagine that the above table did not have any additional indexes. There would be nothing stopping someone from inserting a row with the exact same first_name and last_name as an existing row.

In fact, we can put that theory to the test right now! In Navicat, we can insert a new row directly into the Grid by clicking the plus (+) button:

As expected (feared), the duplicate name was accepted!

Conclusion

The moral of today's story is that, while one can prevent duplicate rows from being inserted, this does not necessarily mean that all data duplication can be prevented. At a minimum, designers/developers must take additional precautions by either employing rigorous normalization in database design or by performing specific validation at the application level.

The periodic reverting of database instances to a baseline dataset is a common practice in development and test environments. Case in point, the office where I work does so on a regular basis, whenever data diverges too much from the baseline. This is required because developers and automated tests expect the data to be of a certain quantity and quality. There is no right way to overwrite table contents, so you should choose an option based on your organization's particular goals and circumstances. In today's blog, I'll share what we do where I work as well as my standard process at Gravelle Web Development.

SQL Scripting at Work

An SQL script is one that contains a set of SQL commands saved as a file, typically with a .sql extension. An SQL script can contain both SQL statements or PL/SQL blocks. SQL scripts provide a simple means of grouping related SQL functionality for the purpose of reusing whenever needed. All popular relational databases can run an SQL script directly from the command line. For example, in MySQL, you can invoke the SQL script as follows:

shell> mysql --user="username" --database="databasename" --password="yourpassword" < "path to sql file"

Creating a Table Refresh Script

In my experience, the easiest way to create a script to reset table data is to use a dump utility. Keeping with MySQL, the installation process includes the mysqldump utility. It can create SQL statements to both truncate the table and repopulate rows with baseline data. Mysqldump has a number of options, but all that is really needed are the database and SQL file names:

shell> mysqldump db_name > backup-file.sql 

Which ever utility you use, it's crucial that the generated SQL includes a DROP TABLE statement before the table population. Mysqldump has a --add-drop-table option, but it's ON by default, so you don't need to include it under normal usage.

Navicat database development and admin clients include a Dump SQL File command. Like mysqldump, it also provides many options, including whether to dump both the structure and data or structure only:

Here is a sample generated file in the Navicat SQL Editor. As you can see, the DROP TABLE IF EXISTS command precedes the CREATE statement:

Truncating a Table

Whereas the above script recreates the table from scratch, you can also TRUNCATE a table and then re-insert the data from a backup table using the INSERT INTO command:

TRUNCATE TABLE dbo.T1; INSERT INTO D1.dbo.T1 SELECT * FROM D2.dbo.T1;

Here's an example in Navicat:

Note that, behind the scenes, the database is still dropping the table and re-creating it via the SQL CREATE TABLE statement. Besides being faster than the EMPTY TABLE command, TRUNCATE TABLE resets all auto-increment fields to start over at 1, which is usually preferable letting them run.

Conclusion

In today's blog, we explored a couple of ways to reset table data to a baseline for development and test environments. Navicat can help create .sql scripts as well as execute them with ease. Moreover, its Automation Tool can schedule scripts to run according to a variety of schedules so that you can set up your jobs and then let Navicat handle the rest.

In relational databases, including MySQL, SQL Server, Oracle, and others, the ORDINAL_POSITION refers to a column's location in terms of ordering within a table or query output. In today's blog, we'll learn how to use ordinal positioning to present columns in our preferred order, using Navicat Premium as our database client.

How the ORDINAL_POSITION Affects Query Output

When viewed in a grid format, columns are ordered from left to right. For instance, here are the columns of an order details table as they appear in Navicat's Grid View:

Meanwhile, in the Table Designer, you can see that the left to right column order above corresponds to the top -> down order there:

By default, columns occupy the same ordering in which they were created. Column ordering is important because it determines their order for every single table, view, and SELECT queries that are in the database. As we'll see a little later on, if we don't like the order of columns in a table, we aren't stuck with it.

Obtaining Ordinal Positioning of a Table

Since you can create views that can obscure the true ordinal positioning of a table, relational databases offer a means of finding out what it is. ORDINAL_POSITION is a column in the INFORMATION_SCHEMA.COLUMNS table. As such, you can find the ordinal position of columns in a table by querying it as follows:

Changing the Ordinal Positioning of a Table

So what do you do if you'd like columns to appear in a different order than they were created? AS I alluded to earlier, you don't have to keep a column's original ORDINAL_POSITION. You can change it using one of the following statements:

ALTER TABLE orderdetails MODIFY COLUMN orderLineNumber smallint(6) AFTER quantityOrdered; 

OR:

ALTER TABLE orderdetails CHANGE COLUMN orderLineNumber smallint(6) AFTER quantityOrdered; 

The above statements move the orderLineNumber column from last position to second-last.

In Navicat, the Table Designer has Move Up and Move Down buttons that make changing column ordering a snap:

After selecting the orderLineNumber column, every click of the Move Up button changes its ORDINAL_POSITION by one place:

After saving the table design, the orderLineNumber is now before the priceEach:

Referencing ORDINAL_POSITION in SELECT Queries

Ordinal Positioning is not just about default column ordering, it can also be referenced in SELECT queries as a short-cut for column names. To illustrate, here's a query that references a couple of columns in the ORDER BY clause:

Rather than spell out each column name in the ORDER BY clause, we can simply refer to the column's ORDINAL_POSITION within the table:

Shorter SQL, same results!

Conclusion

In today's blog, we learned how the ORDINAL_POSITION affects a column's location in terms of ordering within a table or query output, as well as how to use ordinal positioning to present columns in our preferred order.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Ransomware attacks are nothing new. In fact, the first known ransomware virus was created in 1989! So why bring them up now? While the frequency of ransomware attacks has fluctuated over the years, recent statistics show that ransomware attacks rose significantly in frequency in 2019 and won't be letting up any time soon. For that reason, you should know what ransomware attacks are, how they work, and how best to deter malicious entities from targeting your organization. And that is exactly what you'll learn here today!

Ransomware Attacks Explained

Ransomware attacks typically use much the same tricks to gain access to your server(s) as other hacking attacks. For instance, a phishing attack in the form of an e-mail with an infected attachment may be sent to employees of an organization. All it takes is for one user to open the attachment to allow the ransomware to execute on the network. Once inside your network, the ransomware immediately begins to encrypt files on the target system. Moreover, the malicious software may also attempt to encrypt files on any mapped drives or network-connected device that the infected machine has write permission to. In the case of databases, the database file may be encrypted. It's where the database schema and data are stored on the hard drive. What's worse, not only do your data and log files become unreadable, but you can even lose access to your backups. Once your backups are encrypted, you (and your entire organization) may be at the mercy of the attacker(s)!

Once the database file has been encrypted, someone will be contacted by the attacker(s), or the server will respond to requests asking for payment in the form of Bitcoin for a key to decrypt the files.

What Can Be Done to Prevent Ransomware Attacks

The old sports adage that "the best defence is a good offense" holds just as true for cyberattacks. One would hope that your organization maintains a strong firewall that keeps as many malicious emails out of employees' hands as possible. Users also play a vital role as guardians of the organization's infrastructure. As such, it is every employee's responsibility to only open attachments who's contents they know are safe. Even family and friends cannot be completely trusted, as their email accounts may be hacked.

So, what can Database Administrators (DBA) do to help bolster security?

Quite a lot, actually.

In many instances, ransomware attack have capitalized on an unpatched vulnerability. Upon reviewing the thousands of MySQL ransomware attacks a few years ago, investigators determined that the underlying security weakness in the vast majority of cases was either gaping holes in security protocols or even an almost complete lack thereof! The good news is that you can prevent being a victim of many ransomware attacks simply by installing the latest security patches as soon as they are released.

System administrators of MariaDB, MySQL, and SQL Server can help protect their server instances by monitoring performance on a regular basis. Navicat Monitor is the best way to always know exactly what is happening on your database servers. Monitoring is especially effective in cases where hackers make tentative intrusions into systems to ascertain the best targets on which to introduce their malware. Vigilant monitoring helps highlight unusual system activity so it can be addressed by database administrators and security personnel.

Conclusion

With ransomware attacks on the rise in 2020, now is the time to take steps to protect your data and organization. Navicat Monitor helps catch suspicious activity before your database is compromised!

Months can be notoriously difficult to work with due to a variety of factors, including their variability in length. To make database developers' jobs easier, most relational databases (DBMS) offer functions such as MONTH() and MONTHNAME(). These two functions are great for grouping results by month and for displaying their values. In today's blog, we'll learn how to use specialized SQL functions for working with months.

Working with Month Functions

The MONTH() and MONTHNAME() functions are both implemented in MySQL. However, every database type has their own date and time functions, so you will likely have to refer to the documentation to find the equivalent functions for your database. To illustrate, SQL Server does not provide the MONTHNAME() function. Instead, DATENAME() may be employed to return any date part, include the month name.

Navicat greatly simplifies looking up the right date function for your DBMS via its auto-complete list. More than just built-in functions, it includes everything from tables and views to stored procedures and user functions. Here is the MONTH() function:

Once you select a function or procedure, it is inserted into the SQL Editor at the cursor position with tabbable input parameters, ready to be filled in:

MONTH() and MONTHNAME() Described

The MONTH() function accepts a date and returns an integer value which represents the month of a specified date between 1 and 12:

Meanwhile, MONTHNAME() returns the month name for a specified date:

Working with the MONTH() and MONTHNAME() Functions

Using both these functions together allows us to group and/or sort by month order while displaying the month name. To see them in action, let's write a query against the Sakila Sample Database that shows movie rentals for a given month. In case you aren't familiar with the Sakila Sample Database, it's a learning database for MySQL that represents a fictitious movie rental store chain. Here are the first several rows of the rental table in Navicat for MySQL:

Now, let's select the rental date and some film details, ordered by rental_date and film title:

To limit results to a given month, we can add a WHERE clause that includes the MONTH() function as follows:

To see the month name, we just have to include the MONTHNAME() function in the column list:

Filtering Results by Month AND Year

Without specifying a year in the WHERE clause, filtering results by a specified month will show results that span across ALL years of data. In many cases, this is not what you want. The solution is to include the YEAR() function, along with the MONTH() in the WHERE clause. Here's the updated query to limit results to May of 2005:

Conclusion

Database functions such as MONTH() and MONTHNAME() are highly useful for grouping results by month and for displaying their values. The two functions that we looked at today (three if you include YEAR()!) are supported by MySQL. For other databases, be sure to refer to the documentation to find their equivalent functions. After that, you can use Navicat's auto-suggest feature to insert the appropriate function into your SQL statements.

Behind the slick User Interface (UI) of modern web applications, there are asynchronous services fetching data from the database with a multitude of objectives, including loading drop-downs, populating data tables, Synchronizing components, and many others. Any lagging of the back-end processes will be perceived by the user as a slow or even a non-responsive application. This in turn degrades the user experience and sours their opinion of your application. For that reason, it is imperative that you whittle down your query response time to the lowest feasible value. In many cases, this means measuring query turn-around in hundreds of a second, as opposed to seconds.

Needless to say, achieving sub-second response times takes some doing beyond defining indexes on searchable fields. In today's blog, we'll take a look at some techniques for making your queries maximally performant in MySQL 8.

The EXPLAIN Command

A good way to see what a query needs in order to perform better is to use the EXPLAIN command. It returns a formatted description of the query optimizer's execution plan for the specified statement. You can use this information to analyze and troubleshoot the query.

By default, EXPLAIN output represents the query plan as a hierarchy whereby each level represents a single database operation that the optimizer defines to execute the query. It takes a bit of practice to get accustomed to EXPLAIN's output, but the more you do it, the better you'll get understanding where your queries need tweaking.

In Navicat for MySQL, there's a button in the SQL Editor that runs EXPLAIN for me. Results are displayed in an easy-to-read grid format:

Query Profiling

You can use query profiling to measure query execution time. Here's how do to that in MySQL:

• Start the profiler with:

  SET profiling = 1; 

• Then execute your query with:

  SHOW PROFILES;

• You'll then see a list of queries the profiler has statistics for. Choose which query to examine with the statement:

  SHOW PROFILE FOR QUERY 1; 

  ...or whatever number is assigned to your query.

You'll then get is a list where exactly how much time was spent during the query:

You can also get a profile of CPU usage:

Analyzing Query Performance using a Navicat Monitor

Navicat Monitor is an agentless remote server monitoring tool that is packed with powerful features to make your monitoring effective as possible. It works with MySQL, MariaDB and SQL Server, as well as cloud databases like Amazon RDS, Amazon Aurora, Oracle Cloud, Google Cloud and Microsoft Azure. The Query Analyzer screen shows information of all executing queries. You can use it to further analyze and evaluate your query performance:

The screen is divided into several sections:

• Latest Deadlock Query: Shows the transaction information of the latest deadlock detected in the selected instance.
• Process List: Displays the total number of running processes for the selected instance, and lists the last 5 processes including ID, command type, user, database and time information.
• Query Analyzer: Displays information about query statements with customizable and sortable columns.

Conclusion

In today's blog, we'll take a look at some techniques for making your queries as expeditious as possible in MySQL 8.

Interested in Navicat Monitor? You can try it for 14 days completely free of charge for evaluation purposes!

It's fairly common to allow certain users to perform ad-hoc updates or deletions to tables. Data Manipulation Language (DML) operations such as these always come with risk, and incidents may occur where someone accidentally issues a Delete command without a WHERE clause, thereby deleting all rows in a table! Luckily, there are some simple steps you can take to prevent accidental (or deliberate!) destructive DML operations. We'll examine a couple of these in today's blog.

A Dangerous Deletion Procedure

As a starting point, let's take a stored procedure that will delete rows in a table based on a user-supplied where clause. This SQL Server procedure, shown in the Navicat for SQL Serverdevelopment and admin client, accepts a table and whereclause parameter and returns the number of rows deleted:

In Navicat, we can run a procedure right from the editor via the Execute button. Clicking it brings up a dialog to enter parameters:

The delcnt specifies the count of records we expect to be deleted. By leaving it blank, if any rows are deleted, the transaction is rolled back, and the row(s) preserved. Had we supplied a number, the procedure would have compared it to the @@rowcount server variable after the delete operation to determine if the number of records deleted match the expected number. I our case, a message is displayed to tell us how many rows would have been deleted:

As expected, the actcnt is zero, confirming that no rows were actually deleted:

The delcnt parameter acts as a built-in fail-safe in that it forces the user to specify how many rows that he/she expects to be deleted. We could also add one additional layer of safety by checking that the whereclause parameter was supplied:

Maximizing delcnt Variable Checks

Disallowing an empty where clause is not without drawbacks, in that there is nothing stopping someone from entering something like "id is not null". The best we can say about this safety check is that at least it would only fail in cases where the user to deliberately deletes all rows in a table.

For that reason, perhaps a better solution may be to expand on the idea of comparing the actual count of affected (deleted) rows to the expected count (@delcnt). With only a few extra lines of code, we can count the number of rows in the table and rollback the transaction if the number of affected rows is equal to the total number of rows in the table. This can be accomplished using the built-in sp_executesql stored procedure. It supports the use of both input and output parameters so we can store the results of the count(*) function to a variable. Here is the new code:

Now, if we try to run a query that would delete all of the rows in the table such as follows:

...the deletion is prevented:

Conclusion

While there is no sure-fire way to prevent data loss due to accidental or deliberate deletion, every counter-measure that we employ helps minimize the chances of a catastrophic event from taking place.

Interested in Navicat for SQL Server? You can try it for 14 days completely free of charge for evaluation purposes!

Certain relational databases, including MySQL and SQL Server, have an INFORMATION_SCHEMA system database. It contains database metadata, such as the names of databases, tables, the column data types, and even access privileges. It's also sometimes referred to as the data dictionary or system catalog. Regardless of how you refer to it, the INFORMATION_SCHEMA database is the ideal place to obtain details about table columns. In today's blog, we'll use the INFORMATION_SCHEMA database to find out whether or not a column exists and how many columns a particular table has.

Viewing the INFORMATION_SCHEMA database in Navicat

Being a system table, you won't be able to see the INFORMATION_SCHEMA database unless you explicitly tell Navicat to show it. To do that, add the INFORMATION_SCHEMA database to the Databases list defined within a database connection:

That allows us to open the Columns table in the Table Designer or Viewer:

The sheer number of columns should give you some idea as to what types of information we can obtain from the Columns table.

Note: the INFORMATION_SCHEMA is a read-only database, so you can't make any changes to its structure or contents.

Introducing the Column Count Query

The Columns table may be queried like any other to look up information about table columns. Here's the basic syntax to do so:

SELECT count(*) AS anyName FROM information_schema.columns WHERE [table_schema = 'yourSchemaName' AND] table_name = 'yourTableName'; 

The table_schema is the database in which the table resides. It's not crucial to the query, but in cases where you have more than one database with the same column name, it filters results to that particular database table. In a situation where you maintain multiple copies of the same database, the column count will be for of all tables with the same name.

For instance, I have four copies of the Sakila database:

As a result, when I run the query without the table_schema, I get a column count of 51, which is on the high side!

Specifying the table_schema in a more accurate column count of 12:

If we now open the film table in the Table Designer, we can confirm that 12 columns is correct:

Determining Whether or Not a Column Exists

In a dynamic application, you may want to look up information about a column, including whether or not it exists. Here's a query that lists every instance of the "title" column, along with meta-data about each of them, including which schema and table it belongs to, as well as details such as the default value, data type, and maximum length:

Conclusion

In today's blog, we learned how to utilize the INFORMATION_SCHEMA database to find out whether or not a column exists and how many columns a particular table has.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

It's been said that second place is the first loser. So, who needs an SQL statement to find out who these under achievers are? Surprisingly, a lot of people. In fact, the official term for this type of query is "nth highest value of a column". That's because techniques for selecting the 2nd highest value may also be applied for any value. In today's blog, we'll learn how to use ORDER BY DESC in conjunction with the LIMIT clause to obtain the 2nd highest value, and others, from a table.

Introducing the Classic Models Database

The classicmodels database is a MySQL sample database to help learn SQL quickly and effectively. The classicmodels database represents a retailer of scale models of classic cars. It contains typical business data such as customers, products, sales orders, sales order line items, etc.

Here's a peek at the contents of the payments table in Navicat Premium:

We will compose a query that selects the 2nd highest payment from this table.

About the LIMIT Statement

The LIMIT clause may be added to a SELECT statement to constrain the number of rows returned. The LIMIT clause can accept either one or two arguments of zero or positive whole integers.

Here's the syntax:

SELECT     select_listFROM    table_nameLIMIT [offset,] row_count;

• The offset specifies the offset of the first row to return. The offset of the first row is 0, not 1.
• The row_count specifies the maximum number of rows to return.

Selecting the 2nd Highest Payment

Knowing what we know about the LIMIT clause, we can now structure our SELECT statement as follows to fetch the 2nd highest value:

SELECT * FROM yourTableName ORDER BY DESC yourColumnName LIMIT 1,1;

Here is the equivalent statement to SELECT the 2nd highest amount from the payments table:

Verifying the Results

In Navicat, we can sort a table or view by any column by hovering the mousepointer over the column header and then clicking the context menu arrow. We can then choose the sort order from the list:

If we refer to the highlighted row in the image below, we can confirm that it is the correct one.

Selecting the Nth Highest Payment

We can use the same syntax to fetch other amounts. For example, you could return the fourth highest value of a column by using the following syntax:

SELECT * FROM yourTableName ORDER BY DESC yourColumnName LIMIT 3,1;

In fact, we can use this syntax for any ranking:

SELECT * FROM yourTableName ORDER BY DESC yourColumnName LIMIT desiredRank - 1, 1;

Here's the query to fetch the 10th highest payment amount:

Conclusion

In today's blog, we learned how to use ORDER BY DESC in conjunction with the LIMIT clause to obtain the Nth highest value from a table. In next week's blog, we'll accomplish the same task using the TOP statement.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

All-too-often, database developers and administrators use Nulls, Zeroes, and Empty Strings interchangeably within their database tables. That's unfortunate, because Null, Zero, and an Empty String each represent something different in relational databases (RDBMS). As such, using these values incorrectly, or choosing the wrong one, can have enormous ramifications on the operation of your database and applications that rely on it. In today's blog, we'll explore how to best utilize the Null, Zero, and Empty String in database design and general usage.

What is Null?

The value Null has a long history in both relational databases and programming languages. It was devised as a special value to represent the intentional absence of any value. As such, it can be assigned to any nullable column. To designate a column as nullable, simply include the NULL keyword, or just leave it out, as columns are nullable by default:

CREATE TABLE table_name (    column1 datatype [ NULL | NOT NULL ],   column2 datatype [ NULL | NOT NULL ],   ... ); 

Navicat greatly simplifies the creation of tables via its Table Designer. In Navicat, to specify that a column may not contain NULL values, check the Not null checkbox:

In the Table Grid view, NULL values are represented as (Null):

In terms of semantics, a missing value informs us that we do not know it. Put more simply, a Null could mean "???". As we'll see shortly, this is not the case for either Zeroes or Blank Strings.

The Zero Value

A value of Zero (0) is a real number whose meaning is shared by other terms throughout the world, including nought (UK), naught (US), nil, zilch, zip, nada, scratch, and goose egg. Its value can be thought of as "Nothing". Consider a credit limit column. In that context, a value of 0.00 would indicate that the customer does not have credit. If the column is nullable, then a NULL value would mean that we don't know what the customer's credit limit is. In the first instance, we know what the customer's credit limit is, and that limit is zero.

In Navicat, we can set the default value of a column via the Default drop-down:

Note that NULL is the first option.

The Empty String Demystified

Much like Zero, an Empty String ("") differs from a NULL value in that the former specifically implies that the value was set to be empty, whereas NULL means that the value was not supplied or is unknown. As an example, let's consider a column that stores a cell phone number. An empty value would imply that the person does not have a cell phone, whereas a NULL would signify that he or she did not provide a number. These are two very different interpretations!

Conclusion

It is crucial for the database developer and administrator to understand the semantics of Nulls, Zeroes, and Empty Strings because using them incorrectly, or choosing the wrong value, can have enormous ramifications on the operation of the database and applications that interact with it.

To that end, Navicat Database Development and Administration clients facilitate working with Nulls, Zeroes, and Empty Strings by providing a Default drop-down and by clearly denoting Nulls in database tables.

If you have worked with relational databases (RDBMS) for any length of time, you have almost certainly utilized the SQL COUNT() function. As such, you are no doubt already aware that the COUNT() function returns the number of rows or columns in a table, as filtered by the criteria specified in the WHERE clause. Its flexible syntax and widespread support makes it one of the most versatile and useful functions in SQL. In today's blog, we'll take a look at its many permutations and learn how to obtain a variety of counts.

One Function, Many Input Parameter Variations

As an ANSI SQL function COUNT() accepts parameters in the general SQL 2003 ANSI standard syntax. Having said that, different database vendors may have different ways of applying the COUNT() function. MySQL, PostgreSQL, and Microsoft SQL Server all follow the ANSI SQL syntax:

COUNT(*)COUNT( [ALL|DISTINCT] expression )

Meanwhile, the DB2 and Oracle syntax differs slightly:

COUNT ({*|[DISTINCT] expression}) OVER (window_clause)

In this blog, we'll be focusing on the SQL 2003 ANSI standard syntax. Here are what all of the input parameters mean:

• ALL: As its name implies, ALL applies the COUNT to all values so that it returns the number of non-null values.
• DISTINCT: Ignores duplicate values so that COUNT returns the number of unique non-null values.
• expression: An expression made up of:
  • a single constant, variable, scalar function
  • a column name
  • part of a SQL query that compares values against other values
  An expression may not include text or image types. Aggregate functions and subqueries are also not permitted.
• *: COUNTs all the rows in the target table whether or not they include NULLs.

A Practical Example

To sample some of the various syntax permutations and their effects on COUNT output, let's apply the COUNT() function to the following employees table, shown in Navicat for MySQL:

Now, here's a query that counts several things:

• the total number of employees
• the number of managers
• the number of non-managers
• the number of departments

As the above query demonstrates, obtaining different counts is all about how in how you use the COUNT() function. In terms of when to use a specific column name as opposed to the asterisk, note that the former will not count nulls, whereas the latter typically will, because it includes all columns. As to when to use DISTINCT, consider using it for columns that have repeated values, which tends to include columns defined as non-unique and/or not the Primary Key.

Conclusion

The SQL COUNT() function's flexible syntax and widespread support makes it one of the most versatile and useful functions in SQL. Speaking of syntax, if you ever find it difficult to remember the COUNT() function's syntax, you can let Navicat remind you! The auto-complete suggestion list not only provides table and column names, but also stored procedures and functions, including COUNT(). You'll find not one, but two versions of it: one for simple usage and another for more complex uses:

When it comes to storing formatted fields in a database, the adage "store raw, display pretty", usually holds true. In most cases, raw values are the most conducive for working with in the database, allowing them to be queried, sorted, compared, and what-have-you. Yet, there are times that you may want to leave in special characters, where they are essential to formatting, such as HTML markup. In today's blog, we'll explore both options with examples using Navicat Premium.

Parsing Out Special Characters

Consider a field that stores phone numbers. Just in North America, a phone number can be represented in many different formats, including "(800) 555-1212", "800-555-1212" "800 555-1212", or "8005551212". To store short pieces of variable data like phone numbers, it's usually best to strip out special (non-numeric) characters at the application layer before storing to the database. The application would also be responsible for presenting phone numbers in the predetermined display format. If you're concerned that all of this parsing and reformatting of data will place unnecessary strain on the server, rest assured that the processor overhead of formatting a phone number is trivial, taking far less than a microsecond in real-time.

Data Type Considerations

Some people think that numeric data like phone numbers lend themselves to a numeric data type such as int or bigint. That being said, most DBAs choose the char or varchar type over numeric ones, as non numeric characters can be valid in phone numbers. A prime example being + as a replacement for 00 at the start of international numbers.

For evidence of this practice, look no further than the Sakila Sample Database. There, you'll find a phone number column in the address table. Here it is in the Table Designer of Navicat Premium:

Here, the phone field is given a length of 20 in order to accommodate a variety of phone numbers. A quick glance at the table contents shows the varying phone number lengths:

The good thing about varchar fields is that, if you ever needed increase the column's capacity, you could do that easily enough using an ALTER TABLE statement, or simply by changing the Length property in Navicat.

Preserving Formatting of Longer Fields

For longer fields that contain formatted of free form user input, like descriptions, you may find it preferable to store them in a varchar or text column with all of the special characters included because there would be no way to reformat them for displaying later.

Viewing Free Form Content in Navicat

Content that spans more than one line can be difficult to work with because the typical Grid view only shows one row per record:

Navicat offers a couple of ways to view longer fields:

Form View

The Form View allows you to view, update, insert, or delete data as a form, in which the current record is displayed by field name and its value. Form View also provides pop-up menus with the following additional functions:

• set the field value as Null/Empty String
• use current field value as a filter
• format form view

Text Editing

Navicat provides Text/Hex/Image/Web drop-down to view and edit TEXT/BLOB/BFile/HTML field content. To enable viewing/editing of a data type, select the type from the drop-down and toggle it to the ON position. In the case of TEXT, you will see an editor appear at the bottom of the Table Grid:

Conclusion

In today's blog, we learned how to store formatted data using Navicat Premium.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Adding the DISTINCT keyword to a SELECT query causes it to return only unique values for the specified column list so that duplicate rows are removed from the result set. Since DISTINCT operates on all of the fields in SELECT's column list, it can't be applied to an individual field that are part of a larger group. That being said, there are ways to remove duplicate values from one column, while ignoring other columns. We'll be taking a look at a couple of those here today.

The Test Data

In order to test out our queries, we'll need a table that contains duplicate data. For that purpose, I added some extra emails to the Sakila Sample Database's customer table. Here's a screenshot in Navicat Premium's Grid view that shows a customer that has 2 associated emails:

If we were now to add the DISTINCT clause to a query whose field list contains other columns, it does not work because the row as a whole is unique:

So, what does work? Let's find out!

Using Group By

The GROUP BY clause applies aggregate functions to a specific subset of data by grouping results according to one or more fields. When combined with a function such as MIN or MAX, GROUP BY can limit a field to the first or last instance, relative to another field.

Therefore, if we wanted to limit emails to one per customer, we could include a sub-query that groups emails by customer_id. We can then select other columns by joining the email column to the unique ones returned by the sub-query:

Using a Window Function

Another, albeit slightly more advanced solution, is to use a Window function. Window functions are thus named because they perform a calculation across a set of table rows that are related to the current row. Unlike regular aggregate functions, window functions do not cause rows to become grouped into a single output row so that the rows retain their separate identities.

ROW_NUMBER() is a window function that assigns a sequential integer to each row within the partition of a result set, starting with 1 for the first row in each partition. In the case of customers' emails, the 1st email returns 1, the 2nd returns 2, etc. We can then use that value (referenced as "rn" below) to select only the 1st email for each customer.

It should be noted that not all relational databases support window functions. SQL Server does support them, while MySQL introduced them in version 8.

Conclusion

In today's blog, we learned how to remove duplicates from individual fields that are part of a larger group using GROUP BY and Windows functions. There are undoubtedly many other ways of achieving the same end goal, but these two tried-and-true techniques should serve you well.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Grouping query results into buckets of equal size is a common requirement for database developers and database administrators (DBAs) alike. Examples include:

• customers whose last names begin with A - L and M-Z
• products prices that are between 1 - 10 dollars, 11 - 20 dollars, 21 - 20 dollars, etc...
• quarterly sales, i.e., from Jan - Mar, Apr - Jun, Jul- Sep, Oct - Dec

Standard SQL is well suited to this task. By combining the power of the CASE statement with the GROUP BY clause, data can be broken up into whatever range we deem necessary to best interpret our data. In today's blog, we'll compose a couple of range queries in Navicat Premium's excellent Query Editor.

Splitting Grades into Percentiles

Our first example will require a table containing the grades of several students. Here's the SQL to create and populate the grade table:

DROP TABLE IF EXISTS `grade`;CREATE TABLE `grade`  (  `StuID` int(11) NULL DEFAULT NULL,  `Semester` tinyint(4) NULL DEFAULT NULL,  `YEAR` int(11) NULL DEFAULT NULL,  `Marks` int(11) NULL DEFAULT NULL) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (110, 1, 2018, 66);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (110, 3, 2018, 77);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (110, 2, 2018, 86);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (110, 4, 2018, 69);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (100, 1, 2018, 20);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (100, 2, 2018, 39);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (100, 3, 2018, 65);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (100, 4, 2018, 70);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (99, 1, 2018, 50);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (99, 2, 2018, 45);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (99, 3, 2018, 90);INSERT INTO `grade`(`StuID`, `Semester`, `YEAR`, `Marks`) VALUES (99, 4, 2018, 96);

Here is the grade table in Navicat:

Let's say that we wanted to count students' grades by equal percentile quadrants as follows:

• 0 to 25
• 26 - 50
• 51 to 75
• 76 to 100

Here's the query to do that, along with the results generated:

Pay attention to the CASE statement and you'll notice that it defines each ranges using the BETWEEN operator. It selects values within the inclusive range, meaning that the outer values are included in the range. BETWEEN works with many types of data, including numbers, text, and dates.

Working with Dates

In many cases, dates can be split into logical segments using one of the DATE type's many date part functions, such as DAY(), DAYOFMONTH(), DAYOFWEEK(), DAYOFYEAR(), MONTH(), YEAR(), etc. These allow you to break up your ranges by intuitive units.

To demonstrate, here's a query in MySQL using the Sakila Sample Database that calculates the average rental cost for each customer, broken down by year and month:

The advantage to using DATE functions is that they allow us to dispense with the CASE statement because we can GROUP BY the same functions.

Conclusion

In today's blog, we learned how to write range queries using Navicat Premium's excellent Query Editor. Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Output parameters are a feature of stored procedures that is seldom used, which is a shame because they are an excellent option for returning scalar data to the user. In today's blog, we'll learn some uses for Output Parameters and how to use them in your stored procedures.

Syntax

The exact syntax for declaring parameters differs somewhat from one database vendor to another, so let's look at a couple of different examples. Here's one in SQL Server that simply parrots back the input parameter to the user:

In MySQL, there are slight differences in syntax, such as the IN/OUT parameters being located before the parameter names:

Some relational database (RDBMS), such as MySQL, support INOUT parameters. These are a combination of IN and OUT parameters, in that the calling program first passes in the INOUT parameter, and then, the stored procedure modifies it before sending the updated value to the calling program. Other RDMBS, such as SQL Server, treat OUT parameters like an INOUT by allowing them to be passed in to the procedure.

An Slightly More Complex Example (in MySQL)

The Sakila Sample Database was originally created as a learning tool for MySQL, but has since been ported to other DBMS as well. It's themed around a fictional video rental store, and contains a number of user functions and stored procedures. Some of these, such as the film_in_stock procedure, includes both IN and OUT parameters. Here is its definition in Navicat Premium:

The film_in_stock stored procedure determines whether any copies of a given film are in stock at a given store. As such, it declares two input parameters - the film ID and store ID - as well as an output parameter that relays the count of films in stock. A user function could have been employed for this purpose, but a procedure can also list the IDs of every film in stock. That's why there are two SELECT statements in the procedure body (between the BEGIN and END delimiters). The first SELECT fetches the film IDs, while the second populates the output parameter with the number of found rows.

Running the film_in_stock Stored Procedure

In Navicat, we can run a procedure directly from the designer via the Execute button. Clicking it brings up a dialog for entering input parameters:

A stored procedure may return multiple result sets and/or output parameters, so to deal with this, Navicat shows each in its own Result tab. The first tab shows the result set produced by the first query in the procedure, i.e., the inventory IDs of films that are in stock:

The second tab shows the count of films in stock at the store identified by the p_store_id input parameter (basically the number of rows returned by the first query):

Conclusion

In today's blog, we saw how the flexibility provided by the combination of input/output parameters and result sets makes stored procedures a truly powerful tool in the database developer's arsenal.

Theres an adage for user privileges that you should assign a user the least amount of privileges that he or she requires to perform their job function(s) and no more. That is why MySQL offers such a fine-grained access control system. While not the easiest system to grasp, once a DBA does, he or she tends to agree that it really is quite effective. In today's blog, we'll learn how to prevent a user from listing databases in MySQL.

Working with the mysql.user Table

The mysql.user table contains information about users that have permission to access the MySQL server, along with their global privileges. Although it is possible to directly query and update the user table, it's best to use GRANT and CREATE USER for adding users and privileges. To see the contents of the user table, we can use the DESC command:

The privilege that allows a user to obtain a list of databases via the SHOW_DATABASES command is Show_db_priv. Hence, on a new database, we can simply not add the user to it to prevent the user from seeing it. Otherwise, you can see which privileges a user currently has by issuing the SHOW GRANTS command:

SHOW GRANTS FOR 'bob_s'@'localhost';

Here's some example output for the "bob_s@localhost" user in Navicat Premium:

Revoking a User Privilege

The above output confirms that bob_s does have the SHOW DATABASES privilege. If we now wanted to remove that privilege, we can issue the REVOKE command:

REVOKE Show_db_privON sakilaFROM bob_s'@'localhost;

In Navicat, we can set a user's privileges both at the server and database level on the Server Privileges and Privileges tabs of the user details. To access them, click the User button on the main button bar, select the user that you're interested in, and then click the Privilege Manager button on the Objects toolbar:

Here are bob_s@localhost's server-level privileges (including SHOW DATABASES):

To revoke the SHOW DATABASES privilege, we can simply uncheck the box beside the SHOW DATABASES label, and click the Save button.

Here are bob_s@localhost's privileges for the sakila database:

There, we can set also database-specific privileges such as those to create views, show views, drop tables, execute INSERT statements, etc. We can even manage privileges at the table and column level!

Conclusion

In today's blog, we saw how to prevent a user from listing databases in MySQL, both via the MySQL REVOKE command and using Navicat's Server Privileges and Privileges tabs. Which is the easier of the two is up for debate, but I personally find the checkbox approach more intuitive.

To learn more about managing users in Navicat, take a look at the Manage MySQL Users in Navicat Premium series:

• Part 1: Securing the Root
• Part 2: Creating a New User
• Part 3: Configuring User Privileges
• Part 4: The Privilege Manager tool

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

Fetching rows that have a particular value, but not others, is a fairly common task in database development and administration. It sounds like a walk in the park, but limiting the results to those rows that possess one value to the exclusion of others is trickier than it sounds. The reason is, while it's trivial to filter out values using the != not equals or NOT IN comparison operators, these only hide values rather than tell us whether or not an entity possesses these other values. The good news is that there's an easy way to do it. Read on to find out how!

Selecting Users by Role

One thing that all databases - and most applications - have are users. In particular, database users tend to have different roles. (Although application users may also have roles.) Here's an example of such a table in Navicat's Table Designer:

In this case, the role_id would be a Foreign Key (FK) that links to a roles table that would store additional information about each role. In the users table, the inclusion of the role_id leads to the possibility of having multiple rows for each user. Keep that in mind, because that idea will be revisited a little later on...

The Wrong Way to List Users That Have One Role but No Others

If were were to now list users who possess a particular user role, and only that role, we might be tempted to write something like the following:

The problem is that the above query only lists users who have a role_id of 1. It does nothing to address whether or not they have other role_ids as well. Moreover, adding another criteria such as AND role_id NOT IN (2,3,4,5,6,7,8,9) does nothing to help because 1 is obviously not any other number!

So, how do we limit users to those that have only a role_id of 1 and no others?

The RIGHT Way to List Users That Have One Role but No Others

Well, this is technically not "The RIGHT Way" because there are surely others. This solution consists of counting the number of rows for each user. The idea is that, if a user has the role_id that we're interested in AND only has one row in the table, then they're someone we want to see in the results.

We can obtain a count of user_ids for each user in the table using a GROUP BY. Then, the HAVING clause can check that a user only has one row and that his/her one role_id is the one we want:

Now we only see the one user who has a role_id of 1, and no others!

Conclusion

One good thing about this approach is that it can easily be modified to find rows that contain multiple values or more than a certain number of rows in the table.

Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

A short time ago we were introduced the incredibly useful and versatile Case Statement. In that blog, we employed the Case Statement as most DBAs and developers do, in the SELECT clause. Another way to use the Case Statement is within the WHERE clause. There, it may be utilized to alter the data fetched by a query based on a condition. Within that context, the Case Statement is ideally suited to both static queries, as well as dynamic ones, such as those that you would find inside a stored procedure. In today's blog, we'll create a SELECT query in Navicat Premium that returns rows based on the values of another field.

Listing Films by Rental Duration

Before getting to the CASE Statement, let's start with a query that returns a list of movies from the from the Sakila Sample Database. It's a MySQL database that contains a number of tables, views, and queries related to a fictional video rental store. Tables include actor, film, customer, rentals, etc.by film id, title, rental rate, and a rental duration of 5 days.

Our query displays the film id, title, rental rate, and a rental duration columns and narrows down the field to those films whose rental_duration is exactly 5 days. Here's is the statement:

SELECT film_id, title, rental_rate, rental_duration FROM film WHERE rental_duration = 5 ORDER BY rental_rate DESC;

Executing the query in Navicat Premium brings up the following results:

Setting Rental Duration Based On Rental Rate

A Case Statement comes in handy for choosing a value based on several other possible values. For instance, let's suppose that we wanted to set the rental_duration based on the rental_rate. We could do that using a Case Statement like so:

SELECT film_id, title, rental_rate, rental_duration FROM film WHERE rental_duration = CASE rental_rate                        WHEN 0.99 THEN 3                        WHEN 2.99 THEN 4                        WHEN 4.99 THEN 5                        ELSE 6                        END ORDER BY title DESC;

That has the effect of associating rental_rates with the rental_durations. Hence:

• When the rental_rate is equal to 0.99, only include films whose rental_duration is equal to 3.
• When the rental_rate is equal to 2.99, only include films whose rental_duration is equal to 4.
• When the rental_rate is equal to 4.99, only include films whose rental_duration is equal to 5.
• For any other rental_rate, only include films whose rental_duration is equal to 6.

We can see the results in the screen capture below:

Notice how whenever a film has a rental rate of 0.99, the rental_duration is always 3. Likewise, films with a rental rate of 2.99 all have a rental_duration of 4, etc...

Rewriting the CASE Statement

Remember that the Case Statement is only an anternative way of combining two or more OR conditions. As such, we can rewrite our query without the Case Statement, but, as you can see, it takes a lot more SQL:

SELECT film_id, title, rental_rate, rental_duration FROM film WHERE rental_rate = 0.99 AND rental_duration = 3  OR rental_rate = 2.99 AND rental_duration = 4  OR rental_rate = 4.99 AND rental_duration = 5  OR rental_rate NOT IN (0.99, 2.99, 4.99) AND rental_duration = 6ORDER BY title DESC;

Here are the results. Compare these with that of the CASE query:

Conclusion

In today's blog, we created a SELECT query in Navicat Premium that returns a list of films using a CASE Statement in the WHERE clause. Interested in Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

With so many factors to consider, uncovering the root cause(s) of slow query execution takes an organized approach. Luckily, with a bit of effort, you can pin down an issue to one of the more common culprits by checking up on a few things. In today's blog, we'll learn how Navicat Monitor 2 can help you get the the bottom of slow query execution - fast!

Network Issues

Database servers are designed to be accessed over a network, be it an internal or external one, like the World-wide Web. As such, the occasional dropping of a connection, or even outages that last for hours or days are to be expected, on occasion. Good performance in a local environment is a promising sign, but isn't necessarily enough to exclude network issues entirely, as the server itself could be overloaded. You can test for that using a monitoring tool that can track server OS metrics like CPU processes and memory. That's where Navicat Monitor can help, by tracking O/S metrics.

On Windows Type servers, you can configure CPU & Memories section to monitor O/S metrics over Simple Network Management Protocol (SNMP):

Doing so will cause server metrics like CPU, Memory, and Disk Usage to appear in Dashboard Instance Cards:

You can also click on the system metrics to see more details, including Swap Usage, Connections, and Network Throughput. Each metric includes an interactive chart:

Query Monitoring

Once you've ruled out network issues, it's time to take a closer look at the query itself. A query can be functionally correct in that it fetches the correct data, but still be deficient by doing so in an efficient manner. It's essential to design your queries in such a way as to maximize efficiency because, depending on the database engine, all queries are likely to be run sequentially as a queue. Case in point MySQL's MyISAM engine acquires a table level lock when executing queries in order to protect data integrity during transactions. During that time, other processes/queries must wait their turn while the first query completes. If it's a stalwart, that wait could wind up being a long one!

Navicat Monitor's Query Analyzer screen to be very helpful in this regard. It shows the summary information of all executing queries and lets you spot problematic queries, including:

• top queries with cumulative execution time count
• slow queries with unacceptable response time
• deadlocks (when two or more queries permanently block each other)

Conclusion

In today's blog, we learned how Navicat Monitor 2 can help you get the the bottom of slow query execution - and fast!

Navicat Monitor is a safe, simple and agentless remote server monitoring tool for MySQL, MariaDB and SQL Server. It includes a rich set of real-time and historical graphs that allow you to drill down into server statistic details. The latest release of Navicat Monitor (version 2.0) now supports SQL Server as well!

Navicat Monitor version 2.0 is now available for sales at Navicat Online Store and is priced at US$499/token (commercial) and US$199/token (non-commercial). 1 token is needed to unlock 1 MySQL Server / 1 MariaDB Server / 1 SQL Server.

Click here for more details about all of Navicat Monitor's features, or, download the 14-day fully functional free trial!

Although not as proficient at string manipulation as procedural programming languages such as Java, C++, and PHP, SQL does provide many functions for working with string data. These may be employed to trim off extra spaces or characters, determine how long a string is, and concatenate several field values together. String functions are well worth becoming acquainted with as they can help make your code more effective and readable. In today's blog, we'll learn how to count the number of string occurrences within a char, varchar or text field using a couple of native SQL string functions.

Introducing the LENGTH() and REPLACE() Functions

The two functions that we'll be using here today are LENGTH(str) and REPLACE(str, from_str, to_str). LENGTH() returns the length of a string in bytes; REPLACE() returns the string str with all occurrences of the string from_str replaced by the string to_str, by performing case-sensitive matching.

The LENGTH() function returns the length of a string in bytes. This has some important ramifications because it means that for a string containing five 2-byte characters, LENGTH() returns 10. To count straight characters, use CHAR_LENGTH() instead.

Here's an example:

Here's an example of the REPLACE() function that changes the protocol of a URL from "http" to "https":

Let's Get Counting

By combining LENGTH() and REPLACE() with the ROUND() function, we can obtain a count of a specific sub-string in a field that contains textual content. Here's an example using the Sakila Sample database that returns the count of the word "Documentary" in the description field of the film table:

In essence, our query replaces occurrences of the target sub-string with an empty ("") string and compares the resulting string lengths. The difference between them is the number of occurrences of the sub-string in the source field.

Incorporating Our Query Into a User Function

If you plan on performing word counts on many different tables or using a variety of sub-string values, you should consider incorporating the main calculation into a custom User Function. Here's a function, named `count_string_instances`, that I created in Navicat:

Testing the Function

We can test our function in-place by clicking the Execute button. That opens a dialog to accept input parameters:

The results confirm that the function is working correctly:

Invoking Our Function from the Query

With our function in place, we can replace the calculation portion of the query with a call to the count_string_instances() function. As we begin to type the function name, the Navicat auto-suggest list now includes our function!

As with all functions, it is inserted into our query with input parameters ready to set. We can navigate between them via the TAB key:

Here's the updated query with results:

Conclusion

There are many SQL string functions that can help make your code more effective and readable. These can be especially powerful when combined. In today's blog, we learn how to count the number of string occurrences within a char, varchar or text field by creating a custom user function using Navicat's versatile Function and Stored Procedure Editor.

Part 2: When To Use Them

You probably already know that setting a default value on non-null columns helps get rid of those pesky "Field 'xyz' doesn't have a default value" errors. Hopefully you're also aware that keeping error messages at bay is not in-itself a valid reason for supplying default values. There are many reasons for providing default column values - some good, and some, less so. Part 1 explored the ramifications of MySQL's Strict SQL Mode, as well as how to view and set it using Navicat for MySQL 15. In today's follow-up blog, we'll tackle when to use default values, and how to come up with good ones.

Why Not Just Allow Nulls?

Nullable columns don't present the same challenges as non-null ones do, so why not allow nulls in all non-key columns? In many instances, the point of applying the non-null constraint to a column is to force the application or system that populates it to supply a value. Other times, a non-null column might contain audit information, such as the user ID or a timestamp. In either case, you're looking for valid data, and not just filler.

That's an important consideration because it drives home the importance of generating useful defaults as well as front-end validation. I can still remember my first web application. It collected user details such as names, emails, and phone numbers. All of these fields were required, so clever users found all sorts of ways to circumvent entering their real information, such as entering phone numbers of 111-111-1111 and names such as "Elmer J. Fudd".

Generating a Timestamp

Now that we've gone over some reasons why automatically populating fields is worth doing any time you can do so, let's look at a common example of a generated value: an audit timestamp.

Several of the tables in the Sakila Sample Database feature a last_update column. These employ the timestamp data type; its value is set to the output of the MySQL CURRENT_TIMESTAMP function. In Navicat (Premium pictured below), you can set the default value via a drop-down list:

The Default value sets the timestamp on record creation, whereas checking the On Update Current_Timestamp box tells MySQL to update the timestamp on every UPDATE operation.

Sentinel Values

In RDBMS, a sentinel value is one that has a special meaning. For instance, a value of 999 in an age column would signify that it is unknown. I've also seen applications that employed "1900-01-01" for unknown dates. Sentinel values can be useful in cases where you want to assign a value of "unknown", whereas nulls mean "no value". Not everybody is fond of sentinel values because people and applications that work with the database have to be aware of all sentinel values in order to handle them properly.

Conclusion

While default - and by extension - sentinel values have their place in good database design and development, it's worth considering each value's purpose before assigning a value. Simply relying on default values to avoid working with nulls is probably not a good enough reason to do so.

Part 1: Strict SQL Mode

Getting errors when you don't supply a value for a non-null column can be an immense source of frustration. There's a way to minimize the occurrence of such errors by setting a default value for those columns. Seems like an easy fix, but, as in all things, the devil's in the details. You have to be careful that you don't add a bunch of generic - and useless - data to your tables just for the sake of making INSERTs easier. In today's blog, we'll learn about the ramifications of MySQL's Strict SQL Mode, as well as how to view and set it using Navicat for MySQL 15. In part 2 we'll cover when it makes sense to employ default values (and when it doesn't).

Strict SQL Mode and Adjusted Values

In MySQL, you can control how MySQL handles invalid or missing values in data-change statements such as INSERT or UPDATE by turning on Strict SQL Mode. A value is missing when a new row to be inserted does not contain a value for a non-NULL column that has no explicit DEFAULT clause in its definition. If strict mode is not in effect, MySQL inserts adjusted values for both invalid or missing values and produces warnings. Examples of adjusted values would be an empty string, zero, and a timestamp/date of 0000-00-00 00:00:00.

It should be fairly obvious that adjusted values could undermine the whole point of having defaults. Hence, it's usually good idea to activate strict SQL mode and provide a default value where appropriate. In Navicat, you can check your current value for SQL Mode on the Variables tab of the Server Monitor. You'll find it under Tools > Server Monitor in the main menu.

Strict SQL mode is enabled if either STRICT_ALL_TABLES or STRICT_TRANS_TABLES is present. When deciding on which to use, note that the latter is more forgiving as, if a value is missing, MySQL inserts the appropriate adjusted value for the column data type and generates a warning rather than an error. Moreover, processing of the statement continues. Meanwhile, invalid values are converted to the closest valid value.

Strict SQL Mode in Action

Let's compare Strict SQL Mode to the default SQL mode using the Sakila Sample Database. The actor table does not allow nulls in any column, as evidenced by the checkboxes under the Not null header:

If we disable Strict SQL Mode for the current session using the SET command and perform an INSERT that only supplies the last_name, the database accepts it, but provides an empty string for the first_name:

If we re-activate Strict Mode, the same INSERT now fails with an error message:

Conclusion

In today's blog, we learned about the ramifications of MySQL's Strict SQL Mode, as well as how to view and set it using Navicat for MySQL 15. In part 2 we'll cover when and when it doesn't make sense to employ default values.

A Top N query is one that fetches the top records, ordered by some value, in descending order. Typically, these are accomplished using the TOP or LIMIT clause. Problem is, Top N result sets are limited to the highest values in the table, without any grouping. The GROUP BY clause can help with that, but it is limited to the single top result for each group. If you want the top 5 per category, GROUP BY won't help by itself. That doesn't mean it can't be done. In fact, in today's blog, we'll learn exactly how to construct a Top N query by group.

Top N Query Basics

To gain a better understanding of a the Top N Query, let's compose one that selects the top 5 films with the longest running times from the Sakila Sample Database. If you aren't familiar with the Sakila database, it's a MySQL database that contains a number of tables, views, and queries related to a fictional video rental store. Tables include actor, film, customer, rentals, etc.

Grouping Results by Category

The GROUP BY clause applies an aggregate function to one or more fields so that the data relates to the groupings that you specify. It's a step forward in terms of grouping results, but GROUP BY still has a couple of limitations:

• it only provides the very first result (i.e. row) per group and ignores others,
• the columns are limited to those included in the grouping criteria and aggregated field(s). All other columns are not accessible.

This query uses GROUP BY to show the longest running film for each rating:

Notice that we can't include the film title title, because it is not part of either the GROUP BY or aggregated field(s).

Crash Course in Windows Functions

The term "window" in Windows Functions refers to the set of rows on which the function operates because the function uses values from the rows in a window to calculate the returned values. The set of rows within the window are aggregated into a single value.

To use a window function in a query, you have to define the window using the OVER() clause. It does 2 things:

1. Defines window partitions to form groups of rows, via the PARTITION BY clause.
2. Orders rows within a partition, via the ORDER BY clause.

A query can include multiple window functions with the same or different window definitions.

Our query uses the ROW_NUMBER() window function. It assigns a sequential integer number to each row in the query's inner window result set. We can use that value to limit the results for each rating to the top 5. That's done by ordering the length in descending order.

Conclusion

In today's blog we learned how to construct a query that fetches the top 5 rows per category in Navicat Premium. Version 15 adds over 100 enhancements and includes several new features to give you more ways that ever to build, manage, and maintain your databases than ever before!

Deciding whether do create a function in the database or in application code can be a daunting one. All-too-often, you don't realize that you've made the wrong choice until it's a big hassle to make an about-face. Worse still is the fact that many developers base their decision on whether they're most familiar with SQL or application coding! A better approach is to rely on the strengths of a technology to help guide your decision. In today's blog, we'll break down the decision making process when choosing between a user-defined function (UDF) and one that resides on the application side.

Database Power!

There are things that databases can do well, and things that they struggle with. Like stored procedures, functions are written in SQL. As such, they will excel at tasks where SQL shines. Here's a list of such tasks, along with why they are best done in SQL as opposed to application code:

• joins: in application code, this could require complex array manipulation
• filtering data (i.e., where clause): in code, this could require heavy inserting and deleting of items in lists
• selecting columns: again, in application code, this could require heavy list or array manipulation
• aggregate functions: in application code, this could require arrays to hold values and complex switch cases
• foreign key integrity: in application code, this could require queries prior to insert and assumes that no one will access the data outside app
• primary key integrity - in application code, this could also require queries prior to insert and assumes that no one will access the data outside the app

Attempting to do any of the above rather than relying in SQL inevitably leads to writing a lot of code and reduced efficiency, which translates to more code to debug and maintain as well as poor application performance.

On the flip-side, DBMS do not excel at complex procedural processing; that's the domain of application code. It's a big reason why the debugging facilities of an Integrated Development Environment (IDE) such as VS Code or Eclipse are far superior to anything you will find in a database development environment.

A Case Study

The Sakila Sample Database was developed as a learning tool and has been widely shared throughout the database community. It's a MySQL database that contains a number of tables, views, stored procedures, and functions pertaining to a fictional video rental store. One of those functions is called inventory_in_stock. It's a UDF that accepts an inventory_id and returns a boolean value indicating whether or not that film is in stock.

Here is the inventory_in_stock function definition in Navicat Premium's Function Designer:

Let's quickly run it to see how it works.

Clicking the Execute button brings up a dialog to accept the input parameters:

Here are the results:

A value of 1 indicates that the film is in stock

Now, consider what would happen if we were to replace that function with one that resides in the application. It would need to make two calls to the database to execute SQL statements. That would lead to additional network traffic and require us to maintain SQL within the application. This is a bad practice in general because it mixes database and application code together.

Conclusion

In today's blog we learned that you should place your custom function code where it can best benefit from the technology's strengths: within the application where complex procedural processing is required, and in the database where SQL is required.

Interested in finding out more about Navicat Premium? You can try it for 14 days completely free of charge for evaluation purposes!

Part III: Navicat Cloud FAQ

While Navicat Cloud has been around for a few years now, it has really come into its own recently, as the Covid-19 pandemic has forced organizations to implement a work from home protocol. We learned the basics of Navicat Cloud in the last blog, Navicat Cloud and Team Collaboration. If you're following along, that was part 2 in the series. In this last installment, we'll pick up were we left off last week and look at how Navicat Cloud can help your team be more productive while working remotely by answering your biggest questions.

How safe is Navicat Cloud?

Before I am ready to share anything work-related over an open network, I want to be sure that it's secure. In particular, I wouldn't want someone to learn how to connect to the database using connection data. Although Navicat Cloud does allow you to store your connection settings, these are not sufficient for connecting to the database. That's because your database passwords and data are not be stored to Navicat Cloud; only connection settings, queries, model files, and virtual groups are ever stored.

Where does Navicat Cloud store connection settings, queries, model files, and virtual groups?

Behind the scenes, Navicat Cloud uses Amazon Simple Storage Service (Amazon S3) for all online storage. Files are stored using 256-bit AES encryption and transferred between Navicat applications and the Navicat Cloud service over a secure SSL tunnel.

What happens if I lose my connection settings, queries, model files, and/or virtual group information?

Navicat Cloud includes Two-step verification. That's an additional security feature that provides an advanced authentication solution for your Navicat Cloud account. Should you experience hardware and/or software failures on your computer and mobile devices, rest assured that once your files are synced to the cloud, all your files are kept safe and sound.

How do I view my usage?

In Navicat Cloud, each connection, query, model, and virtual group counts as one unit. Your account comes with 150 units of free storage. Once you reach the storage limit, Navicat Cloud stops syncing and displays a warning message. Fear not, you won't lose any information and your files will be synced again automatically once storage space becomes available. How does that happen? You can either clean up stored objects to make room or purchase additional storage units.

You can see how many units you've used up on your Navicat Cloud account from the account profile in Navicat.

On the desktop version:

• Sign in to your Navicat Cloud account.
• Click on your name from the top right corner to open your account profile window.
• Click on View Details.

On the iOS version:

• Sign in to your Navicat Cloud account.
• Click on your avatar from the top left corner to open your account profile window.
• Click on USAGE.

Here's the usage dialog in Windows 10:

Conclusion

In this three-part series on working with databases remotely using Navicat, we learned how we can continue to be productive within a collaborative environment while working from home.

You can find out more about Navicat Premium 15 here. To learn more about Navicat Cloud, visit the product page.

Part II: Navicat Cloud and Team Collaboration

As the Covid-19 pandemic wears on, organizations that can support working from home have continued to remain productive while maintaining physical distancing. In last blog we learned how to access sensitive work data by establishing a secure connection to a remote database via Navicat Premium 15. Today's follow-up will introduce Navicat Cloud, an add-on feature inside Navicat Development and Administration products for collaborating with team members from across town to around the globe.

What is Navicat Cloud

Navicat Cloud is a cloud-based service that allows you to synchronize your connection settings, queries, models, and virtual group information across multiple devices. Storing files in Navicat Cloud will cause them to automatically show up in both desktop and mobile versions of Navicat, providing your team with real-time access to files anytime and from anywhere.

Connecting to Navicat Cloud

Navicat Cloud was integrated into Navicat products in version 11.1. Its access point is located under File > Navicat Cloud on the main menu:

In order to use the service, you'll need to create a Navicat ID and password using the Create Navicat ID link on the Sign In screen. The ID does double duty as both your registered email and ID.

Once submitted, a confirmation email will be sent to the email address that you provided. In the email, click the Activate Now link to login to Navicat Cloud via the Sign In screen.

Added Security using Member Roles

Navicat Cloud provides added security by allowing you to assign a role to coworkers for each project they work on. It grants them access to projects based on the role they play. Each role determines whether they can create, view, and modify project files. You can securely share your projects with members; you can also control who can see and edit the project.

Navicat Cloud defines 4 types of member roles:

• Owner: The owner is a project leader who creates the project. The owner has full privileges on the project and is the only member who can delete the project.
• Admin: The admin is a lead member who handles the administration aspects of the project. Admin had full read/write access to the project to which they are assigned, including the ability to add/remove a project member and change member roles.
• Member: The member is a project member who can read and write all project files. Navicat recommends that you use this role as the default for all members and assign other roles only as needed.
• Guest: The guest is a basic member with read-only access to project files. This role is useful for members who need to view, but not edit, the project.

Navicat Cloud Availability and Pricing

Navicat Cloud is available on all Navicat products and all platforms including Windows, macOS, Linux and iOS (iPhone, iPad and iPod).

It is offered via 2 subscription types:

The free Basic Plan includes:

• 3 Projects
• 150 Units
• 3 members per project

For larger projects, there's the Pro Plan. It costs $9.99 per month or $99 per year and includes:

• 500 Projects
• 5000 Units
• 500 members per project

Going Forward

Today's blog introduced Navicat Cloud, an add-on feature inside Navicat Development and Administration products for collaborating with team members from across town to around the globe.

In part 3, we'll learn how easy Navicat Cloud makes it to share files with team members to increase productivity.

Part 1: Connecting to a Remote Database Instance

Remote work has been on the rise for some time now. Today, for those organizations still operating during the Covid-19 pandemic, it has become a necessity. Luckily, popular database systems (DBMS) have long supported remote connections. Likewise, Navicat's database development and administration products are also well equipped to access databases remotely. In today's blog we'll learn how to establish a secure connection to a remote database instance using Navicat Premium 15.

Local vs. Remote Databases

While it is possible for database and client software to be installed on the same computer, in practice, that tends to only be the case for local development purposes. In an organizational setting, the database usually resides on a server that may be part of the organizational infrastructure or in the Cloud. In either case, the mechanisms for connecting to the database are much the same.

TCP/IP

TCP/IP is short for Transmission Control Protocol/Internet Protocol. It's really a suite of communication protocols used to interconnect network devices over the Internet. However, TCP/IP can also be employed as a communications protocol in a private LAN or WAN network. It's the easiest way to connect to a remote database, but offers the least security because unless the database and client(s) who interact with it both reside within an enclosed network, the data may be seen by anyone who cares to watch.

To establish a connection to the database, you must supply and endpoint. It can be the IP address of the database server or a domain such as acme.com. In some cases a port number will also be required. Here's the connection to an SQL Server instance running on Amazon AWS from Navicat for SQL Server:

In the case of TCP/IP connections, it is imperative that you utilize a secure user password.

SSH Tunneling

If you require a more secure connection, you can use SSH Tunneling. SSH stands for "Secure Shell server". It's called a tunnel because it allows you to "tunnel" a port between your local system and a remote server. Traffic is sent over the encrypted SSH connection, so it can't be monitored or modified in transit.

Here's the completed SSH screen in Navicat:

Secure Sockets Layer(SSL)

Another option for securing transmissions between the client and database is SSL. It's a protocol that was initially developed for transmitting private documents over the Internet. SSL works by binding the identities of entities such as websites and companies to cryptographic key pairs via digital documents known as X.509 certificates. Each key pair consists of a private key and a public key. The private key is kept secure, while the public key can be freely distributed via a certificate. Hence, before you can establish a secure connection, you must first install the OpenSSL Library and a certificate from a trusted authority.

To provide authentication details in Navicat, enable Use authentication and fill in the required information:

• Client Key File:
  The SSL key file in PEM format to use for establishing a secure connection.
• Client Certificate File:
  The SSL certificate file in PEM format to use for establishing a secure connection.
• CA Certificate File:
  The path to a file in PEM format that contains a list of trusted SSL certificate authorities.
• Specified Cipher:
  A list of permissible ciphers to use for SSL encryption.

Conclusion

In today's blog we learned how to establish a secure connection to a remote database using Navicat. In part 2, we'll learn how Navicat Cloud allows you to collaborate with team members from across town to around the globe.

Navicat Premium 15 adds over 100 enhancements and includes several new features to give you more ways that ever to build, manage, and maintain your databases than ever before!

Perform an Internet search for "database synchronization" and you're likely to receive a lot of information on synchronizing database data. Meanwhile, instructions on synchronizing database schema structure is less prevalent. There is an inherent risk of destroying existing data that comes with altering the database structure. For that reason, you have to be extra careful when doing so.

Navicat can be a tremendous ally in synchronizing database structures. In today's blog, we'll learn how to use Navicat Premium 15's Structure Synchronization wizard to update one database's schema structure to match that of another.

About the Structure Synchronization Wizard

The Structure Synchronization Wizard can be launched from the Tools menu. You'll find the Data transfer... and Data Synchronization... commands there as well:

Navicat introduced a new mechanism for structure synchronization back in version 12. It provides an easier and more intuitive way to visually compare and identify the differences between two databases. And it shows side-by-side Data Definition Language (DDL) comparison that makes it easy to locate all the object differences. You can then choose and reorder your synchronization scripts to update the destination database.

It should be noted that Navicat Data Modeler 3.0 also supports Structure Synchronization. It helps you to discover and capture changes made in the model and then apply them to a targeted schema.

Minimizing the Risk of Data Loss

Altering the structure of a database that already contains data is fraught with danger. Therefore, you should always backup your data before attempting to synchronization database structures. This can be easily accomplished using Navicat's Backup utility. You'll find it on the main button bar:

You can backup many types of database entities, including tables, views, functions/stored procedures, and events:

Structure Synchronization Steps

The wizard guides you through the each step of the synchronization process via several screens as follows:

Setting the Source and Destination Databases

The first screen sets the connection and database details where the target database structure will be compared to that of the source:

Structure Comparison

The Structure Comparison screen is where you can compare and identify the differences between two databases. You can group items by Operation or Object Type:

Deploying the Script

The third and final screen shows the generated deployment script:

To run the script, click the Execute button at the bottom of the screen. Results will be displayed on the Message Log tab of the same screen:

You can save the current Synchronization profile for later use or load an existing profile at any time.

There is also a Back button, should you wish to recompare database structures.

Conclusion

In today's blog, we learned about the inherent risk of destroying existing data that comes with altering the database structure and how to minimize it using Navicat Premium 15's Structure Synchronization wizard to update one database's schema structure to match that of another.

Interested in finding out more about Navicat Premium 15? You can try it for 14 days completely free of charge for evaluation purposes!

In databases, the NULL value is one that has a very particular meaning. Thus, it is important to understand that a NULL value is different than a zero value or a field that contains spaces. In today's blog, we'll explore what the NULL value means and how to work with NULLs in Navicat Premium.

What is NULL?

It should be noted that the NULL value is not unique to databases. It's found in most computer programming languages as well, where it evolved as a built-in constant that had a value of zero. In fact, it's the character 0 that's still employed to terminate strings in C. However, over time, it came to mean "nothing". Specifically, NULL is a pointer to a special bit pattern that represents a NULL. This is what is commonly referred to as a "null pointer".

In a database, zero is a value which has meaning, so the value NULL became is a special marker to mean that no value exists. In that sense, NULL does not refer to a memory location, as it does for programming languages. In a database, the NULL value indicates a lack of a value, which is not the same thing as a value of zero. To illustrate, consider the question "How many CDs does Rob own?" The answer may be "zero" (0) or NULL. In the latter case, the NULL value could mean that we do not know how many CDs Rob owns. Later, the value might be updated with a numeric vale once we have ascertained how many CDs Rob owns.

Allowing NULLs as a Column Value

Any column that is part of a KEY must not allow NULLs, but for other fields, it's completely up to you. Including the NOT NULL clause next to your column definitions at table creation time will force the user to include a value for that column in INSERT and UPDATE operations. Otherwise, an error will be thrown, and the operation will fail. Navicat Premium's Table Design screen includes a Not null column to designate a field as nullable or non-nullable. Here it is in Navicat 15's new Dark Mode:

On new fields, the Not null checkbox is automatically deselected.

Referencing NULL Values in Queries

When writing queries, there are times that you'll want to filter out rows with NULL values. Other times, you'll specifically want to retrieve rows that contain NULLs. Here's how to do both:

Queries can filter out nulls using the IS NOT NULL clause. Here's an example that excludes films from the Sakila database that contain a NULL original_language_id:

Likewise, you can find out which rows contain a NULL value by dropping the NOT from the NOT NULL clause. Now, only those rows with a NULL original_language_id are returned:

Conclusion

In today's blog, we explored what the NULL value means and how to work with this special value in relational databases.

Interested in finding out more about Navicat Premium? You can try both for 14 days completely free of charge for evaluation purposes!

One of the changes in MySQL version 5.0.3 included an increase to the maximum length of VARCHAR fields from 255 to 65,535 characters. That made the VARCHAR type more similar to TEXT than ever before. For those of us who design database tables, choosing between VARCHAR and TEXT now became more challenging as a result. In today's blog, we'll outline the key differences between the two and layout the factors to consider when deciding which data type to go with.

Some Differences Between VARCHAR and TEXT

While both data types share a maximum length of 65,535 characters, there are still a few differences:

• The VAR in VARCHAR means that you can set the max size to anything between 1 and 65,535. TEXT fields have a fixed max size of 65,535 characters.
• A VARCHAR can be part of an index whereas a TEXT field requires you to specify a prefix length, which can be part of an index.
• VARCHAR is stored inline with the table (at least for the MyISAM storage engine), making it potentially faster when the size is reasonable. Of course, how much faster depends on both your data and your hardware. Meanwhile, TEXT is stored off table with the table having a pointer to the location of the actual storage.
• Using a TEXT column in a sort will require the use of a disk-based temporary table, as the MEMORY (HEAP) storage engine.

TEXT Types

Should you require the TEXT type, know that there are actually three flavors; in addition to TEXT, there are also MEDIUMTEXT or LONGTEXT varieties. The latter two are for storing textual content that is longer than 65,535 characters. MEDIUMTEXT stores strings up to 16 MB, and LONGTEXT up to 4 GB! It should go without saying that you should avoid using these larger types unless you have a lot of storage space.

Selecting VARCHAR and TEXT Types in Navicat

In both Navicat for MySQL and Navicat Premium, the Object Designer allows you to create and maintain all sorts of database objects, including Tables, Views, Functions, Indexes, and, of course, columns. Under the Type header, you can select a column's data type simply by selecting it from a drop-down. As you can see, it contains the text, mediumtext, and longtext types:

As for the VARCHAR type, you can also select it from the Type drop-down, but then you should edit the Length value if you want a value other than 255 (the default).

TIP: Since TEXT fields can get quite long, Navicat has a FORM view that give them more room:

Conclusion

The take-away we can draw from all of this is that one should use a VARCHAR field instead of TEXT for columns between 255 and 65k characters if possible. That will lead to potentially less disk reads and less writes.

Interested in finding out more about Navicat for MySQL or Navicat Premium? You can try both for 14 days completely free of charge for evaluation purposes!

A repeating group is a series of fields/attributes that are repeated throughout a database table. It is a common problem faced by organizations both large and small, one that can have several ramifications. For example, the same set of information being present in different areas can cause data redundancy and data inconsistency. Moreover, all of this repeating data can eat up a lot of valuable disk space and take a long of time to search through. The problem of repeating groups can be manageable in small organizations, but for larger organizations, whom must manage huge volumes of information, repeating groups can become a nightmare to deal with.

In today's blog, we'll learn how to identify repeating groups both during design time and in existing databases, as well as how to fix them. Since repeating groups are a phenomenon that can affect any relational database, we'll use Navicat Premium as our database development tool.

An Example of a Repeating Group

The Sakila sample database contains a number of database entities relating to a fictional video rental store. Although its tables have been normalized to Third Normal form (3NF), for the purposes of this tutorial, we'll consider that the film table contains data about actors who appear in each film. Here is a sampling of rows from that table:

You can see that each actor is adding an extra row to the table. Worse still, actors' names are repeated every time that they come up. The problem is that an actor is a separate and distinct entity from a film. Hence, they need to go.

Fixing Repeating Groups

Even though repeating groups are not, strictly speaking, a violation of first normal form (1NF), the process of converting your data from Un-Normalized Form (UNF) to 1NF will eliminate repeating groups. Here are the steps for doing that:

• Identify the repeating groups of data.
• Remove the the repeating group fields to a new table, leaving a copy of the primary key with the table that is left.
• The original primary key will not now be unique so assign a new primary key to the relation using the original primary key as part of a composite key.

Since we've already identified the repeating groups, let's re-design the table so that repeating group fields are omitted and given their own table.

Navicat Premium comes with a built-in Data Modeler. It helps you visually design high-quality conceptual, logical and physical data models. From there, you can generate database structures from a model. The Data Modeler also works in reverse, performing reverse engineering from existing databases. Other features include import from ODBC data sources, generate complex SQL/DDL, and print models to files.

Here is a model showing the existing films_and_actors table:

To separate actors from films, we need to add a new table to host the actor attributes. We should also give it an ID PK field that will link to the same (new FK) field in the original table.

You'll also want to rename tables to reflect that the films table only contains films and actors only stores actor information.

Linking the films and actors Tables

How you link the tables together will depend on their particular relationship to each other. In this case, a film may have zero or more actors, and actors may appear in one or more films. Such a many-to-many relationship will require an intermediary table to link films and actors. It will contain only film and actor IDs. Here is the completed model in the Navicat Modeler:

Conclusion

In today's blog we learned how to identify repeating groups both during design time and in existing databases, as well as how to fix them, using Navicat Premium's powerful Data Modeler. Navicat Premium adds over 100 enhancements and includes several new features to give you more ways that ever to build, manage, and maintain your databases than ever before!

ANSI SQL includes several aggregate functions, which allow you to perform a calculation on a set of values to return their result as a single value. These include Count(), Min(), Max(), Sum() and AVG(), and others. By default, aggregate functions apply to all rows, but you can narrow down the field by applying a WHERE clause to the SELECT statement. Moreover, you can conditionally select certain rows using a few more techniques that we'll explore here today using Navicat Premium. These include the use of a CASE statement as well as the GROUP BY clause. We'll apply these techniques on the AVG() function, but they will work equally well with all aggregate functions.

Using the AVG() Function

The AVG() retrieves the average value of a given expression. If the function does not find a matching row, it returns NULL. We'll run our queries against the Sakila sample database. It was originally developed for MySQL, but has since been ported to most popular DBMSes. Navicat Premium is the ideal database client to use here because it supports everything from MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, to SQLite. Moreover, it's compatible with cloud databases like Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas as well.

The film table stores information about individual films for the fictional Sakila video rental store. Columns include title, description, running time, rental cost, rating, and others.

We can use the AVG() function to determine the average rental cost for ALL films as follows:

Using the CASE Statement

The AVG() function accepts an expression. Hence, it can be a column name, but it can be any valid expression. Therefore, we can apply the AVG() function conditionally by passing a CASE statement to the AVG() function as a parameter. We could determine the average rental_rate for only those films that have a PG rating using a CASE statement like so:

The above query shows the total number of films, films that do not have a PG rating, and the average rental rate for all films as well as those with a PG rating. The CONCAT() and FORMAT() functions are employed to display the rental_rate as currency.

Using the GROUP BY Clause

Another way to apply AVG() to only certain rows is to use GROUP BY. It aggregates the results on the basis of selected column. Hence, grouping results by the rating will list the average rental_rate for each rating:

We could narrow down the rows selected further by using a WHERE and/or HAVING clause(s). Both may be employed separately or in tandem. For instance, the next query selects films with a language_id of 1 (English) whose count by rating total less than 200:

Conclusion

In today's blog we employed the CASE statement and GROUP BY clause to conditionally list film records based on averages.

Queries were executed in Navicat Premium 15. It adds over 100 enhancements and includes several new features to give you more ways that ever to build, manage, and maintain your databases than ever before!

SQL makes selecting all fields in a table quite trivial via the SELECT * (SELECT ALL) clause. Unfortunately, as soon as you omit a column from the list, the SELECT ALL statement goes out the window. Writing out every every column name can quickly become tedious, especially if you happen to be dealing with tables that contain dozens of columns. What if we could select every column but one - selecting by exclusion rather than inclusion? It can be done. In fact there are a couple of ways to do it - one simple, the other, a bit less so. These will be the focus of today's blog.

Method 1: Using The INFORMATION_SCHEMA.COLUMNS table

The INFORMATION_SCHEMA provides access to database metadata, information about the MySQL server such as the name of a database or table, the data type of a column, or access privileges. More specifically, the COLUMNS table provides information about columns in tables, including column names.

The Sakila sample database's film table contains the highest number of columns at thirteen.

Here's how we would use the INFORMATION_SCHEMA.COLUMNS table to fetch all but the original_language_id column:

The GROUP_CONCAT function concatenates all of the column names into a single, comma-delimited string. We can then replace the field to omit with an empty string!

Executing the Query

One small hurdle to overcome is that a MySQL query cannot accept dynamic column names. The solution is to employ a Prepared Statement. Here's the code that sets the @sql variable, prepares the statement, and executes it:

SET @sql = CONCAT('SELECT ',                  (SELECT REPLACE(GROUP_CONCAT(COLUMN_NAME), '<columns_to_omit>,', '')                   FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '<table>'                   AND TABLE_SCHEMA = '<database>'),                   ' FROM <table>');PREPARE stmt1 FROM @sql;EXECUTE stmt1;

Inserting the column, table, and schema information into the query yields the results that we're after:

Method 2: Using Navicat

The main goal of database development and administration tools like Navicat is to increase productivity. As such, Navicat is designed to make your job as quick and easy as possible. To that end, the SQL Editor helps you to code faster thanks to Code Completion and customizable Code Snippets that offer suggestions for keywords and strip the repetition from coding. And if that wasn't enough, Navicat also provides a useful tool called Query Builder for building queries visually. It allows you to create and edit queries with only a cursory knowledge of SQL. While the Query Builder is marketed predominantly to more novice coders, those more proficient in SQL can still benefit from the Query Builder for certain tasks. One such tasks is that of choosing columns.

In the Query Builder, there is a checkbox next to the table name to select all of its columns. If we click on it, we can then simply uncheck the original_language_id field to remove it from the column list:

Clicking the OK button then closes the dialog and adds the SQL code to the editor:

Creating queries using the Query Builder offers a few advantages over writing code by hand:

• it minimizes typos
• it generates formatted SQL that's easy to read

Conclusion

In today's blog, we learned a couple of techniques to select every column in a table but one or two.

Interested in finidng out more about Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

Every so often, we need to rebuild indexes in Oracle, because indexes become fragmented over time. This causes their performance - and by extension - that of your database queries, to degrade. Hence, rebuilding indexes every now and again can be quite beneficial. Having said that, indexes should not be rebuilt to often, because it's a resource intensive task. Worse, as an index is being rebuilt, locks will be placed on the index, preventing anyone from accessing it while the rebuilding occurs. Any queries trying to access this index in order to return the required results will be temporarily blocked, until the rebuild is complete.

In today's blog, we'll learn how often to build indexes and how to determine when an index needs to be rebuilt.

How Often to Rebuild Indexes

As mentioned in the introduction, rebuilding indexes is both a resource intensive and blocking task. Both these considerations make it ideal as an offline activity, to be run when as few users as possible are accessing a database. In general, this means during a scheduled maintenance window.

It is not really feasible to devise a catch-all plan with regard to when and how often to rebuild indexes. These decisions are highly dependent on the type of data you work with, as well as the indexes and queries that are utilized. With that in mind, here are a few guidelines regarding when to rebuild indexes:

• Rebuilding Indexes Nightly

  If your indexes fragment rapidly, and you have a nightly maintenance window that allows you to run the Rebuild Index task, in addition to all your other maintenance tasks, then, by all means, go ahead.

• Weekly, at minimum

  If you can't rebuild indexes on a nightly basis, then, it should be done once a week at a minimum. If you wait much longer than a week, you risk hurting your SQL Server's performance due to the negative impact of wasted empty space and logical fragmentation.

• Alternative scheduling

  If you don't have a maintenance window accommodate this task at least once a week, then you need to pay close attention to how your indexes are faring.

Determining if an Index Needs to Be Rebuilt

In Oracle, you can get an idea of the current state of the index by using the ANALYZE INDEX VALIDATE STRUCTURE command. Here's some sample output from the INDEX_STATS Table:

  SQL> ANALYZE INDEX IDX_GAM_ACCT VALIDATE STRUCTURE;  Statement processed.     SQL> SELECT name, height,lf_rows,lf_blks,del_lf_rows FROM INDEX_STATS;     NAME          HEIGHT      LF_ROWS    LF_BLKS    DEL_LF_ROW  ------------- ----------- ---------- ---------- ----------  DX_GAM_ACCT        2          1          3          6      1 row selected. 

There are two rules of thumb to help determine if the index needs to be rebuilt:

1. If the index has height greater than four, rebuild the index.
2. The deleted leaf rows should be less than 20%.

Rebuilding an Index

In Oracle, you can use the Alter Index Rebuild command to rebuild indexes. It rebuilds a spatial index or a specified partition of a partitioned index.

ALTER INDEX REBUILD command has a few forms:

ALTER INDEX [schema.]index REBUILD       [PARAMETERS ('rebuild_params [physical_storage_params]' ) ]      [{ NOPARALLEL | PARALLEL [ integer ] }] ;

OR

ALTER INDEX [schema.]index REBUILD ONLINE      [PARAMETERS ('rebuild_params [physical_storage_params]' ) ]      [{ NOPARALLEL | PARALLEL [ integer ] }] ;

OR

ALTER INDEX [schema.]index REBUILD PARTITION partition       [PARAMETERS ('rebuild_params [physical_storage_params]' ) ];

Handling Unusable Indexes

Navicat for Oracle's Maintain Index facilities has a couple of useful options for handling unusable indexes:

• Rebuild

  To re-create an existing index or one of its partitions or subpartitions. If the index is marked unusable, then a successful rebuild will mark it usable.

• Make Unusable

  To make the index unusable. An unusable index must be rebuilt, or dropped and re-created, before it can be used.

Conclusion

In today's blog, we learned how often to build indexes and how to determine when an index needs to be rebuilt.

If you'd like to learn more about Navicat for Oracle, visit the product page.

The number of images in web applications has been growing steadily in recent years. There is also a need to distinguish between images of different sizes, like thumbnails, web display images, and the like. For example, one application that I recently developed shows news items where each item has a thumbnail and main article image. Another app shows company logos in small and large sizes.

Most of the time, images can be stored on the web server and then referenced using the URL. That only requires storing the path string in the database, rather than the image itself. However, there are times that this is not feasible, such as where the app has insufficient rights on the filesystem. In those cases, you can store images directly in the database and then load them using application code.

Navicat development and administration tools provide excellent support for image management. In today's blog, we'll learn how Navicat makes storing images a simple process. For the purposes of demonstration, I'll be using Navicat Premium against a MySQL 8 database, but the same procedure would apply to other relational databases as well.

Designing the Table

In MySQL, the preferred data type for image storage is BLOB. However, there are actually three flavors of BLOB. The one you choose depends on the size of the images that you will be storing. If in doubt, go to the larger capacity BLOB! Here are the three BLOB types:

• BLOB: Can handle up to 65,535 bytes of data.
• MEDIUMBLOB: The maximum length supported is 16,777,215 bytes.
• LONGBLOB: Stores up to 4,294,967,295 bytes of data.

With that in mind, here's a table definition that would be well suited to thumbnail images, but not much larger:

Besides the image itself, you may find it useful to store other information about the image, such as an ID, name, description, size, type (JPEG, GIF, BITMAP, etc.), category, and so on.

Loading Images into the images Table

Using Navicat, there's no need to write SQL code to load images. Instead, you can use the standard File Browser to locate and insert image files.

Whenever you view table contents in either Grid of Form view, you can select how you want Navicat to treat data from the data type drop-down:

Choosing Image from the drop-down adds an image preview pane underneath the table/row contents:

On the left of the file preview, you'll find three icons: Load, Save to Disk, and Clear. To load an image, simply click the Load icon and select the image using the operating system's standard File Browser dialog. Once inserted, the image - as well as its size in bytes - will appear in the preview pane:

Note that the above image requires a MEDIUMBLOB as its size exceeds 65,535 bytes!

Conclusion

In today's blog, we learned how to store images in a MySQL 8 database using Navicat Premium 15.

Now is the perfect time to purchase Navicat Premium as version 15 adds over 100 enhancements and includes several new features to give you more ways that ever to build, manage, and maintain your databases than ever before!

Perhaps you've heard that version 15 of Navicat's flagship product, Navicat Premium, was officially released on November 25th. It comes packed with numerous improvements and features to address all of your database development and administration needs. In addition to over 100 enhancements, Navicat includes several new features to give you more ways that ever to build, manage, and maintain your databases. In the last blog, we explored the Data Visualization feature. Today, we'll be taking a look at other improvements, including Data Transfer, Query Builder, Data Modeler and more!

Data Transfer

Although Navicat already supports the transfer of database objects from one database and/or schema to another, or to an sql file, version 15 brings a whole new experience along with a number of new functions to the Data Transfer utility. The new design includes an intuitive interface for customizing the fields and specifying the number of rows you wish to transfer. For instance, you can choose specific fields to transfer and even change the field names. You can also limit the number of rows to transfer using custom filters.

Perhaps even more importantly, Navicat 15's Data Transfer is designed to quickly migrate massive amounts of data - quick enough to accomplish the most complex transfers faster than ever before.

Query Builder

Navicat 15 introduces a whole new approach to writing SQL via the new Query Builder. Whereas version 12 showed all of the query syntax in one statement, version 15 breaks it down into clauses: Select, From, Where, Group By, Having, Order By, and Limit. Moreover, the resulting SQL statement is displayed in the right pane so that you can make modifications, should you make any syntax errors.

In addition to the new UI, Navicat 15 now supports subqueries to further fine-tune your query results.

Data Modeling

Navicat 15 includes the new and improved Data Modeler 3.0. The updated Data Modeler introduces a new mechanism for Database Synchronization, starting with a more intuitive way to visually compare and identify the differences between the model and database. It shows a side-by-side DDL comparison that makes locating all the object differences a snap.

Once you're ready, you can choose and reorder your synchronization scripts to update the destination database.

Dark Mode

A lot of people find that bright themes, including the default white of Windows, make their eyes tired after many hours in front of the screen. For that reason, dark themes have become increasing prevalent in recent years. Navicat 15 gives you the choice between the traditional Windows (light) theme and the new Dark Theme. In contrast to the mostly whites of Windows themes, dark theme displays dark surfaces across the majority of the UI.

You'll find it on the General options screen.

Native Linux support

Whereas previous versions of Navicat Modeler required Wine to run on Linux systems, version 15 is a true Linux application, so users can now enjoy a UI that is more in line with other Linux apps!

Conclusion

In today's blog, we got a preview of Navicat Premium 15's most top new features and improvements. Why not download it and try it out for yourself? Here's a free 14 day trial!

Welcome to part 3 of this series on full-text indexing and searching in MySQL. In Part 1, we saw how MySQL provides full-text search capability via FULLTEXT indexing along with the three following distinct types of full-text searches:

• Natural Language Full-Text Searches
• Boolean Full-Text searches
• Query expansion searches

In Part 2, I described how to perform Natural Language full-text searches in Navicat for MySQL. Today's blog follows where part 2 left off and covers the next type of full-text searching: Boolean Full-Text searches.

Boolean Mode Described

Boolean mode is a more word-driven than the natural language search. As such, Boolean full-text search supports very complex queries that include Boolean operators. For experienced users, Boolean full-text searching offers a means to perform some very advanced searches.

Here's how it works:

To perform a full-text search in the Boolean mode, you include the IN BOOLEAN MODE modifier in the AGAINST expression. Recall that, in the last installment, we added a full-text index to the film table of the Sakila sample database so that we could perform full-text searches on the description field. Here's an example that returns all films whose descriptions contain the word "Butler":

Some More Complex Examples

The above search is simple enough to not require full-text searching. It gets a lot more interesting once you start doing things like excluding matches that contain certain keywords. For instance, we can find films whose descriptions contain the word "Butler" that are not documentaries by preceding the word "Documentary" with the exclude Boolean operator ( - ):

That returns 61 rows, compared to 73 for our previous query.

Multi-word Matching

We can also search for rows whose description match multiple words using the ( + ) include operator. Prefixing a word with it tells the search engine to only match rows that contain that word. That becomes an important distinction when there are multiple words, such as "+Butler Hunter Waitress". In that case, all rows whose description contains the word "Butler" and either of the other two words will be returned:

Contrast the above results with those produced by a query with both the words "Butler" and "Hunter" prefixed with the ( + ) include operator:

Now, matching rows must contain both "Butler" and "Hunter" but not necessarily "Waitress".

A Quick Word on Relevancy Rankings

Full-text searches rank results differently for InnoDB than MyISAM because InnoDB full-text search is modeled on the Sphinx full-text search engine, and the algorithms used are based on BM25 and TF-IDF ranking algorithms.

Some operators affect ranking so that we can further fine-tune results. For example, we can search for rows that contain the word "Butler" but rank the row lower if it contains the words "Hunter" or "Waitress":

Conclusion

In today's blog, we learned how to perform Boolean full-text searches using Navicat for MySQL. Interested in finidng out more about Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

For a full listing of the boolean full-text operators, take a look at the official MySQL docs.

November 25 is the official launching date for Navicat Premium 15. Currently, version 15 packs a wallop of new features and improvements, most notably in data transfers, the SQL Builder, and modeling. It also adds Data Visualization, Dark Mode and native Linux support. In today's blog we'll learn how the new Data Visualization feature helps us turn our database into visuals that provide valuable insights into our data through a wide variety of charts and graphs.

Work spaces, Data Sources, and Charts

Data visualization is the graphical representation of information and data using visual components like charts, graphs, and maps. Data visualization tools provide an accessible way to see and understand trends, outliers, and other patterns in our data.

In Navicat, data visualizations are organized in a hierarchy of Workspace, Data Sources, and Charts.

Clicking the Charts button displays the Workspace buttons in the Objects toolbar. From there, clicking on New Workspace opens the Charts Workspace window, which is all of the action happens!

Whereas typical charts are based on a snapshot of data, data visualization can be applied to live data and refreshed at any time.

In a graphical interface, a workspace is a grouping of related objects to help manage them in one place. In Navicat, it's where you'll find all of your Data Sources, Charts, and Dashboards (more on those in a bit).

A Data Source can be based on any valid database connection or even on multiple connections.

Once a Data Source's objects are available, you can drag database objects such as tables, views and queries into the main window and fetch data from them. Navicat will auto-detect relationships between database objects.

Data may be live or based on an archive. In Archive mode, Navicat takes a snapshot of the current data and bases charts on it.

Charts

The main tool of data visualization is charts. That's why Navicat 15 includes all of the major chart types that you need to explore your data. These include (just to name a few):

• Bar
• Stacked bar
• Line
• Area
• Pie
• Donought
• Scatter

Charts are completely customizable in terms of fonts, text placement, colors, opacity, formatting, filtering, and many more. Here's an example that I created in about five minutes!

Dashboards

In this age of Big Data, data dashboards are indispensable to aggregate huge amounts of data from disparate sources into a cohesive and digestible whole. A dashboard displays all this data in the form of tables, line charts, bar charts and gauges. A data dashboard is the most efficient way to track multiple data sources because it provides a central location for businesses to monitor and analyze performance.

In Navicat, a Dashboard is where you can group your charts along with images, shapes, and text to create slideshow presentations for targeted groups, such as developers, data analysts, and management.

Conclusion

In today's blog we explored Navicat 15's new Data Visualization feature. We'll check out some other new features and improvements next week. In the meantime, feel free to download it and give it a whirl!

In Part 1, we saw how MySQL provides full-text search capability via FULLTEXT indexing along with three distinct types of full-text searches. In today's blog, we'll learn how to perform Natural Language full-text searches in Navicat for MySQL.

Natural Language full-text Searching Defined

The idea behind natural language full-text searching is to seek documents (rows) that are relevant to a natural human language query such as "How natural language full-text searches work?". If you've ever used an Internet search engine like Google, this is exactly how it works!

Relevance is ascertained using a positive floating-point number of zero plus, where zero means that there is no similarity. Relevance can be based on various factors, including the number of words in the document, the number of unique words in the document, the total number of words in the collection, and the number of documents (rows) that contain a particular word.

A MySQL Natural Language Full-text Search Example

In MySQL, natural-language full-text searches are performed using the MATCH() and AGAINST() functions. The MATCH() function specifies the column where you want to search, whereas the AGAINST() function determines the search expression to be used.

The Sakila sample database represents a fictional DVD rental store. The film table contains pertinent information about each film in the store's collection. Columns include the film's title, release year, running length, and a description. Here is a sample row in Navicat's Form View. It allows you to view, update, insert, or delete data as a form, in which the current record is displayed as a field name and its value:

Indexing the description Column

In order to search the description field in full-text mode, we first have to create a full-text index on the table. We can easily do that in Navicat as follows:

• Open the description table in the Table Designer
• Select the Indexes tab.
• Click the Add Indexes button.
• Let's call our new index "idx_description".
• In the fields textbox, select the description column.
• Select FULLTEXT from the Index Type drop-down:
  
• Leave the Index method blank as it is not required for a FULLTEXT index.
• Finally, click the Save button to create the index.

Query Time!

Let's open up the Query Editor and write a query that will lookup rows whose description contains the phrase "Database Administrator". Navicat can help us compose our query by suggesting the names of fields even the functions we require:

Here is the final query and results for the phrase "Database Administrator". There are a surprisingly high number of films about DBAs!

Viewing the Scores

As explained above, relevance is ascertained using a positive floating-point number of zero plus, where zero means that there is no similarity. We can view the score of each record by adding the MATCH() and AGAINST() functions to the column list, for example:

This would help us to determine a cut-off point for the closest matches.

Conclusion

In today's blog, we learned how to perform Natural Language full-text searches in Navicat for MySQL.

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes!

Full-text Search, or FTS, is one of the techniques employed by search engines to find results in their database(s). You too can harness the power of FTS to search for patterns that are too complex for the Like operator. In today's blog, we'll learn how full-text searching is implemented in MySQL. In part 2, we'll try our hand at some queries using Navicat for MySQL as our database client.

Full-text Searching Explained

The purpose of FTS is to fetch documents that only loosely match search criteria against textual data. Hence, searching for "cars and trucks", would return results which contain the words separately, as in just "cars" or "trucks", or that contain the words in a different order ("trucks and cars"), or contain variants of the search terms, e.g. "car" and "truck". This allows businesses to guess at what the user is searching for and return more relevant results in a faster time.

Database Management Systems (DBMS) like MySQL do allow quasi text lookups using LIKE expressions. There are however some drawbacks to the Like clause:

• They tend to under-perform on large datasets.
• They're also limited to matching the user's input exactly, which means a query might yield no results even if there are in fact records with relevant information.

Full-Text Searching in MySQL

In order to perform full-text searches in MySQL, you have to add a FULLTEXT index to fields that will support full-text searching. Moreover, full-text indexes can be used only with MyISAM and InnoDB tables. Finally, note that full-text indexes can be created only for CHAR, VARCHAR, or TEXT columns. A FULLTEXT index definition can be given either in the CREATE TABLE statement, or added later using the ALTER TABLE or CREATE INDEX commands. A tip for large data sets: it's much faster to load your data into a table that does not have a FULLTEXT index and then create the index after the data has been loaded, rather than create the FULLTEXT index first and then load the data.

There are three distinct types of full-text searches:

• Natural Language Full-Text Searches
• Boolean Full-Text searches
• Query expansion searches

We'll cover each of these in turn as we go through the list in part 2.

A Basic Example

In the Sakila Sample Database, the film table contains information about each movie in the store's film collection, including its title, running time, and description. We can take a look at a film table record in detail using the Form View. Available in Full Version, the Form View allows us to view, update, insert, or delete data as a form, where the current record is displayed in full detail. There's also navigation bar for switching between records quickly.

We can add full-text searching capability on the description column by adding the FULLTEXT index to it. We could issue either the ALTER TABLE or CREATE INDEX commands, but it Navicat, there's an easier way! The Table Designer contains a number of tabs pertaining to different table objects. These include, column definitions, indexes, foreign keys, triggers, options, and more. We can add a full-text index on the description field by selecting it using the Field Selector dialog, and the choosing FULLTEXT from the Index Type drop-down:

Be sure to leave the Index method blank.

Click the Save button to create the new index:

Conclusion

Now that we've prepped the database for full-text searching, we'll learn how to use Full-Text Search Functions in part 2.

Interested in Navicat for MySQL? You can try it for 14 days completely free of charge for evaluation purposes.

Navicat Monitor, the agentless database server instance monitoring tool for MySQL and MariaDB recently added support for SQL Server. Hence, it can now monitor database process and system resources for locally hosted SQL Server instances as well as those provided via Amazon Web Services (AWS). Today's blog will provide a quick guide for connecting to an SQL Server instance in order to monitor its performance using Navicat Monitor 2.0.

Connecting to an SQL Server Instance

To start monitoring an SQL Server instance, you have to configure the connection via the New Instance button. You'll see the new SQL Server item in the list. It will allow you to connect to both locally hosted SQL Server instances as well as those provided via Amazon Web Services (AWS):

The New Instance dialog states the database type now, i.e "New SQL Server Instance", rather than the generic "New Instance" title of previous Navicat Monitor versions. While the fields and options are laid out using a very similar layout between databases, closer inspection reveals that they are tailored to the specific type of database chosen, in this case, SQL Server:

Note that the Host Name defaults to "localhost". You can change it to your instance server name, and that includes that of an AWS instance, i.e. "sample-instance.abc2defghije.us-west-2.rds.amazonaws.com". In that way, both local and remote connections are fully supported.

For more information on connecting to your database instance(s), take a look at the Configure an Instance in Navicat Monitor for MySQL/MariaDB blog article.

The Dashboard

To accommodate SQL Servers, there is a new SQL Server Filter in the Filters bar. Filters allow you to only show database instances of the selected type(s):

All of the Dashboard functionality and features remain unchanged. You can still sort by Alert Severity, Name, or Instance Type, and instance cards may still be dragged from one Group to another using the handlebar.

You'll know that you can drag the card when the mouse pointer turns into an arrow:

You can choose exactly which metrics are displayed in the Instance Card by clicking on CARD DESIGN. The Card Design dialog metrics are now divided into two columns: one for MySQL/MariaDB and one for SQL Server:

Conclusion

In today's blog, we learned how the latest version of Navicat Monitor has evolved to support SQL Server.

Navicat Monitor version 2.0 is now available for sales at Navicat Online Store and is priced at US$499/token (commercial) and US$199/token (non-commercial). 1 token is needed to unlock 1 MySQL Server / 1 MariaDB Server / 1 SQL Server.

For more details on Navicat Monitor and all its features, please visit: https://www.navicat.com/en/discover-navicat-monitor.

You can download a fully functional 14-day free trial at https://www.navicat.com/en/download/navicat-monitor. Give it a try!

SQL queries often return more than one row of data from the database server. Relational databases provide cursors as a means for iterating over each row of the results set. Does that mean that MongoDB users are out of luck? As it turns out, MongoDB's db.collection.find() function returns a cursor. In MongoDB, cursors themselves provide additional functionality for processing individual rows. In today's blog, we'll learn how to work with MongoDB cursors in Navicat for MongoDB.

A Simple Iteration Example

Executing a query via the db.collection.find() function returns a pointer to the collection of documents returned, which is a cursor. The default behavior of a cursor is to allow an automatic iteration across the results of the query. However, developers can explicitly go through the items returned in the cursor object. One way to do that, is to use the forEach() cursor method.

In Navicat, it's easy to use the find() method. For instance, you can drag a pre-defined snippet into the Query Editor:

Alternatively, you can use the Find Builder. Just select the Collection or View to fetch all the documents:

From there, you can chain the forEach() directly to the results cursor. In the following example, the three documents in our collection are printed to the console. You can view the output in the Print Output tab:

A Loop of a Different Kind

Like all JavaScript objects, a cursor may be stored in a variable for later use. Other JavaScript constructs like while loops are also fully supported thanks to the cursor.hasNext() and cursor.next() methods. As we see in this example, hasNext() informs the loop tester whether or not there is another document to iterate over; next() returns said document.

The printjson() helper method is a convenience method that replaces print(tojson()). It basically outputs documents exactly as they are stored in the database, except that they are represented as JSON rather than BSON, the binary equivalent.

Updating Data

Recall that db.collection.find() returns a pointer to the collection of documents returned. As such, document fields are fully editable. Hence, we can invoke the forEach() method to update documents that match the specified criteria. Here's a function that updates documents where the name equals "Tom Smith":

In our case, there is only one matching document, but, in theory, there could be many.

Data Transformation

One type of data transformation is to simplify a dataset's structure in order to make it easier to consume. To do that, we can use the map() function. This example applies the split() function to the name field to break it down into its first and last constituents. Then, the reverse() and join() methods convert "Tom Smithers" to "Smithers, Tom":

Conclusion

In today's blog, we saw how cursors returned by the db.collection.find() method come packed with all sorts of methods to iterate over documents for the purposes of printing, modifying, deleting, or transforming their contents. Interested in taking Navicat for MongoDB for a spin? You can download a free trial here!

Being the dedicated database developer and/or administrator that you are, I don't need to remind you that the rigorous application of the principles of sound database design via data modeling is one of the cornerstones of data management. To that end, the emergence of specialized software such as Navicat Data Modeler have made the process much easier to accomplish.

It's been around for some time now, so I have written about it a few times, first in a Database Journal article, and then on the Navicat Blog. Now that version 3.0 is in beta, let's explore what it brings to the table, in particular, these 3 exciting new features:

• Database Synchronization (just like the one found in Navicat Database clients!)
• Dark mode UI option
• Native Linux support

Database Synchronization

One of Navicat Modeler's greatest strengths is its ability to generate DDL (Data definition language) statements to create database objects from a Diagram. Version 3 goes one step further by providing Structure Synchronization that brings the schema of two databases in sync. This is a very useful feature for bringing one or more databases up-to-date with one to which you've applied design changes.

A wizard guides you through the process:

• The first screen is where we choose the source model and target database:
  
  


• The second screen presents a graphical comparison of databases objects as well as the DDL statements and deployment scripts:
  
  


• The next screen shows all of the DDL statements that will be applied to the target database in order to synchronize its structure with the model:
  
  
  At this point you can also set deployment options and even edit the script to suit your exact requirements. From there, you can recompare the target database to your changes, taking into account your manual script edits.


• A detailed message log and process statistics are provided for the script's execution:
  
  

Dark Mode UI Option

A dark theme displays dark surfaces across the majority of a UI, as opposed to the mostly white surfaces of Windows themes. Dark themes have become increasingly popular in recent years, for several good reasons!

• Dark themes reduce the luminance emitted by device screens, while still meeting minimum color contrast ratios.
• They help improve visual ergonomics by reducing eye strain, adjusting brightness to current lighting conditions, and facilitating screen use in dark environments all while conserving battery power.
• Devices with OLED screens benefit from the ability to turn off black pixels at any time of day.

You can choose between the traditional Windows (light) theme and the new Dark Theme on the General options screen:

After you restart the application, you'll see your new UI theme:

Native Linux support

Whereas previous versions of Navicat Modeler required Wine to run on Linux systems. Wine is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, including Linux. Navicat Modeler 3.0 is s true Linux application, so users can now enjoy a UI that better matches those of the Linux system!

Conclusion

Navicat Data Modeler 3.0 adds several exciting features to an already stellar modeling tool. Interested in trying out Navicat Data Modeler 3.0? You can download it here!

CASE is a Control Flow statement that acts a lot like an IF-THEN-ELSE statement to choose a value based on the data. The CASE statement goes through conditions and returns a value when the first condition is met. So, once a condition is true, it will short circuit, thereby ignoring later clauses, and return the result. As we'll see in today's blog, it can be used to test for conditions as well as discrete values.

Basic Syntax

The CASE statement comes in two flavors: the first evaluates one or more conditions and returns the result for the first condition that is true. If no condition is true, the result after ELSE is returned, or NULL if there is no ELSE part:

CASE
    WHEN condition1 THEN result1
    WHEN condition2 THEN result2
    WHEN conditionN THEN resultN
    ELSE result
END;

The second CASE syntax returns the result for the first value=compare_value comparison that is true. If no comparison is true, the result after ELSE is returned, or NULL if there is no ELSE part:

CASE compare_value
    WHEN condition1 THEN result1
    WHEN condition2 THEN result2
    WHEN conditionN THEN resultN
    ELSE result
END;

Some Examples

To try out the CASE statement, we'll be writing some queries against the Sakila Sample Database using Navicat Premium. It's a powerful database development and administration tool that can simultaneously connect to most popular databases, including MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases. It's also compatible with many cloud databases like Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas.

Syntax 1

Here's a query that selects a list of movie titles, along with their release year and rental price:

We'll add a column that splits rental prices into three categories: "discount", "regular", and "premium". The price ranges are:

• less than $2.99
• between $2.99 and $4.99
• $4.99 and up

To help with the CASE statement, Navicat provides Code Snippets that you can simply drag & and drop into the SQL editor. Although you can create your own, Navicat comes with many standard SQL statements, including DDL and flow control statements. In fact, you'll find the CASE statement at the top of the Flow Control list:

After you place the code snippet into the editor, editable sections are color coded to help identify them. You can use the Tab key to move from one to the next.

Since the statements are generic in nature, you may have to modify it slightly to suit your particular database type. Here is the complete CASE statement and query for MySQL:

Syntax 2

The 2nd CASE syntax is ideal for testing discrete values against two or more conditions. For example, we could use it to add a target audience column based on the film rating:

Conclusion

In today's blog we learned how the SQL CASE Statement can be employed to choose a value based on the underlying data. Example SQL statements were written in Navicat Premium. It helps you code fast with Code Completion and customizable Code Snippets by getting suggestions for keywords and stripping the repetition from coding. You can try it for 14 days completely free of charge for evaluation purposes.

Navicat Data Modeler 3.0 Highlights:

• a new design for Synchronize to Database feature which delivers a full picture of database differences, and generate scripts to update the destination database;
• set dark theme for your default viewing preference to protect your eyes from the traditionally blinding whiteness of computer;
• native Linux support is added to provide a UI that better matches with user experiences of the Linux system.
• and more.

Try it now - https://www.navicat.com/en/download/navicat-data-modeler-3-beta

There are some very good reasons why data validation is best performed at the database level rather than at the application level. For instance, the same data source may be accessed by multiple applications. Therefore, you can rely on the data being consistent and valid without having to depend on validation logic being applied on the application side, which might not be consistent across different implementations. Moreover, triggers are ideal for validation because they can be executed before data is inserted or updated. Triggers can also can also prevent a database transaction from being applied while providing an error message.

In today's blog, we'll write a trigger in Navicat Premium that will validate insert operations on a MySQL database table.

Designing the Trigger

We'll be working with the Sakila sample database. It contains a number of related tables themed around a fictional video rental store. Here they are in the Navicat Premium navigation pane:

We'll be adding our trigger to the film table. If you open it in the Designer, you'll see that there are several tabs there:

Clicking the Triggers tab reveals that there are already a few triggers defined for that table. For instance, the ins_film trigger copies film information to the film_text table on data inserts. This is a common task allocated to triggers.

Now we'll add a trigger that will make sure that foreign films are inserted with the original_language_id.

A film's language is actually stored in the language lookup table:

Any language_id other than 1 should have an original_language_id as well. Our trigger will check for a value in the original_language_id column.

• In the Design View of the film table, select the Triggers tab and click on the Add Trigger button.
  That will add a new row in the triggers table.
• Assign a name of "ins_validate_language", select BEFORE from the Fires drop-down, and click on the Insert checkbox.
• Here's the trigger Body:
  BEGIN
    IF NEW.language_id != 1 AND NEW.original_language_id IS NULL
    THEN
      SIGNAL SQLSTATE '45000'
        SET MESSAGE_TEXT = 'Original language id is required for Foreign language films.';
    END IF;
  END
  
  Here is our Trigger with all of the fields filled in:
  
• Click the Save button to create the trigger.

Testing the Trigger

Now it's time to verify that our trigger works as expected. To test it, let's add a new row to the film table with a foreign language_id.

• Open the film table in the Editor.
• Navigate to the last row.
• Select the Form View and click on the Plus (+) button to add a new row:
  
• In the form, you only need to enter a title and language_id; all other fields have default values or are not required.
• When you click the Add (checkmark) button, you should see our error message:
  

Conclusion

Triggers are ideal for validation because they can be executed before data is inserted or updated. We saw how a trigger can be employed for validation purposes by writing a trigger in Navicat Premium.

The SQL LIMIT clause constrains the number of rows returned by a SELECT statement. For Microsoft databases like SQL Server or MSAccess, you can use the SELECT TOP statement to limit your results, which is Microsoft's proprietary equivalent to the SELECT LIMIT statement. However, for most relational databases (DBMSes), including MySQL/MariaDB, PostgreSQL, and Oracle, the SQL LIMIT clause can solve several problems. In today's blog, we'll explore a few of these, using Navicat for PostgreSQL.

Keeping Result Sets Manageable

In many production and test databases, table sizes routinely reach millions of rows and have dozens of columns. For that reason, it's never a good idea to run SELECT * queries against your database(s). Keeping the results down to one hundred or one thousand helps keep result sets down to a size that's more easily digestible.

Navicat development and administration tools automatically limit result sets by default in order to prevent straining your database server(s). You can see it in action when you open a table. At the bottom of the application window, the SQL that Navicat executed to fetch the table rows is displayed. It ends with the "LIMIT 1000 OFFSET 0", which means that only the first 1000 records are displayed.

You can change the default number of records to show or turn off limiting entirely on the RECORDS Options screen:

Top N Queries

As the name implies, top-N queries are those that attempt to find the top number of records from a result set. This could be top 1, top 3, top 5, top 10, or top [any] number. Some common examples are:

• Find the top 10 highest paid employees
• Find the top 20 most profitable customers
• Find the top 3 users on the system

These queries are hard to do with just an ORDER BY and WHERE clause alone, but not using the LIMIT clause. Here's an example:

Top 5 Unique Job IDs

Let's say that we wanted to find the top unique Job IDs in a table. Here's a query that does just that:

The DISTINCT keyword makes sure that duplicate IDs are removed from the results.

Closest Rows to a Given Date

It is possible to locate rows closest to a given date using LIMIT. You just have to compare row dates to the given date, order the results, and limit the results to the number of rows that you'd like to see. Here's a query that returns rows whose creation_date is greater than '2018-01-01':

In this case, 2018-01-02 was the closest later date.

Bottom N Queries

The corollary of top N queries are bottom N queries. These are queries that attempt to find the bottom number of records from a result set. We can convert our Top queries into their Bottom equivalents quite easily!

Bottom 5 Unique Job IDs

To return the bottom 5 unique job IDs, all you need to do is remove the DESC modifier in the ORDER BY clause. That will order records in ascending (ASC) order, as is default:

Closest Rows below a Given Date

Locating the closest rows before a given date is likewise fairly easy. We just need to change the greater than '>' operator to less than '<' and reorder results in descending (DESC) order:

Conclusion

In today's blog, we explored a few uses for the LIMIT clause, using Navicat for PostgreSQL. Like to give Navicat for PostgreSQL a try? You can evaluate it for 14 days completely free of charge!

There are times when you need to fetch related data that reside in the same table. For that, a special kind of join is required called a self join. In today's blog, we'll learn how to write a query that includes a self join using Navicat Premium as the database client.

Syntax

The basic syntax of SELF JOIN is as follows:

SELECT a.column_name, b.column_name...
FROM table1 a, table1 b
WHERE a.common_field = b.common_field;

In addition to the linking on common fields, the WHERE clause could contain other expressions based on your specific requirements.

An Example

In the Sakila Sample Database, there is a customer table that contains customer-related information such as their name, email, and address. Here is are the columns in the Navicat Table Desginer:

We can use a self join to retrieve all customers whose last name matches the first name of another customer. We achieve this by assigning aliases to the customer table. The aliases allow us to join the table to itself because they give the table two unique names, which means that we can query the table as though it were two different tables. These are then joined on the last_name and first_name fields:

SELECT
        c1.customer_id as customer_1_id,
        c1.first_name as customer_1_first_name,
        c1.last_name as customer_1_last_name,
  c2.customer_id as customer_2_id,
        c2.first_name as customer_2_first_name,
        c2.last_name
FROM customer c1,
     customer c2
WHERE c1.last_name = c2.first_name
ORDER BY c1.last_name;

Navicat's auto-complete feature is really useful when writing your queries because it helps avoid typos and having to guess at column names. For this reason, it's especially useful for selecting fields:

Executing the query generates the following results:

Using an INNER JOIN

Another way to link a table to itself is to use an INNER JOIN. If you're not sure how to do that, Navicat can help! It provides a useful tool called Query Builder for building queries visually. It allows you to create and edit queries without much knowledge of SQL. The database objects are displayed in left pane. Whereas in the right pane, it is divided into two portions: the upper Diagram Design pane, and the lower Syntax pane.

We can simply drag the last_name field of the first table alias to the first_name of the second table alias and the Query Builder will generate the JOIN for us!

Here's the generated SQL statement:

SELECT
c1.customer_id AS customer_1_id,
c1.first_name AS customer_1_first_name,
c1.last_name AS customer_1_last_name,
c2.customer_id AS customer_2_id,
c2.first_name AS customer_2_first_name,
c2.last_name
FROM
customer AS c1
INNER JOIN customer AS c2 ON c1.last_name = c2.first_name
ORDER BY
customer_1_last_name ASC
;

Conclusion

In today's blog, we learned how to write a query that includes a self join using Navicat Premium. Give Navicat Premium a try. You can evaluate it for 14 days completely free of charge!

Even if your company is still relatively small, it may already be in the process of outgrowing the database that you started with. As this happens, new applications will interface with a larger and more powerful database. Meanwhile, the original database will still play a (reduced) role in business activities. Eventually, you will need to manage a variety of databases, each with its own features, specialized syntax, and connection protocols.

Managing multiple databases either necessitates that you employ multiple client applications or find one that can accommodate all of the databases that you use. One such tool is Navicat Premium. Not only does it support most of the major Database Management Systems (DBMSes), but it is one of the few tools that can simultaneously connect to all of them at once!

In today's blog, we will examine some of the challenges of managing multiple databases and provide some practical examples of how to overcome them using Navicat Premium.

Connecting to Multiple Databases

Establishing connections to multiple databases is not a trivial task because each database product implements its own connection parameters. For instance, some databases require a default database, whereas others do not. Navicat smooths out these differences by providing a consistent Connection dialog for each database type, with only a few minor variations between screens. Here's a comparison of the New Connection dialog for MySQL on Windows and SQL Server on macOS:

For more information on connecting to multiple databases, please see this recent blog.

Querying across Multiple Databases

When it comes to SQL queries, most DBMSes support a standardized set of SQL statements and functions. Beyond that, many database vendors try to set their product(s) apart by including an additional set of extended SQL features. For example, a pivot table is a table of statistics that summarizes the data of a more extensive table (such as from a database, spreadsheet, or business intelligence program). This summary might include sums, averages, or other statistics, which the pivot table groups together in a meaningful way.

Database support for pivot tables varies greatly across DBMSes, as described below:

• PostgreSQL, an object-relational database management system, allows the creation of pivot tables using the tablefunc module.
• MariaDB, a MySQL fork, allows pivot tables using the CONNECT storage engine.
• Microsoft Access supports pivot queries under the name "crosstab" query.
• Oracle database and SQL Server support the PIVOT operation.
• Some popular databases that do not directly support pivot functionality, such as SQLite can usually simulate pivot functionality using embedded functions, dynamic SQL or subqueries.

In Navicat, you can query multiple databases with one statement, as long as you can join the various tables on a common field, and that the syntax is supported by all of the databases included in the query:

Here a blog all about querying multiple databases.

Conclusion

In today's blog, we examined some of the challenges of managing multiple databases and reviewed some practical examples of how to overcome them using Navicat Premium.

Navicat Premium is available for the Windows, macOS, and Linux operating systems and supports MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases. It's also compatible with cloud databases like Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud and MongoDB Atlas. Try it today!

A short time ago, we explored Some SELECT Queries You Must Know. These included determining the lowest and highest value for a column, as well as grouping results by category. Today's blog presents a couple more queries, along with a tip to make your queries almost write themselves!

Get All User Created Tables

These include tables that are part of user created databases, that is to say, that are not part of system database schemas. The exact syntax varies by vendor, but here are a couple of examples to give you the idea.

In SQL Server, this simple one-liner will do the job:

SELECT NAME FROM sys.objects WHERE TYPE='U'

MySQL's syntax is a bit more wordy because you have to specify the system databases in order to omit their tables:

SELECT * from information_schema.tables
WHERE table_schema not in ('information_schema', 'mysql', 'performance_schema')
ORDER BY table_schema, table_name;

So why would you want to query user tables? In addition to table names, the MySQL query returns a great deal of useful information about each table, including the number of rows, the storage engine, their size, the last auto_increment value, and more!

If you only want the table names in MySQL, that's easily done. You can narrow down the list using the WHERE clause, or, you can issue the following command:

SHOW FULL TABLES IN [database_name] WHERE TABLE_TYPE LIKE 'BASE TABLE';

Get All View Names

Again, the exact syntax varies by vendor, but a couple of examples will provide a good starting point.

Here's the SQL Server syntax:

SELECT * FROM sys.views

In MySQL we can narrow down the list to views by limiting the TABLE_TYPE to 'VIEW'. We still have to exclude the sys database as it contains a number of views:

SELECT * FROM information_schema.`TABLES`
WHERE TABLE_TYPE = 'VIEW'
AND table_schema != 'sys';

Here are the results in Navicat Premium:

Looking for views of a specific database? You can just change the WHERE clause to:

AND TABLE_SCHEMA LIKE '[database_name]'

The following command will also work:

SHOW FULL TABLES IN [database_name] WHERE TABLE_TYPE LIKE 'VIEW';

That will return the view names and their type, which is always "view":

General Tip: Using Table Aliases

Writing SQL queries is an art as much as a science. There are some good habits that you can develop that will pay dividends in productivity and/or ease of writing. For example, table (or SQL) aliases are used to give a table, or a column in a table, a temporary name that only exists for the duration of the query. Aliases may be employed to make column names more readable and less error prone.

All you need to do is include the "AS [alias_name]" after the table name in the FROM clause:

SELECT column_name(s)
FROM table_name AS alias_name;

Aliases really earn their keep when you use a Query Editor, like Navicat's. Suppose that we want to select some fields from the actor table. First, we would leave the column list empty and we would enter the FROM clause, complete with a table alias:

SELECT

FROM actor as a

Now, when we enter our shorter table alias, Navicat presents an auto-complete list with all the table columns:

Writing queries in this way is not only faster, but it eliminates the chance of misspelling a column!

Conclusion

In today's blog, we learned a couple of queries and a tip to make our SELECTs almost write themselves using Navicat Premium as the database client. Navicat helps you code fast with Code Completion and customizable Code Snippets by getting suggestions for keywords and stripping the repetition from coding. You can try it for 14 days completely free of charge for evaluation purposes.

Many database management and development tools support multiple connections to homogeneous databases, i.e., where they are all of the same type, ALL MySQL, ALL SQL Server, ALL Oracle, etc. On the other hand, very few support heterogeneous database servers, i.e. MySQL AND SQL Server AND Oracle, etc. Don't believe me? Just google it!

One of the few tools which does support heterogeneous database products is Navicat Premium. Moreover, it can connect simultaneously to MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases from a single application. It is also compatible with most cloud databases, including Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud, Alibaba Cloud, Tencent Cloud, MongoDB Atlas and Huawei Cloud.

In today's tip, we'll learn how to set up multiple connections in Navicat Premium - one to a local MySQL instance and another to Microsoft Azure.

Connecting to a Local Database Instance

Navicat Premium supports a wide array of connection types for both local instances and cloud services. Moreover, connections may be established over secure protocols such as SSL and SSH. The latter, Secure Tunneling Protocol (SSH), is a good option in situations where your Internet Service Provider (ISP) does not provide direct access to its server; HTTP is another good choice.

Here are the steps for connecting to a local MySQL instance:

• Click Connection and select your server type. Then, enter the necessary information in the Connection dialog:
  
  
  
  Note that Navicat authorizes you to make connection to remote servers running on different platforms (i.e. Windows, macOS, Linux and UNIX), and supports PAM and GSSAPI authentication.
  
  Later, you can edit the connection properties by right-clicking the connection and selecting Edit Connection.


• You can test your connection properties by clicking the Test Connection button.


• If you'd like to customize your database list, you may do so on the Databases tab:
  
  


• After closing the New Connection dialog, your new connection will appear in the Navigation Pane, on the left:
  
  

Connecting to a Cloud Service

The procedure for connecting to a cloud service is similar. Let's connect to Microsoft Azure SQL Database instance.

• Select Connection -> Microsoft Azure -> Microsoft Azure SQL Database... from the main menu.
• Once again, enter the necessary information in the General tab of the Connection dialog:
  
  
  
  You'll find the Host value under the Server name header on the Overview page of your Azure instance. There's even a button to copy the name to the clipboard!
  
  


• After closing the New Connection dialog, your new connection will appear in the Navigation Pane. You can open your connections by double-clicking them (I color-coded these to highlight them):
  
  

Conclusion

In today's tip we learned how to set up multiple connections in Navicat Premium. Being able to connect to multiple database instances simultaneously offers many benefits, from being able to query multiple instances using the same SELECT statement, to easier migration. In fact, we explored How to Query across Multiple Databases in the previous tip. Interested in trying out Navicat Premium? You can evaluate it for 14 days completely free of charge!

With Master-Slave topologies and modern practices such as Database sharding becoming increasingly ubiquitous, database administrators (DBAs) and developers are working with multiple databases more than ever before. Doing so is made a lot easier by software that can accommodate multiple database connections.

That's where Navicat Premium comes in. It's a database development, administration and management tool that allows you to simultaneously connect to MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases. Navicat is also compatible with most cloud databases, including Amazon RDS, Amazon Aurora, Amazon Redshift, Microsoft Azure, Oracle Cloud, Google Cloud, Alibaba Cloud, Tencent Cloud, MongoDB Atlas and Huawei Cloud.

In today's blog, we'll learn how to construct and execute a SELECT query that will fetch data from multiple databases using navicat Premium's SQL Editor.

Setting up the Environment

We'll be needing a couple of tables, each within their own database. As it happens, I've got a few copies of the Sakila Sample Database. I've created copies of the actors table and split its contents down the middle, so that names starting with A to L are in the first database and names starting with M to Z are in the other. That will allow us to combine the two groups of names into one result set. Here is their layout in the Navicat object pane:

Multiple Database SELECT Syntax

Just as you can refer to a table within the default database as tbl_name in your SELECT statements, you can also prefix the table name with the database name, e.g. db_name.tbl_name, to specify a database explicitly. The database prefix can also be employed to combine different databases within one SELECT statement's table list, as specified after the FROM keyword. Hence, the following is valid SQL:

SELECT database1.table1.field1,
       database2.table1.field1
FROM database1.table1,
      database.table1
WHERE database1.table1.age > 12;

Using Table JOINs

You can JOIN tables just as you normally would; just be sure to fully qualify the table names by prepending the database name:

SELECT *
FROM database1.table1 T1
JOIN database2.table1 AS T2 ON T1.id = T2.id

If you don't need to JOIN the tables on a common field, you can combine multiple SELECTs using the UNION operator:

SELECT *
 FROM database1.table1 T1
 WHERE T1.age > 12
UNION
SELECT *
 FROM database2.table1 T2
 WHERE T2.age > 12;

Now that we know how to query two tables at a time, let's try out a similar query on our actors table. We'll SELECT actors whose IDs are between a certain range:

SELECT T1.actor_id,
       T1.first_name,
                         T1.last_name
 FROM sakila.`actor_a-l` T1
 WHERE T1.actor_id BETWEEN 30 AND 50
UNION
SELECT T2.actor_id,
                         T2.first_name,
           T2.last_name
 FROM sakila2.`actor_m-z` T2
 WHERE T2.actor_id BETWEEN 30 AND 50
 ORDER BY last_name;

You can be the results that there are actors who are stored in the A - L table, while some originate from the M - Z table:

Conclusion

In today's blog, we learned how to construct and execute a SELECT query to fetch data from multiple databases using Navicat Premium's SQL Editor. Navicat helps you code fast with Code Completion and customizable Code Snippets by getting suggestions for keywords and stripping the repetition from coding. You can try it for 14 days completely free of charge for evaluation purposes.

The Some SELECT Queries You Must Know blog presented a couple of the most important queries to know, along with some examples. Continuing with that theme, today's blog focuses on the invaluable BETWEEN operator.

Limiting Values to a Certain Range

One way to filter the number of rows returned from a query is to limit the values of one or more fields to those that fall within a range. Typically, this can be done using the >= and <= operators. To illustrate, here's a query that returns information about Sakila film rentals that occurred between the 5th and 6th of July of 2005:

SELECT
        customer_list.`name`,
        rental.rental_date,
        film.title
FROM
        customer_list
        INNER JOIN rental ON customer_list.ID = rental.customer_id
        INNER JOIN film ON rental.inventory_id = film.film_id
WHERE
        rental_date >= '2005-07-05'AND rental_date <= '2005-07-06'

A shorter and more readable way to delineate the same range is to use the BETWEEN operator. The BETWEEN operator is used to select the value within a certain range. The values defined as part of the BETWEEN range are inclusive i.e. the values that are mentioned in the range are included at the start and end values:

WHERE rental_date BETWEEN '2005-07-05' AND '2005-07-06'

In both cases, the results are constrained to the given date range:

While ideal for dates, the BETWEEN operator works equally well with other data types. Consider this further filtering of the above data that limits the results to those rentals that cost between 2.99 and 4.99:

Conclusion

Today's blog presented the all-important BETWEEN operator, along with some examples using Navicat Premium as the database client. Navicat helps you code fast with Code Completion and customizable Code Snippets by getting suggestions for keywords and stripping the repetition from coding. You can try it for 14 days completely free of charge for evaluation purposes.

Data is a core part of many businesses both big and small. For example, Facebook stores each user's profile information, including data about their friends and posts within a database system. SQL (short for Structured Query Language) is the programming language that enables developers and database administrators to work with that data.

There are a few frequently used SQL commands you should be familiar with for database work. Not including Data Definition Languages (DDL) or Data Manipulation Language (DML) statements, SQL commands include those to fetch data from tables and views using the SELECT statement. Today's blog will present a couple of the most important queries to know, along with some examples using Navicat Premium as the database client.

Determining the Lowest/Highest Value for a Column

The Sakila sample database contains a number of tables themed around the film industry that cover everything from actors and film studios to video rental stores. The queries that we'll be building here today will run against it, so you may want to refer to the Generating Reports on MySQL Data tutorial for instructions on downloading and installing the Sakila database.

One of the central tables in the Sakila database is the film table. It contains details about every film that our fictional video rental store owns. It includes information such as the film titles, release year, as well as the rental price:

Suppose that we wanted to know what the price range was - that is to say, the lowest and highest rental rates? We could easily find out using the MIN() and MAX() aggregate functions. An aggregate function performs a calculation on a set of values and returns a single value result. There are many aggregate functions, including AVG, COUNT, SUM, MIN, MAX, etc. Here's a query that applies MIN() and MAX() to the rental_rate field of the film table:

SELECT MIN(f.rental_rate) as lowest_price,
       MAX(f.rental_rate) as highest_price
FROM film f;

As expected, each function returns a single value:

Grouping Results by Category

One of the most powerful clauses in SQL is GROUP BY. It group rows that have the same values into summary rows. As such, the GROUP BY statement is often used with aggregate functions (COUNT, MAX, MIN, SUM, AVG) to group the result set by one or more columns.

We can use the GROUP BY clause to list the minimum and maximum rental cost for each movie rating - i.e. General, PG, PG-13, etc. All we need to do is add the rating field to the column list and append the GROUP BY clause to the end of our existing query:

SELECT f.rating,
       MIN(f.rental_rate) as lowest_price,
       MAX(f.rental_rate) as highest_price
FROM film f
GROUP BY f.rating;

Our results show that each movie rating has films that range in price from $0.99 to $4.99:

Conclusion

Today's blog presented a couple of the most important queries to know, along with some examples using Navicat Premium as the database client. Navicat helps you code fast with Code Completion and customizable Code Snippets by getting suggestions for keywords and stripping the repetition from coding. You can try it for 14 days completely free of charge for evaluation purposes.

In last week's How the MySQL 8 Performance Schema Helps Diagnose Query Deadlocks blog, we had a crash course on Mutexes and Threads, learned about the MySQL Performance Schema, and applied a few queries against it for investigating performance bottlenecks. In today's follow-up will present a different approach to bottlenecks and deadlock investigation using Navicat Monitor.

Navicat Monitor at a Glance

Navicat Monitor is an agentless remote server monitoring tool for MySQL/MariaDB that is packed with features to make monitoring your database (DB) instances as effective and easy as possible. The term "agentless" is key because it describes a server-based architecture that does not require any software installation on the servers being monitored. Moreover, Navicat Monitor is accessible from anywhere via a web browser, thus providing you unhampered access to easily and seamlessly track your servers from anywhere in the world, at any time of day or night.

It boasts a whole host of features. Here are some of them, listed by screen:

• Real-time Interactive Overview
  
  • View all registered MySQL/MariaDB instances and availability groups on one central web-based interface
  • Monitor the live MySQL/MariaDB metrics, CPU, memory and swap usage on host machines
  • Explore historical metrics in an hour
• Instance Details
  
  • Agentless remote monitoring
  • Generate reports for server performance metrics
  • Export reports to PDF files
• Alerts
  
  • Come preconfigured with over 40 fully-customizable alert policies
  • Get helpful advice on how to improve server performances
  • Use SMTP, SMS, SNMP Trap and Slack with customizable thresholds
• Query Analyzer
  
  • Analysing Slow Query Log and General Query Log
  • Find out queries having the biggest impact on your system
  • Store a history to diagnose deadlock problems
• Replication Monitoring
  
  • Displays your Replication Topologies and enable you to quickly see the status of each replication
  • Replication error history for replication trouble shooting
  • Send alerts when any replication problems are detected
• Security Monitoring
  
  • Control of access to your monitoring assets and features
  • Improve MySQL/MariaDB security by sending you alerts
  • Detect MySQL/MariaDB hacking activities
• User Management
  
  • Role-based access control
  • User integration for OpenLDAP or Active Directory
  • Restrict login or role access by IP Address
• Configuration Export and Restore
  
  • Save the most recent configuration and restore whenever you like
  • Migrate Navicat Monitor to new host
  • Allow Repository Database migration

Spotting Deadlocked Queries

The Query Analyzer screen shows the summary information of all executing queries and helps you identify problematic queries, such as top queries with cumulative execution time count, slow queries, and deadlocks caused by two or more queries blocking each other. You'll find the Latest Deadlocked Query in the top portion of the screen:

You can view previous deadlocks by clicking the View All button. Doing so opens the Deadlock page. It displays all deadlocks detected on the selected instance:

All monitored instances are shown in the left pane. Selecting an instance brings up deadlocks for that instance. You can filter the list by providing a value in the "Search for a deadlock" text box.

By default, the deadlock list refreshes every 5 seconds automatically. You can change the auto-refresh time using the Refresh Time drop-down menu. To pause the auto refresh, click the Pause button:

You can also set the number of rows to display via the Rows to Display drop-down menu.

Conclusion

In today's blog, we learned how to spot bottlenecks and/or deadlocks in MySQL 8 using Navicat Monitor. Thinking about purchasing Navicat Monitor for MySQL/MariaDB? It's now available via monthly and yearly subscription!

MySQL 5.5 saw the addition of the performance_schema and information_schema databases. As we saw in last week's blog, tables in information_schema contain statistical information about tables, plugins, partitions, processlist, status and global variables. As the name suggests, the tables of the performance_schema can be utilized to improve performance of our MySQL instances. Just how to do that will be the topic of today's blog. Just like last time, we'll be using Navicat Premium to demo the various queries.

A Brief Overview

The Performance Schema is a tool for monitoring MySQL Server execution at a low level. The Performance Schema's storage engine shared the "performance_schema" name in order to easily distinguish it from other storage engines. Having its own engine allows us to access information about server execution while having minimal impact on server performance. Moreover, it uses views or temporary tables so as to minimize persistent disk storage. Finally, memory allocation is all done at server startup, so there is no further memory reallocation or sizing, which greatly streamlines performance.

The Performance Schema is enabled by default as of MySQL 5.6.6. Before that version, it was disabled by default. You can verify its status using this statement:

If you need to, you can always enable it explicitly by starting the server with the --performance-schema=ON flag.

Now let's get into some practical uses for the performance_schema.

A Crash Course on Mutexes and Threads

A mutex is a synchronization mechanism used in the code to enforce that only one thread at a given time can have access to some common resource. The resource is said to be "protected" by the mutex. The word "Mutex" is an informal abbreviation for "mutex variable", which is itself is short for "mutual exclusion". In MySQL, it's the low-level object that InnoDB uses to represent and enforce exclusive-access locks to internal in-memory data structures. Here's how it works:

Once the lock is acquired, any other process, thread, and so on is prevented from acquiring the same lock. In InnoDB, multiple threads of execution access shared data structures. InnoDB synchronizes these accesses with its own implementation of mutexes and read/write locks. When two threads executing in the server (for example, two user sessions executing a query simultaneously) need to access the same resource, such as a file, a buffer, or some piece of data, these two threads will compete against each other, so that the first query to obtain a lock on the mutex will cause the other query to wait until the first is done and unlocks the mutex. Should the first thread take a long time to complete, it can thus hold up other processes.

Some Useful Queries

All of the mutexes are listed in the mutex_instances table of the Performance Schema, which can be extremely helpful in investigating performance bottlenecks. The mutex_instances.LOCKED_BY_THREAD_ID and rwlock_instances.WRITE_LOCKED_BY_THREAD_ID columns are extremely important for investigating performance bottlenecks or deadlocks. Here's how to use them:

Suppose that thread 1 is stuck waiting for a mutex.

You can determine what the thread is waiting for:

Say the query result identifies that the thread is waiting for mutex A, found in events_waits_current.OBJECT_INSTANCE_BEGIN.

You can determine which thread is holding mutex A:

Say the query result identifies that it is thread 2 holding mutex A, as found in mutex_instances.LOCKED_BY_THREAD_ID.

You can see what thread 2 is doing using this query:

Conclusion

In today's blog, we learned how to use the Performance Schema to diagnose bottlenecks and/or deadlocks in MySQL 8. An even easier way is to use Navicat Monitor. It has a Query Analyzer that shows the summary information of all executing queries and lets you easily detecting deadlocks, such as when two or more queries permanently block each other. We'll explore that next time.

MySQL 5.5 saw the addition of the performance_schema and information_schema databases. As we saw in last week's blog, tables in information_schema contain statistical information about tables, plugins, partitions, processlist, status and global variables. As the name suggests, the tables of the performance_schema can be utilized to improve performance of our MySQL instances. Just how to do that will be the topic of today's blog. Just like last time, we'll be using Navicat Premium to demo the various queries.

A Brief Overview

The Performance Schema is a tool for monitoring MySQL Server execution at a low level. The Performance Schema's storage engine shared the "performance_schema" name in order to easily distinguish it from other storage engines. Having its own engine allows us to access information about server execution while having minimal impact on server performance. Moreover, it uses views or temporary tables so as to minimize persistent disk storage. Finally, memory allocation is all done at server startup, so there is no further memory reallocation or sizing, which greatly streamlines performance.

The Performance Schema is enabled by default as of MySQL 5.6.6. Before that version, it was disabled by default. You can verify its status using this statement:

If you need to, you can always enable it explicitly by starting the server with the --performance-schema=ON flag.

Now let's get into some practical uses for the performance_schema.

A Crash Course on Mutexes and Threads

A mutex is a synchronization mechanism used in the code to enforce that only one thread at a given time can have access to some common resource. The resource is said to be "protected" by the mutex. The word "Mutex" is an informal abbreviation for "mutex variable", which is itself is short for "mutual exclusion". In MySQL, it's the low-level object that InnoDB uses to represent and enforce exclusive-access locks to internal in-memory data structures. Here's how it works:

Once the lock is acquired, any other process, thread, and so on is prevented from acquiring the same lock. In InnoDB, multiple threads of execution access shared data structures. InnoDB synchronizes these accesses with its own implementation of mutexes and read/write locks. When two threads executing in the server (for example, two user sessions executing a query simultaneously) need to access the same resource, such as a file, a buffer, or some piece of data, these two threads will compete against each other, so that the first query to obtain a lock on the mutex will cause the other query to wait until the first is done and unlocks the mutex. Should the first thread take a long time to complete, it can thus hold up other processes.

Some Useful Queries

All of the mutexes are listed in the mutex_instances table of the Performance Schema, which can be extremely helpful in investigating performance bottlenecks. The mutex_instances.LOCKED_BY_THREAD_ID and rwlock_instances.WRITE_LOCKED_BY_THREAD_ID columns are extremely important for investigating performance bottlenecks or deadlocks. Here's how to use them:

Suppose that thread 1 is stuck waiting for a mutex.

You can determine what the thread is waiting for:

Say the query result identifies that the thread is waiting for mutex A, found in events_waits_current.OBJECT_INSTANCE_BEGIN.

You can determine which thread is holding mutex A:

Say the query result identifies that it is thread 2 holding mutex A, as found in mutex_instances.LOCKED_BY_THREAD_ID.

You can see what thread 2 is doing using this query:

Conclusion

In today's blog, we learned how to use the Performance Schema to diagnose bottlenecks and/or deadlocks in MySQL 8. An even easier way is to use Navicat Monitor. It has a Query Analyzer that shows the summary information of all executing queries and lets you easily detecting deadlocks, such as when two or more queries permanently block each other. We'll explore that next time.

MySQL 5.5 saw the addition of the performance_schema and information_schema databases. As we saw in last week's blog, tables in information_schema contain statistical information about tables, plugins, partitions, processlist, status and global variables. As the name suggests, the tables of the performance_schema can be utilized to improve performance of our MySQL instances. Just how to do that will be the topic of today's blog. Just like last time, we'll be using Navicat Premium to demo the various queries.

A Brief Overview

The Performance Schema is a tool for monitoring MySQL Server execution at a low level. The Performance Schema's storage engine shared the "performance_schema" name in order to easily distinguish it from other storage engines. Having its own engine allows us to access information about server execution while having minimal impact on server performance. Moreover, it uses views or temporary tables so as to minimize persistent disk storage. Finally, memory allocation is all done at server startup, so there is no further memory reallocation or sizing, which greatly streamlines performance.

The Performance Schema is enabled by default as of MySQL 5.6.6. Before that version, it was disabled by default. You can verify its status using this statement:

If you need to, you can always enable it explicitly by starting the server with the --performance-schema=ON flag.

Now let's get into some practical uses for the performance_schema.

A Crash Course on Mutexes and Threads

A mutex is a synchronization mechanism used in the code to enforce that only one thread at a given time can have access to some common resource. The resource is said to be "protected" by the mutex. The word "Mutex" is an informal abbreviation for "mutex variable", which is itself is short for "mutual exclusion". In MySQL, it's the low-level object that InnoDB uses to represent and enforce exclusive-access locks to internal in-memory data structures. Here's how it works:

Once the lock is acquired, any other process, thread, and so on is prevented from acquiring the same lock. In InnoDB, multiple threads of execution access shared data structures. InnoDB synchronizes these accesses with its own implementation of mutexes and read/write locks. When two threads executing in the server (for example, two user sessions executing a query simultaneously) need to access the same resource, such as a file, a buffer, or some piece of data, these two threads will compete against each other, so that the first query to obtain a lock on the mutex will cause the other query to wait until the first is done and unlocks the mutex. Should the first thread take a long time to complete, it can thus hold up other processes.

Some Useful Queries

All of the mutexes are listed in the mutex_instances table of the Performance Schema, which can be extremely helpful in investigating performance bottlenecks. The mutex_instances.LOCKED_BY_THREAD_ID and rwlock_instances.WRITE_LOCKED_BY_THREAD_ID columns are extremely important for investigating performance bottlenecks or deadlocks. Here's how to use them:

Suppose that thread 1 is stuck waiting for a mutex.

You can determine what the thread is waiting for:

Say the query result identifies that the thread is waiting for mutex A, found in events_waits_current.OBJECT_INSTANCE_BEGIN.

You can determine which thread is holding mutex A:

Say the query result identifies that it is thread 2 holding mutex A, as found in mutex_instances.LOCKED_BY_THREAD_ID.

You can see what thread 2 is doing using this query:

Conclusion

In today's blog, we learned how to use the Performance Schema to diagnose bottlenecks and/or deadlocks in MySQL 8. An even easier way is to use Navicat Monitor. It has a Query Analyzer that shows the summary information of all executing queries and lets you easily detecting deadlocks, such as when two or more queries permanently block each other. We'll explore that next time.

In relational databases, database metadata, such as information about the MySQL server, the name of a database or table, the data type of a column, or access privileges are stored in the data dictionary and/or system catalog. MySQL's provides database metadata in a special schema called INFORMATION_SCHEMA. There is one INFORMATION_SCHEMA within each MySQL instance. It contains several read-only tables that you can query to obtain the information that you are looking for. In today's blog, we'll explore a few practical uses for the INFORMATION_SCHEMA, as demonstrated using Navicat Premium.

Obtaining Table Information

The information_schema.tables table contains metadata about, you guessed it, tables! Beside the table names, you can also retrieve their type (base table or view) and engine:

SELECT table_name, table_type, engine
FROM information_schema.tables
WHERE table_schema = 'sakila'
ORDER BY table_name;

Here is the above query and results in Navicat:

You can also query information_schema.tables to get the size of a table:

SELECT
    table_name AS `Table`,
    round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB`
FROM information_schema.TABLES
WHERE table_schema = "sakila"
AND table_name = "film";

Here are the results in Navicat Premium:

With a few tweaks, you can list the size of every table in every database:

SELECT
     table_schema as `Database`,
     table_name AS `Table`,
     round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB`
FROM information_schema.TABLES
ORDER BY (data_length + index_length) DESC;

You can even use information_schema.tables to list the size of every database in a MySQL instance!

SELECT
     table_schema as `Database`,
     table_name AS `Table`,
     round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB`
FROM information_schema.TABLES
ORDER BY (data_length + index_length) DESC;

Viewing Table Statistics

The INFORMATION_SCHEMA.STATISTICS table contains cached values. As such, these expire after 24 hours, by default. If there are no cached statistics or statistics have expired, statistics are retrieved from storage engines when querying table statistics columns.

One use for the INFORMATION_SCHEMA.STATISTICS table is to see indexes for all tables within a specific schema:

SELECT DISTINCT
    TABLE_NAME,
    INDEX_NAME
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = 'your_schema';

Here are the results in Navicat for the sakila database:

You can view all indexes in all schemas by simply removing the where clause. In that case you may want to add the database name as well:

SELECT DISTINCT
    stat.TABLE_SCHEMA as 'DATABASE',
    TABLE_NAME,
    INDEX_NAME
FROM INFORMATION_SCHEMA.STATISTICS stat;

Conclusion

In today's blog we learned just a few of the many ways to use the MySQL INFORMATION_SCHEMA to obtain metadata information about a variety of objects within an MySQL instance, form the databases to tables, columns, indexes, and more. Although queries were run in Navicat Premium, Navicat for MySQL will work just as well! Try each of them for yourself; both come with a 14 day free trial!

If you work regularly with MySQL or MariaDB, then you will probably find Navicat Premium or Navicat for MySQL to be indispensable. In addition to MySQL and MariaDB, Navicat for MySQL also supports a number of cloud services, including Amazon RDS, Amazon Aurora, Oracle Cloud, Google Cloud, Microsoft Azure, Alibaba Cloud, Tencent Cloud and Huawei Cloud. Navicat Premium is a database development tool that allows you to simultaneously connect to MySQL, MariaDB, MongoDB, SQL Server, Oracle, PostgreSQL, and SQLite databases from a single application, and is also compatible with cloud databases. Both help you create views, queries and functions using an easy-to-use GUI interface. Moreover, you can save your work to the Cloud for reuse and collaborating with team members.

In today's blog, I'll be sharing a few tips and tricks for MySQL that you can apply using either Navicat for MySQL or Navicat Premium.

1: Retrieve Unique Values from a Single Column

Suppose you have a database filled with thousands of employee records and youd like to know how many unique employee last names there are within the thousands of rows. We can create a SELECT DISTINCT query that will do that:

SELECT DISTINCT
    lastname
FROM
    employees
ORDER BY lastname;

Rather than execute the above query every time that we wanted to see the distict employees we could create a view that we can execute queries against:

CREATE VIEW distinct_emp_names AS
SELECT DISTINCT
    lastname
FROM
    employees
ORDER BY lastname;

Here are the results:

2: Retrieve Unique Data from Multiple Columns

The DISTINCT clause also works with more than one column. In this case, MySQL relies on the combination of values in these columns to determine their uniqueness in the result set. For example, to get the unique combination of city and state from a table, you can create the following view:

CREATE VIEW distinct_cities_and_states AS
SELECT DISTINCT
    state, city
FROM
    customers
WHERE
    state IS NOT NULL
ORDER BY state, city;

Here are the results of the view:

3: Modify a Column Name

Suppose you just want to change the name of a column, you could run an ALTER TABLE statement to do so.

In Navicat, if you right-click a field in the Table Designer, you can choose to add, insert, delete and, of course, rename the field:

4: Split a Full Name into First and Last Names

There is often a need to split a column that contains a full name (i.e. full_name) into two columns, such as first_name and last_name. Here's how using the ALTER TABLE statement:

ALTER TABLE emails
        ADD COLUMN `first_name` VARCHAR(30) AFTER `full_name`,
        ADD COLUMN `last_name` VARCHAR(30) AFTER `first_name`;
UPDATE emails
SET
        # Trim the white space
        `full_name` = LTRIM(RTRIM(`full_name`)),
        # Get the first name and copy it to a new column
        `first_name` = SUBSTRING_INDEX(`full_name`, ' ', 1),
        # Get the second name and copy it to a new column
        `last_name` = SUBSTRING_INDEX(`full_name`, ' ', -1)

Here's is the above statement as is appears in the Navicat Query Editor:

Conclusion

In today's blog we learned a few tips and tricks for MySQL that we can apply using either Navicat Premium or Navicat for MySQL. Navicat database management tools make most DBA and development tasks a lot easier to carry out. Try them for yourself; both come with a 14 day free trial!

A database view is a virtual or logical table which is comprised of a SELECT query. Much like a database table, a view also consists of rows and columns that you can query against. Most database management systems, including MySQL, even allow you to update data in the underlying tables through the view, but with some caveats. In today's blog, we'll learn what a view is and how to create one for MySQL 8 using Navicat Premium as our client.

Basic Syntax

In MySQL, you use the CREATE VIEW statement to create a new view. Here is the basic syntax:

Now, let's examine the syntax in more detail.

View Processing Algorithms

The ALGORITHM attribute tells MySQL which mechanism to use when creating the view. MySQL provides three algorithms: MERGE, TEMPTABLE, and UNDEFINED:

• The MERGE algorithm combines the input query with the SELECT statement, which defines the view, into a single query. MySQL then executes the combined query to return the merged result set. The MERGE algorithm cannot be applied to SELECT statements that contain aggregate functions such as MIN, MAX, SUM, COUNT, AVG or DISTINCT, GROUP BY, HAVING, LIMIT, UNION, and UNION ALL. If the MERGE algorithm cannot be applied, MySQL automatically changes the algorithm to UNDEFINED.
• The TEMPTABLE algorithm first creates a temporary table based on the SELECT statement that defines the view, and then it executes the input query against this temporary table. Because MySQL has to create a temporary table to store the result set and moves the data from the base tables to the temporary table, the TEMPTABLE algorithm is less efficient than the MERGE algorithm.
• UNDEFINED is the default algorithm when you create a view without specifying an explicit algorithm. The UNDEFINED algorithm lets MySQL make a choice of using MERGE or TEMPTABLE algorithm. MySQL chooses the MERGE algorithm first, due to its greater efficiency, but falls back to the TEMPTABLE algorithm if MERGE cannot be employed.

View Name

You can choose whatever name you wish for your view, so long as you follow the same naming rules as for tables. Moreover, views and tables share the same namespace within the database, so you can't give your view the same name as any existing table of view.

SELECT Statement

In the SELECT statement you can query data from any table or view that exists in the database. However, there are a few rules that the SELECT statement adhere to:

• The SELECT statement may contain a subquery in the WHERE clause but not in the FROM clause.
• The SELECT statement cannot refer to any variables including local variables, user variables, and session variables.
• The SELECT statement cannot refer to the parameters of prepared statements.

Creating a View in Navicat

In Navicat, you can create a new view by clicking the View button on the main toolbar and then clicking "New view" on the Objects toolbar:

The Definition tab is where you write your SQL. You can even use the View Builder to help write your statement!

The Algorithm can be found on the Advanced tab, along with a few other options:

Once you're done, you can test your View using the Preview button and then save it by clicking on Save.

Conclusion

Views are a great way to combine data from one or more tables in a format that you can query, but keep in mind that there are some disadvantages of using database views. For one, querying data from a database view can be slow - especially if the view is created based on other views. Also, you have to remember to change the view whenever you change the structure of a table that your view refers to.

Most relational databases - including MySQL, MariaDB, and SQL Server - support stored procedures and functions. Stored procedures and functions are actually very similar, and can in fact be utilized to accomplish the same task. That being said, there are some crucial differences between the two that need to be considered when deciding which to use for a given job. We'll go over these in today's blog

Stored Procedures

A stored procedure - or "proc" for short - is a set of Structured Query Language (SQL) statements with an assigned name, which are stored in a relational database management system as a group, so it can be reused and shared by multiple programs. Stored procedures can access or modify data in a database, but it is not tied to a specific database or object. This loose coupling is advantageous because it's easy to reappropriate a proc for a different but similar purpose.

Stored procedures can accept input parameters and return multiple values of output parameters; moreover, stored procedures can program statements to perform operations in the database and return a status value to a calling procedure or batch.

Finally, stored procedures can execute multiple SQL statements, call functions, and even iterate over results sets, performing complex operations akin to programming code. When completed, the proc typically returns one of more result sets to the calling application.

User Functions

A function is similar to a stored procedure in that it contains a set of SQL statements that perform a specific task. The idea behind Functions is to foster code reusability. If you have to repeatedly write large SQL scripts to perform the same task, you can create a function that performs that task so that, next time, instead of rewriting the SQL, you can simply call that function. Databases typically include a set of built-in functions that perform a variety of tasks, so always take a look at these before writing your own.

A function accepts inputs in the form of parameters and returns a value. Unlike a stored procedure, a function cannot return a result set. Moreover, functions cannot modify the server environment or operating system environment.

Main Differences

While both procs and functions can be employed in similar ways, functions are designed to send their output to a query or SQL statement. Meanwhile, stored procedures are designed to return outputs (i.e. one or more result sets) to the application.

Another difference is that you can group a set of SQL statements and execute them within a stored procedure, stored procedures cannot be called within SQL statements. Functions, on the other hand, may be invoked directly from your queries and/or stored procedures.

Finally, a limitation of functions is that they have to be called for each row. Therefore, if you are using functions with large data sets, you can encounter performance issues.

Viewing Stored Procedures and Functions in Navicat

In Navicat database management and development tools, you'll see both procs and functions under "Functions". The stored procedures have the "Px" prefix, while functions have an "fx":

Conclusion

Stored procedures and functions are very similar in many ways, but each serve a different purpose. You can think of a stored proc as a grouping of SQL statements, while a function takes input and returns an output value based on the input parameters.

Regular expressions (regex) provide a way to match strings against a pattern so that your searches are "fuzzy" rather than exact. MongoDB comes with a regex engine built in so you can dig up documents even if you don't know exactly what the exact Field value is that you're looking for. In today's blog we'll learn how to use regexes in MongoDB, using Navicat for MongoDB.

Basic Syntax

MongoDB provides the regex operator for searching strings in the collection. The following example shows how it's done, using the Sakila Sample Database:

Let's say that we wanted to find movies with actors named "DAN", "DANNY", or even "DANIEL". Here's a statement to do that:

Once the command is executed successfully, the following Output will be shown:

We can simplify the statement by removing the "$regex:" qualifier and enclosing the search string within forward slashes (/) instead of quotes, as forward slashes denote a regex:

Searching with Multiple Search Strings

We can include more than on search string to match various combinations. Let's say that we wanted to find movies with Carrie-Anne Moss by matching "Carrie Moss" or "moss carrie-anne". Here's a statement to do that:

$elemMatch will return those records, where an array element matches both criterias. In contrast, using a plain $and (which is the default for a list of criterias) without $elemMatch would return movies with "Carrie-Anne Moss", but also those where "Sandra Moss" and "Carrie-Anne Fisher" are featured. That would be more of a superset of the information we want to retrieve. Also note the "i" flag; it makes the regex case-insensitive. It's useful for searches that are entered by the user, because we can't rely on the user to use mixed case.

The options Parameter

We can also provide additional instructions to our regexes via the options parameter.

• i: Case insensitivity to match upper and lower cases.
• m: For patterns that include anchors (i.e. ^ for the start, $ for the end), match at the beginning or end of each line for strings with multiline values. Without this option, these anchors match at beginning or end of the string.
• x: Extended capability to ignore all white space characters in the $regex pattern unless escaped or included in a character class. Unlike other flags, this one requires $regex with $options syntax
• s: Allows the dot character (.) to match all characters including newline characters.

Conclusion

The $regex operator provides an easy means of pattern matching in MongoDB. For best results, make sure that the document fields that you are searching are indexed. That way, the query will use make use of indexed values to match the regular expression. This makes the search very fast as compared to the regular expression scanning the whole collection.

If you'd like to learn more about Navicat for MongoDB, please visit the product page. Do you work with many database types? Navicat Premium 12.1 also supports MongoDB!

Open up any document in a MongoDB database and you'll notice an _id field:

In fact, the ObjectId/_id is the only field that exists across every MongoDB document. In today's blog, we'll explore what it is and why it's important to your MongoDB database.

The Structure of ObjectId

As a quick, opening summary, these are a few of _id's principal characteristics:

• _id is the primary key on documents in a collection; with it, documents (records) can be differentiated from each one another.
• _id is automatically indexed. Lookups specifying { _id: <someval> } refer to the _id index as their guide.
• By default the _id field is of type ObjectID, one of MongoDB's BSON types. Users can also override _id to something other than an ObjectID, if desired.

ObjectIDs are 12 bytes long, composed of several 2-4 byte chains. Each chain represents and designates a specific aspect of the document's identity. The following values make up the full 12 byte combination:

• a 4-byte value representing the seconds since the Unix epoch
• a 3-byte machine identifier
• a 2-byte process id
• a 3-byte counter, starting with a random value

Typically, you don't have to concern yourself with generating the ObjectID. If a document has not been assigned an _id value, MongoDB will automatically generate one.

Creating New ObjectId

If you want to generate a new ObjectId yourself, you can use the following code:

You can also type it directly into the Navicat editor.

That will generate a unique _id such as: