Overview

9.1 has many improvements to the platform and several individual applications.

Platform

The application order in the rack has been revised
Event logs have been reimplemented, columns have been added, improvement improved, and options added.
Many usability improvements in the installation & setup wizard
App downloads now show progress correctly.
Many apps settings have been moved from postgres to files
Local directory users are now saved in a file (vs slapd)
BerkeleyDB has been removed.
DHCP renewing no longer restarts networking (nor disrupt VPN connections) (#8955)
SIP helper is now disabled by default (#7236)
Improve start-up time.
App display order has been changed in the rack.

Virus Blocker(s)

Virus Blocker has been renamed to "Virus Blocker Lite."
Kaspersky Virus Blocker has been renamed to "Virus Blocker"
The new "Virus Blocker" (paid) application uses a new engine from a better OEM provider.
On upgrade, users will continue to have Kaspersky Virus Blocker until their subscription expires and they will automatically convert to Virus Blocker.

The conversion table:

9.0 and prior	9.1 and after
Kaspersky Virus Blocker	Virus Blocker
Virus Blocker	Virus Blocker Lite

Spam Blocker & Booster

Spam Blocker has been renamed to "Spam Blocker Lite."
A new premium "Spam Blocker" application has been added.
The "Commtouch Spam Booster" application has been folded into the "Spam Blocker" application.
It is not recommended to run both the lite and full spam blocker(s). This results no additional spam caught and worse system performance.
On upgrade the the following conversion will take place such that in all scenarios the server will functionally be the same, but the apps will be slightly different:

The conversion table:

9.0 and prior	9.1 and after
Spam Blocker	Spam Blocker Lite
Spam Blocker + Commtouch Spam Booster	Spam Blocker

Also added a new "pass outbound message" to the event log so it is obvious when running a backwards bridge.

Web Filter

Web Filter now uses an expanded list of categories for greater granularity.
Web Filter has new defaults to reflect the new categories.
New category settings will be converted from old category settings on upgrade.

Web Filter Lite

Ads have been removed from blockpages
Database is now stored in an in-memory hash table.
Performance is improved, but more memory is required.

Spyware Blocker

Now enforces google's safebrowsing malware list.
Obsolete ActiveX functionality removed.

Phish Blocker

Now enforces google's safebrowsing phish list.

Firewall

Firewall rules now use matchers like Bandwidth Control and Port Forwards.
Firewall rules can now match on Username (with Directory Connector) and Group.
Firewall rule ID now displayed in UI and event log.

WAN Balancer

WAN Balancer now installs in the "off" state (#8966)
WAN Balancer now ignores inbound connections (#8942)

Bandwidth Control

Username rules can now match on multiple usernames and globbed expressions (#8879)

OpenVPN

Now distributes openvpn 2.2.1 to remote clients
New client supports windows 8.
Note: these changes only effect newly distributed clients, already configured remote clients are unchanged.

Architectural Changes

Below are some changes made under the cover that are not necessarily visible as a user.

Settings files

We are in the process of moving settings out of the postgress to files. Previously settings were stored in tables in the database. This worked but it had a few downsides:

Created an unnecessary dependence on having the database accessible at all times
Made the code to handle settings reading/writing and modification more difficult than it should be.
Made it impossible to just view your settings manually as you had to run a bunch of database queries.
Its slow

Settings now live in files in a JSON format in /usr/share/untangle/settings/. In 9.1 some of the apps have been converted to use the new settings format, and in future version the rest of the apps and platform will also be converted.

This has a few benefits:

Its much simpler
- which makes reading/writing settings much easier
- highly simplifies other tasks like backup/restore
- allows for external daemons to easily read settings
- etc
Its faster
It allows for advanced users to just edit and backup settings files
It paves the way for command center functionality (as now you can easily have a "standard" web filter settings file)

The following applications/functions now have their settings in files:

Web Filter
Web Filter Lite
Web Cache
Bandwidth Control
Firewall
Spyware Blocker
Spam Blocker
Spam Blocker Lite
Phish Blocker
Attack Blocker
Ad Blocker
Protocol Control
Local Directory

Dropping Berkeley DB

Previous we used a special database to store the Web Filter Lite and Spyware Blocker databases. This has been a performance bottleneck and source of excess disk I/O for little benefit. The databases aren't big enough to justify this.

In 9.1 we store the Web Filter Lite categorization table and Spyware Blocker lists in memory. This significantly speeds things up, but also means more memory is required. In our testing, Web Filter Lite probably uses about 100 megs more memory. Of course, no memory is wasted on the berkeley DB cache now either which is some cases was larger than the database itself.

Ultimately this is just simpler and faster and only requires marginally more memory. Note: Web Filter is NOT effected. Web Filter uses a paid online dynamic categorization service and has its own much smaller/faster cache implementation.

Event Logs

There are two layers of events in Untangle. The first level is the fully-normalized events. Each application will write its own event as things happen. For example, Protocol Control might log that a session is HTTP, and Web Filter might log that it passed the "http://untangle.com" as a Technology categorized site. Meanwhile Bandwidth Control logs the same session as being prioritized as "High," and Directory Connector logs that the "jimbob" user created this session.

These flush to the database every few seconds/minutes.

Later the Reports app would come along and compile de-normalized "fact tables" out of all these separate tables. These "fact tables" were easy and fast to query and generally contained all the information in one row. This allows for you quickly to query all events relating to a certain user for a certain time period to a certain site, without doing excess joins on all the separate normalized tables.

In 9.0 and prior, the event logs queried the normalized tables. This worked fine but users often requested extra fields that simply could not be added without adding joins. When we did add joins to add these extra fields the queries became too slow and sometimes not return at all, and it would often stress a live server enough to slow network traffic. As such we didn't add new columns.

In 9.1 the event logs query the de-normalized fact tables, so the queries are much faster, simpler, and have all the relevant columns easily accessible.

As with all things there are trade-offs:

This means that the fact-table compilation must happen before you can view the fact tables. Currently, we do this periodically in the background (the interval depends on your hardware) and on-demand when the "refresh" button is pressed in the Event Log UI. This means the actual retrieving of the events is faster, but the first visit may require fact-table compilation process to be run if you want more recent events. On a fast server this process takes only a second or two, on a slow server it can take a few minutes.

The good news is that event logs and reports now draw from the same data pool and should match exactly. This also paves the way for more featured event logs. Columns are now all sortable. In the future when we move to a newer extjs javascript library add even more features such as filtering/sorting/searching and even graphs and visualization tools because the database queries can all be done in semi-realtime.

9.1.0 Changelog

Contents