Web Monitor

Web_Monitor Web_Monitor#Categories Web_Monitor#Flag_Sites Web_Monitor#Pass_Sites Web_Monitor#Pass_Clients Web_Monitor#Rules Web_Monitor#Advanced

File:WebMonitor 128x128.png     Web Monitor

Other Links: Web Monitor Description Page Web Monitor Video Demo Web Monitor Screenshots Web Monitor Forums Web Monitor Reports Web Monitor FAQs

---

About Web Monitor

Web Monitor monitors HTTP traffic on your network to monitor user behavior and flag inappropriate content.

• Real-time classification and updates: When your users visit a site, Untangle sends the URL to the cloud to be categorized. When the data is returned, Untangle keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at zVelo and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention, plus you have the option of requesting recategorization of sites.

• HTTPS Filtering: Web Monitor has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.

• Detailed categorization: Web Monitor offers over 140 categories and over 450 million categorized sites. The Web Monitor database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to flag websites related to Sex, but allow sites dealing with Sexual Education or Pregnancy.

Traffic Flow

When scanning traffic, Web Monitor evaluates the pass lists, flag lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the request is received from the client and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of monitoring over both resources that are requested, and content that is returned in response.

HTTP Request

When evaluating HTTP requests, Web Monitor applies the configured rules and lists in the following order:

1. A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Monitor as well as other applications.
2. The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
3. The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
4. The destination site of the request is checked against the Flag Sites list. If a match is found, the traffic is flagged.
5. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the rule that was matched.
6. The category determined in step #1 is compared to the Categories list, and the traffic is allowed and possibly flagged based on the corresponding match. If the category could not be determined, the traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Monitor applies the configured rules and lists in the following order:

1. The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
2. The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
3. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the rule that was matched.

Settings

This section reviews the different settings and configuration options available for Web Monitor.

Categories

Categories allows you to customize which categories of sites will be flagged. Categories that are flagged will allow the user to access the site, but will be silently flagged as a violation for event logs and Reports. These flag actions operate the same way for all of the different Web Monitor options.

File:WM categories.png

Categories

Flag Sites

Under Flag Sites you can add individual domain names you want to be flagged - just enter the domain name (e.g. youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

File:WM flagSites.png

A few sites entered into the Flag Sites list

Pass Sites

Pass Sites is used to pass content that would have otherwise been flagged. This can be useful for "unflagging" sites that you don't want flagged according to flag settings. Any domains you add to the Passed Sites list will be allowed, even if flagged by category or by individual URL - just add the domain and save. Unchecking the pass option will allow the site to be flagged as if the entry was not present. This list uses URL Matcher syntax.

File:WM passSites.png

A few sites entered into the Pass Sites list

Pass Clients

If you add an IP address to this list, Web Monitor will not flag any traffic from that IP regardless of the flagged categories or sites. Just add the IP and save. Unchecking the pass option will have the flag/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

If you have a few users that need to completely bypass Web Monitor controls, consider using pass lists. If you have users that simply need different Web Monitor settings, you should set up a separate rack using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.

File:WM passClients.png

A few different entries in the Pass Clients list

Rules

The Rules tab allows you to specify rules to Flag traffic that passes through Web Monitor.

The Rules documentation describes how rules work and how they are configured. Web Monitor uses rules to determine when to flag specific session. Flagging a session marks it in the logs for reviewing in the event logs or reports, but has no direct effect on the network traffic.

File:WM rules.png

Rules

Rule Actions

• Flag: Allows the traffic which matched the rule to flow, and flags the traffic for easier viewing in the event log.

Rule Types

In previous versions of Web Monitor, there were dedicated lists for flagging certain file extensions or MIME types. This capability is still available using the more flexible filter rules. For flagging specific file extensions, you can create a rule with the condition Web Filter: Response File Extension that has a comma separated list of the extensions to flag in the Value field. For flagging MIME types, you would create a rule with the condition Web Filter: Response Content Type that has a comma separated list of the content types to flag in the Value field.

Below are tables that list the default file extensions and MIME types that were available in previous versions. Note that these lists are not exhaustive, but are included here as a reference, and to simplify creation of such rules via copy/paste of the values in the tables.

Extension	Category	Description
exe	executable	an executable file format
ocx	executable	an executable file format
dll	executable	an executable file format
cab	executable	an ActiveX executable file format
bin	executable	an executable file format
com	executable	an executable file format
jpg	image	an image file format
png	image	an image file format
gif	image	an image file format
jar	java	a Java file format
class	java	a Java file format
swf	flash	the flash file format
mp3	audio	an audio file format
wav	audio	an audio file format
wmf	audio	an audio file format
mpg	video	a video file format
mov	video	a video file format
avi	video	a video file format
hqx	archive	an archived file format
cpt	compression	a compressed file format

Content	Category	Description
application/octet-stream	unspecified data	byte stream
application/x-msdownload	Microsoft download	executable
application/exe	executable	executable
application/x-exe	executable	executable
application/dos-exe	DOS executable	executable
application/x-winexe	Windows executable	executable
application/msdos-windows	MS-DOS executable	executable
application/x-msdos-program	MS-DOS program	executable
application/x-oleobject	Microsoft OLE Object	executable
application/x-java-applet	Java Applet	executable
audio/mpegurl	MPEG audio URLs	audio
audio/x-mpegurl	MPEG audio URLs	audio
audio/mp3	MP3 audio	audio
audio/x-mp3	MP3 audio	audio
audio/mpeg	MPEG audio	audio
audio/mpg	MPEG audio	audio
audio/x-mpeg	MPEG audio	audio
audio/x-mpg	MPEG audio	audio
application/x-ogg	Ogg Vorbis	audio
audio/m4a	MPEG 4 audio	audio
audio/mp2	MP2 audio	audio
audio/mp1	MP1 audio	audio
application/ogg	Ogg Vorbis	audio
audio/wav	Microsoft WAV	audio
audio/x-wav	Microsoft WAV	audio
audio/x-pn-wav	Microsoft WAV	audio
audio/aac	Advanced Audio Coding	audio
audio/midi	MIDI audio	audio
audio/mpeg	MPEG audio	audio
audio/aiff	AIFF audio	audio
audio/x-aiff	AIFF audio	audio
audio/x-pn-aiff	AIFF audio	audio
audio/x-pn-windows-acm	Windows ACM	audio
audio/x-pn-windows-pcm	Windows PCM	audio
audio/basic	8-bit u-law PCM	audio
audio/x-pn-au	Sun audio	audio
audio/3gpp	3GPP	audio
audio/3gpp-encrypted	encrypted 3GPP	audio
audio/scpls	streaming mp3 playlists	audio
audio/x-scpls	streaming mp3 playlists	audio
application/smil	SMIL	audio
application/sdp	Streaming Download Project	audio
application/x-sdp	Streaming Download Project	audio
audio/amr	AMR codec	audio
audio/amr-encrypted	AMR encrypted codec	audio
audio/amr-wb	AMR-WB codec	audio
audio/amr-wb-encrypted	AMR-WB encrypted codec	audio
audio/x-rn-3gpp-amr	3GPP codec	audio
audio/x-rn-3gpp-amr-encrypted	3GPP-AMR encrypted codec	audio
audio/x-rn-3gpp-amr-wb	3gpp-AMR-WB codec	audio
audio/x-rn-3gpp-amr-wb-encrypted	3gpp-AMR_WB encrypted codec	audio
application/streamingmedia	Streaming Media	audio
video/mpeg	MPEG video	video
audio/x-ms-wma	Windows Media	video
video/quicktime	QuickTime	video
video/x-ms-asf	Microsoft ASF	video
video/x-msvideo	Microsoft AVI	video
video/x-sgi-mov	SGI movie	video
video/3gpp	3GPP video	video
video/3gpp-encrypted	3GPP encrypted video	video
video/3gpp2	3GPP2 video	video
audio/x-realaudio	RealAudio	audio
text/vnd.rn-realtext	RealText	text
audio/vnd.rn-realaudio	RealAudio	audio
audio/x-pn-realaudio	RealAudio plug-in	audio
image/vnd.rn-realpix	RealPix	image
application/vnd.rn-realmedia	RealMedia	video
application/vnd.rn-realmedia-vbr	RealMedia VBR	video
application/vnd.rn-realmedia-secure	secure RealMedia	video
application/vnd.rn-realaudio-secure	secure RealAudio	audio
audio/x-realaudio-secure	secure RealAudio	audio
video/vnd.rn-realvideo-secure	secure RealVideo	video
video/vnd.rn-realvideo	RealVideo	video
application/vnd.rn-realsystem-rmj	RealSystem media	video
application/vnd.rn-realsystem-rmx	RealSystem secure media	video
audio/rn-mpeg	MPEG audio	audio
application/x-shockwave-flash	Macromedia Shockwave	multimedia
application/x-director	Macromedia Shockwave	multimedia
application/x-authorware-bin	Macromedia Authorware binary	multimedia
application/x-authorware-map	Macromedia Authorware shocked file	multimedia
application/x-authorware-seg	Macromedia Authorware shocked packet	multimedia
application/futuresplash	Macromedia FutureSplash	multimedia
application/zip	ZIP	archive
application/x-lzh	LZH archive	archive
image/gif	Graphics Interchange Format	image
image/png	Portable Network Graphics	image
image/jpeg	JPEG	image
image/bmp	Microsoft BMP	image
image/tiff	Tagged Image File Format	image
image/x-freehand	Macromedia Freehand	image
image/x-cmu-raster	CMU Raster	image
image/x-rgb	RGB image	image
text/css	cascading style sheet	text
text/html	HTML	text
text/plain	plain text	text
text/richtext	rich text	text
text/tab-separated-values	tab separated values	text
text/xml	XML	text
text/xsl	XSL	text
text/x-sgml	SGML	text
text/x-vcard	vCard	text
application/mac-binhex40	Macintosh BinHex	archive
application/x-stuffit	Macintosh Stuffit archive	archive
application/macwriteii	MacWrite Document	document
application/applefile	Macintosh File	archive
application/mac-compactpro	Macintosh Compact Pro	archive
application/x-bzip2	block compressed	compressed
application/x-shar	shell archive	archive
application/x-gtar	gzipped tar archive	archive
application/x-gzip	gzip compressed	compressed
application/x-tar	4.3BSD tar archive	archive
application/x-ustar	POSIX tar archive	archive
application/x-cpio	old cpio archive	archive
application/x-bcpio	POSIX cpio archive	archive
application/x-sv4crc	System V cpio with CRC	archive
application/x-compress	UNIX compressed	compressed
application/x-sv4cpio	System V cpio	archive
application/x-sh	UNIX shell script	executable
application/x-csh	UNIX csh script	executable
application/x-tcl	Tcl script	executable
application/x-javascript	JavaScript	executable
application/x-excel	Microsoft Excel	document
application/mspowerpoint	Microsoft Powerpoint	document
application/msword	Microsoft Word	document
application/wordperfect5.1	Word Perfect	document
application/rtf	Rich Text Format	document
application/pdf	Adobe Acrobat	document
application/postscript	Postscript	document

Advanced

The Advanced section allows you to configure additional Web Monitor options.

• Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in #HTTPS Options.

• Process HTTPS traffic by hostname in server certificate when SNI information not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server and the server name on the certificate will be used for categorization and filtering purposes.

• Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available: If this option is enabled and neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in #HTTPS Options.

• Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.

File:WM advanced.png

Advanced options

Reports

Web_Monitor_Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Monitor.

Reports

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries: {{#section:All_Reports|'Web Monitor'}}

The tables queried to render these reports:

• http_events
• http_query_events

HTTPS Options

There are many ways to handle HTTPS. An overview of the various techniques is described here.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Monitor processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.

• Process HTTPS traffic by SNI (Server Name Indication) if present.
• Process HTTPS traffic by hostname in server certificate when SNI information not present
• Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

If the SNI-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, it will see "wellsfargo.com" as the SNI information. If enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If "Banking" is flagged it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list. If "wellsfargo.com" is flaggeed it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list.

If No SNI information is present and Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled, then the hostname will be pulled from the certificate presented to the client.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

If no SNI or certificate information was available and Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available the session will be processed and categorized by IP address. If the IP-based processing and categorization of the web requests determines the session should be flagged, the session is reset and no more processing of this session will be done. If the IP-based processing and categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS process (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required. Read here for more information

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Monitor FAQs

Why is there a pass list if Web Monitor can't block sites?

Web Monitor is useful for monitoring web activity, and as part of that it is often useful to flag certain web activity to make it more visible in reports. Adding a site to the pass list will prevent the site from being flagged even if it otherwise would be because the category is flagged or a rule flags it.

Can I block sites with Web Monitor?

No. Web Monitor is for monitoring web activity only. In order to modify or block web content Web Filter is required.