Web Filter: Difference between revisions

From Edge Threat Management Wiki - Arista
Jump to navigationJump to search
No edit summary
 
(29 intermediate revisions by 6 users not shown)
Line 6: Line 6:
<span style="display:none" class="helpSource web_filter_block_categories">Web_Filter#Block_Categories</span>
<span style="display:none" class="helpSource web_filter_block_categories">Web_Filter#Block_Categories</span>
<span style="display:none" class="helpSource web_filter_categories">Web_Filter#Categories</span>
<span style="display:none" class="helpSource web_filter_categories">Web_Filter#Categories</span>
<span style="display:none" class="helpSource web_filter_search_terms">Web_Filter#Search_Terms</span>
<span style="display:none" class="helpSource web_filter_site_lookup">Web_Filter#Site_Lookup</span>
<span style="display:none" class="helpSource web_filter_site_lookup">Web_Filter#Site_Lookup</span>
<span style="display:none" class="helpSource web_filter_block_sites">Web_Filter#Block_Sites</span>
<span style="display:none" class="helpSource web_filter_block_sites">Web_Filter#Block_Sites</span>
Line 18: Line 19:
{| width='100%'
{| width='100%'
|-
|-
| align="center" | [[Image:WebFilter_128x128.png]] &nbsp; &nbsp; '''Web Filter'''
| align="center" | [[Image:WebFilter.png|128px]] &nbsp; &nbsp; '''Web Filter'''
| align="center" |
| align="center" |
{|
{|
Line 41: Line 42:
== About Web Filter ==
== About Web Filter ==


Web Filter monitors HTTP traffic on your network to monitor user behavior and block inappropriate content. Web Filter also appeals to customers who require an added level of protection or are subject to regulations, for example Web Filter helps libraries comply with the [http://en.wikipedia.org/wiki/Children%27s_Internet_Protection_Act Children's Internet Protection Act]). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.
Web Filter monitors HTTP and HTTPS traffic on your network to filter and log web activities and block inappropriate content. Web Filter also appeals to customers who require an added level of protection or are subject to regulations, for example Web Filter helps libraries comply with the [http://en.wikipedia.org/wiki/Children%27s_Internet_Protection_Act Children's Internet Protection Act]). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.


* '''Real-time classification and updates''': When your users visit a site, Untangle sends the URL to the [http://zvelo.com/technology/zvelodb-url-database cloud] to be categorized. When the data is returned, Untangle keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to block or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at [http://zvelo.com zVelo] and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or blocked according to your settings without additional intervention, plus you have the option of requesting [http://zvelo.com/partners/test-a-site recategorization] of sites.
* '''Real-time classification and updates''': When your users visit a site, NG Firewall sends the URL to the [https://www.webroot.com/us/en/business/threat-intelligence/internet/web-classification-and-reputation-services Webroot BrightCloud®] to be categorized. When the data is returned, NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at [https://www.webroot.com Webroot] and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention, plus you have the option of requesting [https://www.brightcloud.com/tools/url-ip-lookup.php recategorization] of sites.


* '''HTTPS Filtering''': Web Filter has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.
* '''HTTPS Filtering''': Web Filter has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.


* '''Detailed categorization''': Web Filter offers over 140 categories and over 450 million categorized sites. The Web Filter database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to block websites related to Sex, but allow sites dealing with Sexual Education or Pregnancy.  
* '''Detailed categorization''': Web Filter offers 79 categories and tens of billions of URLs. The Web Filter database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to flag websites related to nudity, but allow sites dealing with Sexual Education.  


* '''Advanced features''': Force safe-search on search engines, log user searches, restrict google domains, and more!
* '''Advanced features''': Force safe-search on search engines, filter and log user searches, restrict google domains, and more!


=== Traffic Flow ===
=== Traffic Flow ===
Line 63: Line 64:
# The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
# The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
# If 'Restrict Google applications' is enabled, the appropriate header is added to the request using the 'Allowed Domains' that have been configured.
# If 'Restrict Google applications' is enabled, the appropriate header is added to the request using the 'Allowed Domains' that have been configured.
# If 'Pass if referer matches Pass Sites' is enabled, the referer is checked against the Pass Sites list. If a match is found, the traffic is allowed.
# If 'Pass if referrer matches any Pass Sites' is enabled, the referrer is checked against the Pass Sites list. If a match is found, the traffic is allowed.
# If the Unblock option is enabled, the destination site and the source IP of the request are checked against the unblock list. If a match is found, the traffic is allowed.
# If the Unblock option is enabled, the destination site and the source IP of the request are checked against the unblock list. If a match is found, the traffic is allowed.
# If 'Block pages from IP only hosts' is enabled, the request will be evaluated and blocked if the destination is an IP address.
# If 'Block pages from IP only hosts' is enabled, the request will be evaluated and blocked if the destination is an IP address.
Line 93: Line 94:
=== Categories ===
=== Categories ===


Categories allows you to customize which categories of sites will be blocked or flagged. Categories that are blocked will display a block page to the user; categories that are flagged will allow the user to access the site, but will be silently flagged as a violation for event logs and [[Reports]]. These block/flag actions operate the same way for all of the different Web Filter options.
Categories enables you to customize which categories of sites will be blocked or flagged. Categories that are blocked will display a block page to the user; categories that are flagged will allow the user to access the site, but will be silently flagged as a violation for event logs and [[Reports]]. These block/flag actions operate the same way for all of the different Web Filter options.


{{AppScreenshot|web-filter|categories}}
{{AppScreenshot|web-filter|categories}}
=== Search Terms ===
Search Terms filtering enables you to flag or block specific search terms your users perform on popular search sites including Google, Bing, Ask, Yahoo, and YouTube. For example, if someone searches Google and includes the word "suicide", or they search for "twerking" videos on YouTube you can have these activities flagged or blocked.
Under Search Filter you can add terms you want to be blocked or flagged. Search Filter terms use the [[Glob Matcher]] syntax.
{{AppScreenshot|web-filter|search-terms}}
In many cases you may have an existing list of search terms that you want to import. For example, you can find several banned words from Facebook, YouTube, Wordpress and other sources [https://www.freewebheaders.com/full-list-of-bad-words-banned-by-google/ here]. The import feature enables you to import these lists from a comma separated or newline delimited file. You can also import in JSON format in case you are transferring a list from another NG Firewall.
{{Note|text='''Note:''' [[SSL Inspector]] must be installed and enabled to use search filtering. }}


=== Site Lookup ===
=== Site Lookup ===
Line 122: Line 135:
If you add an IP address to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses [[IP Matcher]] syntax.
If you add an IP address to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses [[IP Matcher]] syntax.


:If you have a few users that need to completely bypass Web Filter controls, consider using pass lists. If you have users that simply need different Web Filter settings, you should set up a separate rack using [[Policy Manager]]. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.
:If you have a few users that need to completely bypass Web Filter controls, consider using pass lists. If you have users that simply need different Web Filter settings, you should set up a separate policy using [[Policy Manager]]. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.


{{AppScreenshot|web-filter|pass-clients}}
{{AppScreenshot|web-filter|pass-clients}}
Line 455: Line 468:


=== Advanced ===
=== Advanced ===
The Advanced section allows you to configure additional web filter options.
The Advanced section enables you to configure additional web filter options.
 
==== Secure Name Indication ====


* '''Process HTTPS traffic by SNI (Server Name Indication) if present''': If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in [[#HTTPS Options]].
* '''Process HTTPS traffic by SNI (Server Name Indication) if present''': If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in [[#HTTPS Options]].
Line 462: Line 477:


* '''Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available''': If this option is enabled ''and'' neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in [[#HTTPS Options]].  
* '''Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available''': If this option is enabled ''and'' neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in [[#HTTPS Options]].  
==== Safe Browsing ====


* '''Enforce safe search on popular search engines''':  When this option is enabled, safe search will be enforced on all searches using supported search engines: Google, Yahoo!, Bing, Ask.
* '''Enforce safe search on popular search engines''':  When this option is enabled, safe search will be enforced on all searches using supported search engines: Google, Yahoo!, Bing, Ask.
* '''Enforce restrict mode on YouTube''':  When this option is enabled, [https://support.google.com/youtube/answer/7354993?hl=en restrict mode] will be enforced on all YouTube content.
* '''Force searches through kid-friendly search engine''':  When this option is enabled, all searches in popular search engines will be redirected through [https://www.kidzsearch.com kidzsearch.com]. [https://en.wikipedia.org/wiki/KidzSearch kidzsearch] is a visual child-safe search engine and web portal powered by Google Custom Search with academic autocomplete that emphasizes safety for children.
{{Note|text='''Note:''' [[SSL Inspector]] must be installed and enabled to use all Safe browsing options. }}
==== Block Options ====
* '''Block QUIC Sessions (UDP port 443)''':  If enabled, Web Filter prevents browsers from using the [https://en.wikipedia.org/wiki/QUIC QUIC] protocol. QUIC is used by Chrome browser to access many types of Google applications and services. By allowing QUIC, NG Firewall has less visibility and control over this type of traffic.


* '''Block pages from IP only hosts''': When this option is enabled, users entering an IP address rather than domain name will be blocked.
* '''Block pages from IP only hosts''': When this option is enabled, users entering an IP address rather than domain name will be blocked.


* '''Pass if referers match Pass Sites'''. When this option is checked, if a page contains external content from any site in ''Pass Sites'', that external content will be passed regardless of other block policies.
* '''Pass if referrer matches any Pass Sites'''. When this option is checked, if a site allowed via a Pass Site entry links to external content using a referrer, that external content will be passed regardless of Category settings. Please note that this setting requires [[SSL Inspector]] to work with HTTPS sites and many sites no longer use referrers.
 
* '''Close connection for blocked HTTPS sessions without redirecting to block page'''. If enabled, secure sites blocked by Web Filter do not redirect the user to a denial page and close the connection without any notice to the user. This is useful when you are not using [[SSL Inspector]] and the server's root certificate authority is not installed on the client device.
 
==== Google Restrictions ====
 
* '''Restrict Google applications''': When this option is enabled, only domains listed in '''Domain''' are allowed to access Google applications such as Gmail.  All others are blocked by Google.  Multiple domains can be specified, separated by commas such as '''google.com,domain.com'''. This adds an X-GoogApps-Allowed-Domains header to web requests which then gets enforced on Google's servers. More information on this feature can be found [https://support.google.com/a/answer/1668854?hl=en here].


* '''Restrict Google applications''': When this option is enabled, only domains listed in '''Domain''' are allowed to access Google applications such as Gmail. All others are blocked by Google. Multiple domains can be specified, separated by commas such as:
{{Note|text='''Note:''' [[SSL Inspector]] must be installed and enabled to restrict Google applications. }}
::<tt>untangle.com,domain.com</tt>.
 
:''NOTE: [[SSL Inspector]] must be installed and running with the Inspect Google Traffic configured to Inspect.''
==== Custom Block Page ====
[https://support.google.com/a/answer/1668854?hl=en 1]
 
* '''Custom block page URL''':  Set an external location where to redirect users when they are denied access to a web site by Web Filter. This is useful if you want to use your own server to process the denial in a different way than the built-in denial options.
The following query string variables are passed to the forward location so the receiving system can process the information.
 
{| class="wikitable"
|+
! style="text-align: left" | Variable
! style="text-align: left" | Description
|-
| reason
| The reason the user was denied access.
|-
| appname
| The NG Firewall app that is responsible for the denial.
|-
| appid
| The ID of the NG Firewall app that is responsible for the denial.
|-
| clientAddress
| The IP Address of the denied device.
|-
| url
| The denied URL that the user requested.
|}
 
==== Unblock Options ====


* '''Unblock''':  This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.
* '''Unblock''':  This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.
Line 478: Line 536:
:If Unblock is set to '''None''' no users will be allowed to bypass the block page. If Unblock is set to '''Temporary''' users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to '''Permanent and Global''' then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.
:If Unblock is set to '''None''' no users will be allowed to bypass the block page. If Unblock is set to '''Temporary''' users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to '''Permanent and Global''' then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.


:You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the Untangle or you can set a new, separate password only for the Unblock feature.
:You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the NG Firewall or you can set a new, separate password only for the Unblock feature.


* '''Clear Category URL Cache''': This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.
* '''Clear Category URL Cache''': This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.
Line 505: Line 563:


<blockquote>
<blockquote>
For example, if the user visits "https://wellsfargo.com/welcome" in the browser, it will see "wellsfargo.com" as the SNI information. If enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If "Banking" is flagged it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list. If "wellsfargo.com" is flaggeed it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list.
For example, if the user visits "https://wellsfargo.com/welcome" in the browser, "wellsfargo.com" is seen as the SNI information. If SNI-based categorization is enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If Web Filter is configured to flag "Financial Services" , then "https://wellsfargo.com/welcome" will be flagged, unless "wellsfargo.com" is in the pass list or the client IP address is in the client IP pass list.  
</blockquote>
</blockquote>


Line 520: Line 578:
</blockquote>
</blockquote>


'''Note:''' Neither HTTPS process (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required. Read [[HTTPS|here for more information]]
'''Note:''' Neither HTTPS processing (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required. Read [[HTTPS|here for more information]]


To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.
To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.


== Web Filter FAQs ==
== Web Filter FAQs ==

Latest revision as of 14:55, 16 September 2022

    Web Filter
Other Links:
Web Filter Description Page
Web Filter Demo
Web Filter Forums
Web Filter Reports
Web Filter FAQs



About Web Filter

Web Filter monitors HTTP and HTTPS traffic on your network to filter and log web activities and block inappropriate content. Web Filter also appeals to customers who require an added level of protection or are subject to regulations, for example Web Filter helps libraries comply with the Children's Internet Protection Act). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.

  • Real-time classification and updates: When your users visit a site, NG Firewall sends the URL to the Webroot BrightCloud® to be categorized. When the data is returned, NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at Webroot and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention, plus you have the option of requesting recategorization of sites.
  • HTTPS Filtering: Web Filter has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.
  • Detailed categorization: Web Filter offers 79 categories and tens of billions of URLs. The Web Filter database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to flag websites related to nudity, but allow sites dealing with Sexual Education.
  • Advanced features: Force safe-search on search engines, filter and log user searches, restrict google domains, and more!

Traffic Flow

When scanning traffic, Web Filter evaluates the pass lists, block lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the request is received from the client and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of filtering and control over both resources that are requested, and content that is returned in response.

HTTP Request

When evaluating HTTP requests, Web Filter applies the configured rules and lists in the following order:

  1. A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Filter as well as other applications.
  2. The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  3. The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  4. If 'Restrict Google applications' is enabled, the appropriate header is added to the request using the 'Allowed Domains' that have been configured.
  5. If 'Pass if referrer matches any Pass Sites' is enabled, the referrer is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  6. If the Unblock option is enabled, the destination site and the source IP of the request are checked against the unblock list. If a match is found, the traffic is allowed.
  7. If 'Block pages from IP only hosts' is enabled, the request will be evaluated and blocked if the destination is an IP address.
  8. The destination site of the request is checked against the Block Sites list. If a match is found, the traffic is blocked.
  9. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.
  10. The category determined in step #1 is compared to the Categories list, and the traffic is allowed, flagged, or blocked based on the corresponding match. If the category could not be determined, the traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Filter applies the configured rules and lists in the following order:

  1. The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  2. The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  3. If the Unblock option is enabled, the site from which the response was received and the client IP are checked against the unblock list. If a match is found, the traffic is allowed.
  4. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.

Settings

This section reviews the different settings and configuration options available for Web Filter.


Status

This displays the current status and some statistics.


Categories

Categories enables you to customize which categories of sites will be blocked or flagged. Categories that are blocked will display a block page to the user; categories that are flagged will allow the user to access the site, but will be silently flagged as a violation for event logs and Reports. These block/flag actions operate the same way for all of the different Web Filter options.

Search Terms

Search Terms filtering enables you to flag or block specific search terms your users perform on popular search sites including Google, Bing, Ask, Yahoo, and YouTube. For example, if someone searches Google and includes the word "suicide", or they search for "twerking" videos on YouTube you can have these activities flagged or blocked.

Under Search Filter you can add terms you want to be blocked or flagged. Search Filter terms use the Glob Matcher syntax.

In many cases you may have an existing list of search terms that you want to import. For example, you can find several banned words from Facebook, YouTube, Wordpress and other sources here. The import feature enables you to import these lists from a comma separated or newline delimited file. You can also import in JSON format in case you are transferring a list from another NG Firewall.

Note: SSL Inspector must be installed and enabled to use search filtering.

Site Lookup

Site Lookup allows you to find the categorization of a URL. Clicking it brings up a dialog. In Site URL specify the URL to find and click Search to find the URL's categorization.

If you feel the current categorization is incorrect, check Suggest a different category, select a new category from the list, and click Suggest to submit the category change for consideration.

NOTE: This is only a suggestion - it will trigger the URL categorization provider to recategorize all categories for the URL.

Block Sites

Under Block Sites you can add individual domain names you want to be blocked or flagged - just enter the domain name (e.g. youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

Pass Sites

Pass Sites is used to pass content that would have otherwise been blocked. This can be useful for "unblocking" sites that you don't want blocked according to block settings. Any domains you add to the Passed Sites list will be allowed, even if blocked by category or by individual URL - just add the domain and save. Unchecking the pass option will allow the site to be blocked as if the entry was not present. This list uses URL Matcher syntax.

Pass Clients

If you add an IP address to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

If you have a few users that need to completely bypass Web Filter controls, consider using pass lists. If you have users that simply need different Web Filter settings, you should set up a separate policy using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.

Rules

The Rules tab allows you to specify rules to Block or Flag traffic that passes through Web Filter.

The Rules documentation describes how rules work and how they are configured. Web Filter uses rules to determine to block or flag the specific session. Flagging a session marks it in the logs for reviewing in the event logs or reports, but has no direct effect on the network traffic.

Rule Actions

  • Flag: Allows the traffic which matched the rule to flow, and flags the traffic for easier viewing in the event log.
  • Block: Blocks the traffic which matched the rule.

Rule Types

In previous versions of Web Filter, there were dedicated lists for blocking certain file extensions or MIME types. This capability is still available using the more flexible filter rules. For blocking specific file extensions, you can create a rule with the condition HTTP: Response File Extension that has a comma separated list of the extensions to block in the Value field. For blocking MIME types, you would create a rule with the condition HTTP: Content Type that has a comma separated list of the content types to block in the Value field.

Below are tables that list the default file extensions and MIME types that were available in previous versions. Note that these lists are not exhaustive, but are included here as a reference, and to simplify creation of such rules via copy/paste of the values in the tables.

Extension Category Description
exe executable an executable file format
ocx executable an executable file format
dll executable an executable file format
cab executable an ActiveX executable file format
bin executable an executable file format
com executable an executable file format
jpg image an image file format
png image an image file format
gif image an image file format
jar java a Java file format
class java a Java file format
swf flash the flash file format
mp3 audio an audio file format
wav audio an audio file format
wmf audio an audio file format
mpg video a video file format
mov video a video file format
avi video a video file format
hqx archive an archived file format
cpt compression a compressed file format


Content Category Description
application/octet-stream unspecified data byte stream
application/x-msdownload Microsoft download executable
application/exe executable executable
application/x-exe executable executable
application/dos-exe DOS executable executable
application/x-winexe Windows executable executable
application/msdos-windows MS-DOS executable executable
application/x-msdos-program MS-DOS program executable
application/x-oleobject Microsoft OLE Object executable
application/x-java-applet Java Applet executable
audio/mpegurl MPEG audio URLs audio
audio/x-mpegurl MPEG audio URLs audio
audio/mp3 MP3 audio audio
audio/x-mp3 MP3 audio audio
audio/mpeg MPEG audio audio
audio/mpg MPEG audio audio
audio/x-mpeg MPEG audio audio
audio/x-mpg MPEG audio audio
application/x-ogg Ogg Vorbis audio
audio/m4a MPEG 4 audio audio
audio/mp2 MP2 audio audio
audio/mp1 MP1 audio audio
application/ogg Ogg Vorbis audio
audio/wav Microsoft WAV audio
audio/x-wav Microsoft WAV audio
audio/x-pn-wav Microsoft WAV audio
audio/aac Advanced Audio Coding audio
audio/midi MIDI audio audio
audio/mpeg MPEG audio audio
audio/aiff AIFF audio audio
audio/x-aiff AIFF audio audio
audio/x-pn-aiff AIFF audio audio
audio/x-pn-windows-acm Windows ACM audio
audio/x-pn-windows-pcm Windows PCM audio
audio/basic 8-bit u-law PCM audio
audio/x-pn-au Sun audio audio
audio/3gpp 3GPP audio
audio/3gpp-encrypted encrypted 3GPP audio
audio/scpls streaming mp3 playlists audio
audio/x-scpls streaming mp3 playlists audio
application/smil SMIL audio
application/sdp Streaming Download Project audio
application/x-sdp Streaming Download Project audio
audio/amr AMR codec audio
audio/amr-encrypted AMR encrypted codec audio
audio/amr-wb AMR-WB codec audio
audio/amr-wb-encrypted AMR-WB encrypted codec audio
audio/x-rn-3gpp-amr 3GPP codec audio
audio/x-rn-3gpp-amr-encrypted 3GPP-AMR encrypted codec audio
audio/x-rn-3gpp-amr-wb 3gpp-AMR-WB codec audio
audio/x-rn-3gpp-amr-wb-encrypted 3gpp-AMR_WB encrypted codec audio
application/streamingmedia Streaming Media audio
video/mpeg MPEG video video
audio/x-ms-wma Windows Media video
video/quicktime QuickTime video
video/x-ms-asf Microsoft ASF video
video/x-msvideo Microsoft AVI video
video/x-sgi-mov SGI movie video
video/3gpp 3GPP video video
video/3gpp-encrypted 3GPP encrypted video video
video/3gpp2 3GPP2 video video
audio/x-realaudio RealAudio audio
text/vnd.rn-realtext RealText text
audio/vnd.rn-realaudio RealAudio audio
audio/x-pn-realaudio RealAudio plug-in audio
image/vnd.rn-realpix RealPix image
application/vnd.rn-realmedia RealMedia video
application/vnd.rn-realmedia-vbr RealMedia VBR video
application/vnd.rn-realmedia-secure secure RealMedia video
application/vnd.rn-realaudio-secure secure RealAudio audio
audio/x-realaudio-secure secure RealAudio audio
video/vnd.rn-realvideo-secure secure RealVideo video
video/vnd.rn-realvideo RealVideo video
application/vnd.rn-realsystem-rmj RealSystem media video
application/vnd.rn-realsystem-rmx RealSystem secure media video
audio/rn-mpeg MPEG audio audio
application/x-shockwave-flash Macromedia Shockwave multimedia
application/x-director Macromedia Shockwave multimedia
application/x-authorware-bin Macromedia Authorware binary multimedia
application/x-authorware-map Macromedia Authorware shocked file multimedia
application/x-authorware-seg Macromedia Authorware shocked packet multimedia
application/futuresplash Macromedia FutureSplash multimedia
application/zip ZIP archive
application/x-lzh LZH archive archive
image/gif Graphics Interchange Format image
image/png Portable Network Graphics image
image/jpeg JPEG image
image/bmp Microsoft BMP image
image/tiff Tagged Image File Format image
image/x-freehand Macromedia Freehand image
image/x-cmu-raster CMU Raster image
image/x-rgb RGB image image
text/css cascading style sheet text
text/html HTML text
text/plain plain text text
text/richtext rich text text
text/tab-separated-values tab separated values text
text/xml XML text
text/xsl XSL text
text/x-sgml SGML text
text/x-vcard vCard text
application/mac-binhex40 Macintosh BinHex archive
application/x-stuffit Macintosh Stuffit archive archive
application/macwriteii MacWrite Document document
application/applefile Macintosh File archive
application/mac-compactpro Macintosh Compact Pro archive
application/x-bzip2 block compressed compressed
application/x-shar shell archive archive
application/x-gtar gzipped tar archive archive
application/x-gzip gzip compressed compressed
application/x-tar 4.3BSD tar archive archive
application/x-ustar POSIX tar archive archive
application/x-cpio old cpio archive archive
application/x-bcpio POSIX cpio archive archive
application/x-sv4crc System V cpio with CRC archive
application/x-compress UNIX compressed compressed
application/x-sv4cpio System V cpio archive
application/x-sh UNIX shell script executable
application/x-csh UNIX csh script executable
application/x-tcl Tcl script executable
application/x-javascript JavaScript executable
application/x-excel Microsoft Excel document
application/mspowerpoint Microsoft Powerpoint document
application/msword Microsoft Word document
application/wordperfect5.1 Word Perfect document
application/rtf Rich Text Format document
application/pdf Adobe Acrobat document
application/postscript Postscript document

Advanced

The Advanced section enables you to configure additional web filter options.

Secure Name Indication

  • Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in #HTTPS Options.
  • Process HTTPS traffic by hostname in server certificate when SNI information not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server and the server name on the certificate will be used for categorization and filtering purposes.
  • Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available: If this option is enabled and neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in #HTTPS Options.

Safe Browsing

  • Enforce safe search on popular search engines: When this option is enabled, safe search will be enforced on all searches using supported search engines: Google, Yahoo!, Bing, Ask.
  • Enforce restrict mode on YouTube: When this option is enabled, restrict mode will be enforced on all YouTube content.
  • Force searches through kid-friendly search engine: When this option is enabled, all searches in popular search engines will be redirected through kidzsearch.com. kidzsearch is a visual child-safe search engine and web portal powered by Google Custom Search with academic autocomplete that emphasizes safety for children.
Note: SSL Inspector must be installed and enabled to use all Safe browsing options.

Block Options

  • Block QUIC Sessions (UDP port 443): If enabled, Web Filter prevents browsers from using the QUIC protocol. QUIC is used by Chrome browser to access many types of Google applications and services. By allowing QUIC, NG Firewall has less visibility and control over this type of traffic.
  • Block pages from IP only hosts: When this option is enabled, users entering an IP address rather than domain name will be blocked.
  • Pass if referrer matches any Pass Sites. When this option is checked, if a site allowed via a Pass Site entry links to external content using a referrer, that external content will be passed regardless of Category settings. Please note that this setting requires SSL Inspector to work with HTTPS sites and many sites no longer use referrers.
  • Close connection for blocked HTTPS sessions without redirecting to block page. If enabled, secure sites blocked by Web Filter do not redirect the user to a denial page and close the connection without any notice to the user. This is useful when you are not using SSL Inspector and the server's root certificate authority is not installed on the client device.

Google Restrictions

  • Restrict Google applications: When this option is enabled, only domains listed in Domain are allowed to access Google applications such as Gmail. All others are blocked by Google. Multiple domains can be specified, separated by commas such as google.com,domain.com. This adds an X-GoogApps-Allowed-Domains header to web requests which then gets enforced on Google's servers. More information on this feature can be found here.
Note: SSL Inspector must be installed and enabled to restrict Google applications.

Custom Block Page

  • Custom block page URL: Set an external location where to redirect users when they are denied access to a web site by Web Filter. This is useful if you want to use your own server to process the denial in a different way than the built-in denial options.

The following query string variables are passed to the forward location so the receiving system can process the information.

Variable Description
reason The reason the user was denied access.
appname The NG Firewall app that is responsible for the denial.
appid The ID of the NG Firewall app that is responsible for the denial.
clientAddress The IP Address of the denied device.
url The denied URL that the user requested.

Unblock Options

  • Unblock: This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.
If Unblock is set to None no users will be allowed to bypass the block page. If Unblock is set to Temporary users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to Permanent and Global then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.
You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the NG Firewall or you can set a new, separate password only for the Unblock feature.
  • Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.

Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Filter.

Reports

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries: {{#section:All_Reports|'Web Filter'}}

The tables queried to render these reports:



HTTPS Options

There are many ways to handle HTTPS. An overview of the various techniques is described here.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Filter processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.

  • Process HTTPS traffic by SNI (Server Name Indication) if present.
  • Process HTTPS traffic by hostname in server certificate when SNI information not present
  • Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

If the SNI-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, "wellsfargo.com" is seen as the SNI information. If SNI-based categorization is enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If Web Filter is configured to flag "Financial Services" , then "https://wellsfargo.com/welcome" will be flagged, unless "wellsfargo.com" is in the pass list or the client IP address is in the client IP pass list.

If No SNI information is present and Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled, then the hostname will be pulled from the certificate presented to the client.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

If no SNI or certificate information was available and Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available the session will be processed and categorized by IP address. If the IP-based processing and categorization of the web requests determines the session should be flagged, the session is reset and no more processing of this session will be done. If the IP-based processing and categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS processing (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required. Read here for more information

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Filter FAQs

How do Web Filter and Web Monitor work?

Web Filter and Web Monitor both transparently scan HTTP and HTTPS traffic in order to monitor web activity on the network. Additionally, Web Filter can block inappropriate web activity as determined by the network administrator. Websites can be blocked or logged based on Category (Pornography, Social Networking, etc.), URL (facebook.com, youtube.com, etc.), content-type and much more.


Can I use both Web Filter and Web Monitor?

Web Monitor is a subset of the functionality of Web Filter. If you have Web Filter installed, there is no additional benefit of installing Web Monitor. If Web Filter has a valid license or is currently installed, Web Monitor will not be shown as installable.


How is Web Filter diffrent than Web Monitor?

Web Monitor is for monitoring web activity. Web Filter is for monitoring and blocking web activity. Web Filter is identical to Web Monitor, except with the ability to block and modify web content, such as:

  • Blocking web sites & categories
  • Enforce safe search
  • Restrict google applications


Can I install Web Filter or Web Monitor on a single computer to use as Parental Control software?

No. NG Firewall is designed to operate as a gateway or transparent bridge for an entire network and is not meant to filter the computer it is installed on. Installing NG Firewall will wipe out all existing data on the PC it is installed to. For filtering a single PC, other Internet filter/Parental Control software can be used.

Can I grant privileged access to some users while still blocking sites for everyone else?

There are several ways to accomplish this:

  • Policy Manager can be used to create multiple policies, which allows you to have separate filtering settings for individuals or groups of users or times of day, etc. The easiest example is a school, where you would want Teachers to have more relaxed internet filter settings than the students. Different settings can be applied to any individual or group in your organization such as CEOs, Administrative Assistants or Accounting Departments.
  • The Passed Client IPs List allows you to exempt specific client IPs from all filtering inside Web Filter.
  • The Unblock option displays a button that, when clicked, will allow users to bypass the block page. Web Filter has an additional option to require a password for this.


Can I let users access certain sites during lunch or after hours?

You can leverage Policy Manager to set up specific filtering settings for different days or time periods, such as allowing Facebook during breaks or after work hours.

Create a new policy for lunch or after hours, then create policy rules to send traffic to be processed by those policies during the desired hours.

How do I submit a mis-categorized or uncategorized site?

You can go to BrightCloud's website and submit the correct (or new) categorization. It will be reviewed immediately by a human. Once the new categorization takes effect you may need to flush your category cache in Web Filter to see the new categorization.

Does Web Filter use a lot of memory and CPU?

If your NG Firewall server is operating well without Web Filter, then you won't see much of a difference if you run Web Filter. Web Filter does not use much memory, and its cloud-based architecture adds very little to CPU utilization.

How do real-time updates work?

When a client first vists a site, Web Filter queries Webroot Brightcloud® to get the categories the site is under to make a decision to block or pass based on your configuration. The category information is also written to a local cache so it doesn't have to be checked the next time a user visits that site.

How long does Web Filter cache category information for sites?

Several days. Web Filter flushes non-frequently used cache. The websites that you visit daily will not be cleared from cache.

Can I add additional categories?

Custom categories are not available, however we provide over 140 categories for granular control over what your clients can access. If you feel there are categories that we can add to make it even better, just let us know.


How should I handle false positives?

While the fastest way to allow clients to access a site that is currently blocked is to add the site to your pass list, you can request recategorization of sites here - the turnaround time is usually less than two days.


Can I use Web Filter to block HTTPS/SSL sites?

Yes - because Web Filter has access to a separate database of IP addresses, it can categorize HTTPS traffic based on certificate, SNI, or the destination IP address. This is not done by individual domain, but by category - for example, if you simply block 'facebook.com'. Please note that this does not mean Web Filter can parse HTTPS content as it is encrypted. This means other forms of blocking like URI, file-type, mime-type, etc can not be done on HTTPS as the stream is encrypted and these require parsing of the HTTP protocol.

To accomplish this level of blocking SSL Inspector is required. More options about handling SSL are described here.


Can I block all web sites except certain ones?

Yes, simply block all categories (including "Uncategorized"). Then add whatever sites you'd like to pass to the Pass List. Please be aware that the complex nature of the web and the fact that many applications communicate over HTTP can make this approach difficult.

Alternatively, the rules can be used. Just add a rule to block all web traffic, then explicitly add any sites to the pass list or in rules above the block rule.


Why did 'Youtube for Schools' disappear?

Google/YouTube stopped supporting their YouTube for schools features. These features relied on NG Firewall adding an identification header to HTTP requests and then YouTube would enforce the policy on the server. Since this feature is no longer supported by their servers the feature has been removed.

Windows computers showing "No Internet Access" but everything is fine. Why?

Make sure you're not blocking access to the domain www.msftncsi.com; this is part of a test that Microsoft runs to see if there is an active internet connection. Once you've verified this domain is not blocked, simply restart the PC and that should take care of it.


Why is a site not being properly displayed even though I added it to the Pass List?

It's common for a web site to display links, banners and content from other web sites as part of their web pages. There are two easy methods to re-integrate the content while maintaining your access controls. A good example is Facebook - when you go to 'facebook.com', much of the site is loaded from 'fbcdn.net'. Sometimes additional sites must be added to the pass list. To fix these problems, look at the Blocked Web Events in Reports to see what is being blocked. Continue to add related sites that are being blocked to the pass list until it works as desired.

Retrieved from "https://wiki.edge.arista.com/index.php?title=Web_Filter&oldid=28408"